Model automatic evaluation task scheduling method and related products

Abstract

The application provides a model automatic evaluation task scheduling method and related products. The method comprises: obtaining an evaluation task specification of a current evaluation task, and generating an evaluation task dependency graph comprising a plurality of evaluation task nodes according to the evaluation task specification; querying corresponding reference evaluation task information, determining evaluation condition change information according to the evaluation task specification and the query result; determining an evaluation task node affected as a target evaluation task node, when the reference evaluation task information is queried and there is an evaluation task node not affected, reusing a corresponding existing evaluation product associated with the reference evaluation task information; executing the target evaluation task node according to the evaluation task dependency graph, and obtaining and outputting a current evaluation result according to at least one of a newly generated evaluation product and a reused existing evaluation product. Thus, the node execution range can be adjusted according to the evaluation condition change, and repeated processing can be reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method for arranging automatic model evaluation tasks and related products. Background Technology

[0002] With the application of large language models and intelligent agents built upon them in scenarios such as information retrieval, intelligent customer service, business consultation, and task execution, the output quality of these models and agents directly impacts business processing efficiency and user experience. Therefore, before deploying a model or intelligent agent or updating its version, it is typically necessary to evaluate its task execution capabilities and output quality under different business scenarios.

[0003] Existing model evaluation methods mainly include manual sampling evaluation, offline evaluation based on a fixed evaluation set, evaluation implemented through a general evaluation platform, and automated evaluation triggered by the development pipeline. While these methods can solve some of the automation issues, model evaluation typically involves multiple processing stages, such as evaluation data preparation, model interface calls, execution information collection, call result evaluation, and evaluation result output. Different processing stages may be implemented by different tools, scripts, or platform functions.

[0004] In some solutions, there is a lack of unified organization and coordination between different processing stages, requiring execution according to a pre-set fixed process, resulting in a fragmented evaluation process. When the tested object, evaluation data, calling configuration, or evaluation configuration changes, the original process is difficult to adapt to the actual needs of the evaluation, usually requiring reconfiguration and re-execution of all or most processing stages.

[0005] Furthermore, intermediate data and processing results generated during model evaluation are typically tied to the corresponding evaluation process. When subsequent evaluations only adjust certain aspects, the intermediate data and processing results already generated during historical evaluations are difficult to directly use in subsequent evaluations, easily leading to duplicate processing of evaluation data, duplicate calls to model interfaces, and duplicate evaluation of call results. Especially when the amount of evaluation data is large or the model call cost is high, the aforementioned duplicate processing increases the execution time and computational resource consumption of the evaluation task.

[0006] Therefore, existing model evaluation schemes suffer from problems such as fragmented processes between different processing stages, low adaptability of the evaluation process to changes in tasks, and excessive repetitive processing. Summary of the Invention

[0007] The embodiments of this application propose a method for arranging automatic model evaluation tasks and related products.

[0008] In a first aspect, this application provides a method for automatically evaluating model tasks, the method comprising: Obtain the evaluation task specifications for the current evaluation task; An evaluation task dependency graph is generated according to the evaluation task specification, wherein the evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependency relationships, and the multiple evaluation task nodes include at least a called node of the tested intelligent agent and a called result evaluation node; Query the reference evaluation task information corresponding to the current evaluation task to obtain the query results; Based on the evaluation task specifications and the query results, determine the evaluation condition change information; Based on the evaluation condition change information, the affected evaluation task nodes are identified as target evaluation task nodes. If the query result indicates that the reference evaluation task information is found and there are unaffected evaluation task nodes, the existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes are reused. The target evaluation task node is executed according to the evaluation task dependency graph, and the current evaluation result is obtained and output based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

[0009] In some optional implementations, the evaluation task specification includes configuration information of the tested intelligent agent and evaluation process configuration information. The evaluation process configuration information is used to determine the processing object and execution parameters of each evaluation task node. Obtaining the evaluation task specification of the current evaluation task includes: Obtain the evaluation requirements for the tested intelligent agent; Based on the evaluation requirements, the configuration information of the tested intelligent agent and the evaluation process configuration information are analyzed to obtain the task configuration information; The task configuration information is verified for completeness. Missing configuration items in the task configuration information are completed or the missing configuration items are identified as configuration items to be confirmed. Conflicting configuration items in the task configuration information are also identified as configuration items to be confirmed. If the task configuration information meets the integrity requirements and there are no configuration items to be confirmed, the evaluation task specification is generated based on the task configuration information. If the configuration item to be confirmed exists, the configuration item to be confirmed is output, and in response to receiving confirmation information for the configuration item to be confirmed, the task configuration information is updated according to the confirmation information, and the evaluation task specification is generated according to the updated task configuration information.

[0010] In some optional implementations, determining the evaluation condition change information based on the evaluation task specifications and the query results includes: If the query result indicates that the reference evaluation task information is found, the task configuration information corresponding to the current evaluation task is compared with the task configuration information recorded in the reference evaluation task information to determine the configuration items that have changed, the affected evaluation task nodes are determined based on the changed configuration items, and evaluation condition change information is generated based on the affected evaluation task nodes. If the query results indicate that the reference evaluation task information is not found, evaluation condition change information is generated to indicate that all of the multiple evaluation task nodes are affected.

[0011] In some optional implementations, the query result indicates that the reference evaluation task information has been retrieved, and the method further includes: The evaluation image specification is generated or loaded according to the reference evaluation task information. The evaluation image specification records the non-tested evaluation conditions associated with the reference evaluation task information, excluding the tested version parameters and the environmental parameters used to locate the tested version, as well as the version information or consistency verification information corresponding to the non-tested evaluation conditions. According to the evaluation mirror specification, the consistency of the non-tested evaluation conditions corresponding to the current evaluation task is checked. If the non-tested evaluation conditions pass the consistency check, and the evaluation condition change information indicates that the changed evaluation conditions only include at least one of the tested version parameters and the environment parameters, then the non-tested evaluation conditions recorded in the evaluation mirror specification are determined as the non-tested evaluation conditions of the current evaluation task. At least one of the tested version parameters and the environment parameters that have changed and are associated with the reference evaluation task information is replaced with the corresponding parameters in the current evaluation task. Based on the replaced evaluation conditions, the target evaluation task node is executed to obtain the current evaluation result. The current evaluation result is then compared with the reference evaluation result corresponding to the reference evaluation task, and a version comparison result is output. If at least one of the non-tested evaluation conditions fails the consistency check, the evaluation condition change information is updated based on the non-tested evaluation condition that failed the consistency check. The target evaluation task node is then re-determined based on the updated evaluation condition change information, and the re-determined target evaluation task node is executed based on the evaluation conditions corresponding to the current evaluation task.

[0012] In some optional implementations, the target evaluation task node includes an evaluation data processing node, the evaluation product generated by the evaluation data processing node includes an evaluation set, and executing the evaluation data processing node includes: Read the evaluation set source parameters from the evaluation task specifications; If the evaluation set source parameter points to an existing evaluation set, the existing evaluation set is determined as the initial evaluation set; When the evaluation set source parameter points to the seed evaluation set, based on the evaluation scenarios, scenario descriptions and user query examples in the seed evaluation set, an extended evaluation sample corresponding to at least one of boundary problems, query variants and combined scenarios is generated, and the initial evaluation set is determined based on the seed evaluation set and the extended evaluation sample. The evaluation samples in the initial evaluation set are cleaned, deduplicated, and quality checked to obtain the evaluation set. The fields of each evaluation sample in the evaluation set are parsed to obtain the user query, and at least one of the expected execution steps, calling constraints and result assertions associated with the user query; The fields corresponding to the user query are mapped to the input fields required for the tested agent to call the interface, and at least one of the expected execution steps, calling constraints, and result assertions associated with the user query is determined as the evaluation basis for evaluating the model calling results generated by the tested agent in response to the user query.

[0013] In some optional implementations, the target evaluation task node includes the tested agent invocation node, and executing the tested agent invocation node includes: Obtain the business space corresponding to the tested intelligent agent and the evaluation set used to call the tested intelligent agent; The template is invoked based on the business space and the intelligent agent under test query model; If an existing model invocation template corresponding to the business space and the tested intelligent agent is found, the existing model invocation template is determined as the target model invocation template; If no existing model call template is found, obtain the call chain document or call log corresponding to the tested agent, parse the interface call relationship and request parameters according to the call chain document or call log, generate a model call template according to the parsing result, and determine the generated model call template as the target model call template. Based on the target model, a debug call process and a full call process are generated using the call template. Select a portion of the evaluation samples from the evaluation set, call the tested intelligent agent according to the debugging call process, and set the current evaluation task to a waiting confirmation state. In response to receiving the continue execution information, the test agent is invoked using the evaluation set according to the full call process, and the model call results and corresponding execution trajectory identifiers generated by the test agent for each evaluation sample are obtained. In response to receiving correction information, the corresponding evaluation task parameters are updated according to the correction information, and the evaluation task node using the updated evaluation task parameters is re-executed.

[0014] In some optional implementations, the target evaluation task node includes the call result evaluation node, and executing the call result evaluation node includes: Obtain the evaluation samples used by the calling node of the tested intelligent agent, the model calling results output by the calling node of the tested intelligent agent, and the execution trajectory identifier corresponding to the model calling results; Based on the execution trajectory identifier, obtain the model execution process information, tool call information, and interface execution information generated when the tested intelligent agent processes the corresponding evaluation sample; The evaluation samples, model call results, model execution process information, tool call information, and interface execution information are associated to generate an evaluation record; If there is no existing evaluation record set corresponding to the current evaluation task, an evaluation record set is created based on multiple evaluation records, and the created evaluation record set is determined as the target evaluation record set. If there is an existing evaluation record set corresponding to the current evaluation task, update the existing evaluation record set according to multiple evaluation records, generate a new evaluation record set version, and determine the new evaluation record set version as the target evaluation record set; The existing evaluator is read according to the evaluation task specification and the existing evaluator is determined as the target evaluator, or an evaluator is generated according to the evaluation requirement document and evaluation rules corresponding to the current evaluation task and the generated evaluator is determined as the target evaluator; Determine the evaluator identifier and evaluator version of the target evaluator; The target evaluation record set, the evaluator identifier, and the evaluator version are submitted as an evaluation task, and the running status of the evaluation task is polled until the evaluation task reaches the end state, and the evaluation status of each evaluation record in the target evaluation record set is obtained. Based on the evaluation status of each evaluation record, determine the evaluation records that have been successfully evaluated and the evaluation records that have not been successfully evaluated, and obtain the evaluation results corresponding to the evaluation records that have been successfully evaluated. In the absence of any evaluation records that have not been successfully completed, the evaluation result corresponding to the evaluation record that has been successfully completed shall be determined as the current evaluation result; In the case of evaluation records that have not been successfully evaluated, the target evaluator is used to re-evaluate the evaluation records that have not been successfully evaluated to obtain the retry evaluation result. The evaluation result corresponding to the evaluation record that has been successfully evaluated is then merged with the retry evaluation result to obtain the current evaluation result.

[0015] In some optional implementations, the target evaluation task node further includes an evaluation result generation node. Executing the evaluation result generation node includes: Obtain the current evaluation result and the evaluation sample corresponding to the current evaluation task, and obtain the sample-level evaluation result corresponding to each evaluation sample from the current evaluation result; The sample-level evaluation results are aggregated to obtain aggregated evaluation results; The evaluation details are generated by integrating the sample-level evaluation results, the aggregated evaluation results, and at least one of the expected execution steps, invocation constraints, and result assertions in the evaluation samples. The evaluation indicators are calculated based on the evaluation details, and the failed samples are determined based on the evaluation indicators and the sample-level evaluation results. Obtain the execution trajectory and execution trajectory identifier corresponding to the failed sample, and perform failure cause analysis on the failed sample based on the execution trajectory to obtain the failure analysis result; An evaluation report is generated based on the evaluation details, the evaluation indicators, and the failure analysis results, and an association is established between the failure analysis results and the corresponding execution trajectory identifiers. A quality access control conclusion is generated based on the evaluation indicators and preset quality judgment conditions.

[0016] In some optional implementations, the target evaluation task node further includes a reflective decision generation node, and executing the reflective decision generation node includes: Based on at least one of the current evaluation results, the failed samples, the evaluation indicators, and the quality access control conclusion, determine the evaluation anomaly information; Obtain the execution trajectory corresponding to the evaluation anomaly information and the historical evaluation information corresponding to the tested intelligent agent; Based on the evaluation anomaly information, the execution trajectory, and the historical evaluation information, determine the failure type and the root cause of the failure; The improvement targets are determined based on the failure types; Based on the root cause of failure and the execution trajectory, failure evidence is generated, and a reflective decision is generated including the failure evidence, the root cause of failure, and improvement actions for the improvement target. The reflected decisions will be incorporated into the evaluation knowledge base.

[0017] Secondly, this application provides a model automatic evaluation task orchestration device, the device comprising: The specification acquisition unit is used to acquire the evaluation task specification for the current evaluation task; The dependency graph generation unit is used to generate an evaluation task dependency graph according to the evaluation task specification, wherein the evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependency relationships, and the multiple evaluation task nodes include at least a called node of the tested intelligent agent and a called result evaluation node; The information query unit is used to query reference evaluation task information corresponding to the current evaluation task and obtain query results. The change information determination unit is used to determine the change information of the evaluation conditions based on the evaluation task specifications and the query results; The node orchestration unit is used to determine the affected evaluation task nodes as target evaluation task nodes based on the evaluation condition change information, and to reuse the existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes when the query result indicates that the reference evaluation task information is found and there are unaffected evaluation task nodes. The task execution unit is used to execute the target evaluation task node according to the evaluation task dependency graph, and obtain and output the current evaluation result based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

[0018] In some optional implementations, the evaluation task specification includes configuration information of the tested intelligent agent and evaluation process configuration information. The evaluation process configuration information is used to determine the processing object and execution parameters of each evaluation task node. The specification acquisition unit is specifically used for: Obtain the evaluation requirements for the tested intelligent agent; Based on the evaluation requirements, the configuration information of the tested intelligent agent and the evaluation process configuration information are analyzed to obtain the task configuration information; The task configuration information is verified for completeness. Missing configuration items in the task configuration information are completed or the missing configuration items are identified as configuration items to be confirmed. Conflicting configuration items in the task configuration information are also identified as configuration items to be confirmed. If the task configuration information meets the integrity requirements and there are no configuration items to be confirmed, the evaluation task specification is generated based on the task configuration information. If the configuration item to be confirmed exists, the configuration item to be confirmed is output, and in response to receiving confirmation information for the configuration item to be confirmed, the task configuration information is updated according to the confirmation information, and the evaluation task specification is generated according to the updated task configuration information.

[0019] In some optional implementations, the change information determination unit is specifically used for: If the query result indicates that the reference evaluation task information is found, the task configuration information corresponding to the current evaluation task is compared with the task configuration information recorded in the reference evaluation task information to determine the configuration items that have changed, the affected evaluation task nodes are determined based on the changed configuration items, and evaluation condition change information is generated based on the affected evaluation task nodes. If the query results indicate that the reference evaluation task information is not found, evaluation condition change information is generated to indicate that all of the multiple evaluation task nodes are affected.

[0020] In some optional implementations, the query result indicates that the reference evaluation task information has been retrieved, and the apparatus further includes an evaluation image processing unit.

[0021] The evaluation image processing unit is configured to: generate or load an evaluation image specification based on the reference evaluation task information, wherein the evaluation image specification records non-tested evaluation conditions associated with the reference evaluation task information, excluding the tested version parameters and environmental parameters used to locate the tested version, and version information or consistency verification information corresponding to the non-tested evaluation conditions; perform consistency verification on the non-tested evaluation conditions corresponding to the current evaluation task based on the evaluation image specification; and, if the non-tested evaluation conditions pass the consistency verification, and the evaluation condition change information indicates that the changed evaluation conditions only include at least one of the tested version parameters and the environmental parameters, determine the non-tested evaluation conditions recorded in the evaluation image specification as the current evaluation task. For non-tested evaluation conditions, at least one of the changed parameters of the tested version and the environment associated with the reference evaluation task information is replaced with the corresponding parameters in the current evaluation task. The target evaluation task node is executed based on the replaced evaluation conditions to obtain the current evaluation result. The current evaluation result is compared with the reference evaluation result associated with the reference evaluation task information, and a version comparison result is output. If at least one of the non-tested evaluation conditions fails the consistency check, the evaluation condition change information is updated according to the non-tested evaluation conditions that failed the consistency check. The target evaluation task node is re-determined according to the updated evaluation condition change information, and the re-determined target evaluation task node is executed according to the evaluation conditions corresponding to the current evaluation task.

[0022] In some optional implementations, the task execution unit includes an evaluation data processing unit, which is used to execute the evaluation data processing node, and the evaluation products generated by the evaluation data processing node include an evaluation set.

[0023] The evaluation data processing unit is specifically used for: reading evaluation set source parameters from the evaluation task specification; determining the existing evaluation set as the initial evaluation set when the evaluation set source parameters point to an existing evaluation set; generating extended evaluation samples corresponding to at least one of boundary problems, query variants, and combined scenarios based on the evaluation scenarios, scenario descriptions, and user query examples in the seed evaluation set when the evaluation set source parameters point to a seed evaluation set, and determining the initial evaluation set based on the seed evaluation set and the extended evaluation samples; cleaning, deduplicating, and quality-checking the evaluation samples in the initial evaluation set to obtain the evaluation set; parsing the fields of each evaluation sample in the evaluation set to obtain the user query, and at least one of the expected execution steps, calling constraints, and result assertions associated with the user query; mapping the fields corresponding to the user query to the input fields required by the tested intelligent agent to call the interface, and determining at least one of the expected execution steps, calling constraints, and result assertions associated with the user query as the evaluation basis for evaluating the model call results generated by the tested intelligent agent for the user query.

[0024] In some optional implementations, the task execution unit includes a test agent invocation unit, which is used to execute the test agent invocation node.

[0025] The tested agent invocation unit is specifically used for: obtaining the business space corresponding to the tested agent and the evaluation set for invoking the tested agent; querying the model invocation template according to the business space and the tested agent; if an existing model invocation template corresponding to the business space and the tested agent is found, determining the existing model invocation template as the target model invocation template; if no existing model invocation template is found, obtaining the call chain document or call log corresponding to the tested agent, parsing the interface call relationship and request parameters according to the call chain document or call log, generating the model invocation template according to the parsing result, and then using the generated model invocation template... The board is determined as the target model invocation template; a debug invocation process and a full invocation process are generated according to the target model invocation template; a portion of the evaluation samples are selected from the evaluation set, and the tested agent is invoked according to the debug invocation process, and the current evaluation task is set to a waiting confirmation state; in response to receiving the continue execution information, the tested agent is invoked using the evaluation set according to the full invocation process, and the model invocation results and corresponding execution trajectory identifiers generated by the tested agent for each evaluation sample are obtained; in response to receiving the correction information, the corresponding evaluation task parameters are updated according to the correction information, and the evaluation task node with the updated evaluation task parameters is re-executed.

[0026] In some optional implementations, the task execution unit includes a call result evaluation unit, which is used to execute the call result evaluation node.

[0027] The call result evaluation unit is specifically used for: obtaining the evaluation samples used by the calling node of the tested agent, the model call results output by the calling node of the tested agent, and the execution trajectory identifier corresponding to the model call results; obtaining the model execution process information, tool call information, and interface execution information generated by the tested agent when processing the corresponding evaluation samples according to the execution trajectory identifier; associating the evaluation samples, the model call results, the model execution process information, the tool call information, and the interface execution information to generate evaluation records; if there is no existing evaluation record set corresponding to the current evaluation task, creating an evaluation record set according to multiple evaluation records, and determining the created evaluation record set as the target evaluation record set; if there is an existing evaluation record set corresponding to the current evaluation task, updating the existing evaluation record set according to multiple evaluation records to generate a new evaluation record set version, and determining the new evaluation record set version as the target evaluation record set; reading the existing evaluator according to the evaluation task specification and determining the existing evaluator as the target evaluator, or according to... An evaluator is generated based on the evaluation requirement document and evaluation rules corresponding to the current evaluation task, and the generated evaluator is identified as the target evaluator. The evaluator identifier and evaluator version of the target evaluator are determined. The target evaluation record set, the evaluator identifier, and the evaluator version are submitted as an evaluation task, and the running status of the evaluation task is polled until the evaluation task reaches its end state. The evaluation status of each evaluation record in the target evaluation record set is obtained. Based on the evaluation status of each evaluation record, evaluation records that have successfully completed evaluation and those that have not successfully completed evaluation are determined, and the evaluation results corresponding to the evaluation records that have successfully completed evaluation are obtained. If there are no evaluation records that have not successfully completed evaluation, the evaluation results corresponding to the evaluation records that have successfully completed evaluation are determined as the current evaluation result. If there are evaluation records that have not successfully completed evaluation, the target evaluator is used to re-evaluate the evaluation records that have not successfully completed evaluation, obtaining a retry evaluation result. The evaluation results corresponding to the evaluation records that have successfully completed evaluation are merged with the retry evaluation results to obtain the current evaluation result.

[0028] In some optional implementations, the task execution unit further includes an evaluation result generation unit, which is used to execute the evaluation result generation node.

[0029] The evaluation result generation unit is specifically used for: obtaining the current evaluation result and the evaluation sample corresponding to the current evaluation task, and obtaining the sample-level evaluation result corresponding to each evaluation sample from the current evaluation result; aggregating the sample-level evaluation results to obtain an aggregated evaluation result; integrating the sample-level evaluation result, the aggregated evaluation result, and at least one of the expected execution steps, calling constraints, and result assertions in the evaluation sample to generate evaluation details; calculating evaluation indicators based on the evaluation details, and determining failed samples based on the evaluation indicators and the sample-level evaluation results; obtaining the execution trajectory and execution trajectory identifier corresponding to the failed sample, and performing failure cause analysis on the failed sample based on the execution trajectory to obtain failure analysis results; generating an evaluation report based on the evaluation details, the evaluation indicators, and the failure analysis results, and establishing the association between the failure analysis results and the corresponding execution trajectory identifier; and generating a quality access control conclusion based on the evaluation indicators and preset quality judgment conditions.

[0030] In some optional implementations, the task execution unit further includes a reflective decision generation unit, which is used to execute the reflective decision generation node.

[0031] The reflection decision generation unit is specifically used for: determining evaluation anomaly information based on at least one of the current evaluation result, the failure sample, the evaluation index, and the quality gate conclusion; acquiring the execution trajectory corresponding to the evaluation anomaly information and the historical evaluation information corresponding to the tested intelligent agent; determining the failure type and failure root cause based on the evaluation anomaly information, the execution trajectory, and the historical evaluation information; determining the improvement target based on the failure type; generating failure evidence based on the failure root cause and the execution trajectory, and generating a reflection decision including the failure evidence, the failure root cause, and improvement actions for the improvement target; and writing the reflection decision into the evaluation knowledge base.

[0032] Thirdly, this application provides an electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the above-mentioned one or more programs are executed by the above-mentioned one or more processors, the above-mentioned one or more processors implement the method as described in any embodiment of the first aspect of this application.

[0033] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any embodiment of the first aspect of this application.

[0034] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method described in any embodiment of the first aspect of this application.

[0035] The model automatic evaluation task orchestration method and related products provided in this application first obtain the evaluation task specification for the current evaluation task and generate an evaluation task dependency graph based on the specification. The evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependencies, and these nodes include at least a node for the agent under test to invoke the evaluation and a node for evaluating the invocation result. Then, reference evaluation task information corresponding to the current evaluation task is queried to obtain the query results, and evaluation condition change information is determined based on the evaluation task specification and the query results. Next, based on the evaluation condition change information, affected evaluation task nodes are identified as target evaluation task nodes. If reference evaluation task information is found and unaffected evaluation task nodes exist, existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes are reused. Finally, the target evaluation task nodes are executed according to the evaluation task dependency graph, and the current evaluation result is obtained and output based on at least one of the evaluation products generated by the target evaluation task nodes and the reused existing evaluation products.

[0036] This application generates an evaluation task dependency graph based on the evaluation task specification and connects different evaluation task nodes using input-output dependencies. This organizes interconnected processing steps such as calling the tested agent and evaluating the call results into a unified evaluation execution flow, reducing the fragmentation of different evaluation processing steps and the reliance on manual connection. By determining the evaluation condition change information based on the query results of the evaluation task specification and reference evaluation task information, and identifying the affected evaluation task nodes based on the evaluation condition change information, the execution scope of the current evaluation task can be adapted to the actual changes in the evaluation conditions. This reduces the need to re-execute all evaluation processing steps due to changes in some evaluation conditions, improving the adaptability of the evaluation process to task changes. When reference evaluation task information is found and there are unaffected evaluation task nodes, the existing evaluation products corresponding to the unaffected evaluation task nodes are reused. This reduces the need for repeated data processing, tested agent calling, and result evaluation of evaluation content that has not undergone substantial changes, thereby reducing the number of model interface calls and the computational resources consumed during the evaluation process. By executing the target evaluation task node according to the evaluation task dependency graph, and combining the newly generated evaluation products of the target evaluation task node with the reused existing evaluation products to obtain the current evaluation result, the current evaluation task can be completed while maintaining the data connection between each evaluation processing link, reducing the processing time added by repeated execution, thereby improving the orchestration efficiency and execution efficiency of the model evaluation task. Attached Figure Description

[0037] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the scope of this application. In the drawings: Figure 1 This is a schematic diagram of the system architecture in which the automatic model evaluation task orchestration method provided in the embodiments of this application can be applied; Figure 2 This is a flowchart illustrating the automatic model evaluation task orchestration method provided in the embodiments of this application; Figure 3 This is an exploded flowchart of an embodiment of step S201 of this application; Figure 4 This is an exploded flowchart of an embodiment of step S204 of this application; Figure 5 This is a schematic diagram of the structure of the automatic model evaluation task orchestration device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0038] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] Figure 1 An exemplary system architecture 100 is shown that can be applied to the model automatic evaluation task orchestration method provided in the embodiments of this application.

[0041] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various communication connection types, such as wired communication links, wireless communication links, etc.

[0042] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send data. Various client applications can be installed on terminal devices 101, 102, and 103, such as model evaluation applications, agent management applications, evaluation data management applications, evaluation task management applications, evaluation report viewing applications, artificial intelligence applications, web browser applications, development pipeline management applications, instant messaging tools, email clients, cloud storage applications, etc. Users can input or select evaluation requirements for the current evaluation task through these client applications, submit the evaluation task, confirm or correct evaluation task parameters, and receive the current evaluation results output by server 105.

[0043] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display and data interaction functions, including but not limited to smartphones, tablets, laptops, desktop computers, and workstations. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. Terminal devices 101, 102, and 103 can be implemented as multiple software programs or software units, such as software units for obtaining evaluation requirements, submitting evaluation tasks, and displaying current evaluation results, or they can be implemented as a single software program or software unit; this application does not specifically limit this.

[0044] Server 105 can be a server that provides various data processing services, such as a backend server that orchestrates and executes evaluation tasks submitted by terminal devices 101, 102, and 103. For example, server 105 can obtain the evaluation task specification of the current evaluation task and generate an evaluation task dependency graph based on the evaluation task specification.

[0045] In some cases, the automatic model evaluation task orchestration method provided in this application can be jointly executed by terminal devices 101, 102, and 103 and server 105. For example, the steps of obtaining evaluation requirements, submitting evaluation tasks, receiving confirmation or correction information, and displaying the current evaluation results can be executed by terminal devices 101, 102, and 103; the steps of obtaining evaluation task specifications, generating evaluation task dependency graphs, querying reference evaluation task information, determining evaluation condition change information, determining target evaluation task nodes, reusing existing evaluation products, executing target evaluation task nodes, and obtaining the current evaluation results can be executed by server 105. This application does not specifically limit this. Correspondingly, different functional units in the automatic model evaluation task orchestration device can also be respectively set in terminal devices 101, 102, and 103 and server 105.

[0046] In some cases, the automatic model evaluation task orchestration method provided in this application can be executed by server 105, and correspondingly, the automatic model evaluation task orchestration device can also be set in server 105. In this case, system architecture 100 may not include terminal devices 101, 102, and 103, and server 105 can obtain the evaluation task specifications, reference evaluation task information, and evaluation data required to execute the automatic model evaluation task orchestration method from other task triggering systems, development pipelines, model management systems, or data storage systems.

[0047] In some cases, the automatic model evaluation task orchestration method provided in this application can be executed by terminal devices 101, 102, and 103, and correspondingly, the automatic model evaluation task orchestration device can also be set in terminal devices 101, 102, and 103. In this case, the system architecture 100 may not include server 105.

[0048] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software units, such as software units used to provide services for obtaining evaluation task specifications, generating evaluation task dependency graphs, analyzing changes in evaluation conditions, orchestrating evaluation task nodes, and executing evaluation tasks. It can also be implemented as a single software program or software unit. This application does not impose specific limitations on this.

[0049] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown in the diagram is for illustrative purposes only. Depending on actual implementation needs, system architecture 100 may include any number of terminal devices, networks, and servers.

[0050] All information, data and signals involved in this application have been obtained with the user's authorization or with the full authorization of the relevant parties, and the collection, use and processing of evaluation requirements, evaluation samples, model call results, execution trajectories and other related data comply with the laws, regulations and standards of the relevant countries and regions.

[0051] Continue to refer to Figure 2 , Figure 2 The flowchart S200 of the automatic model evaluation task orchestration method provided in the embodiments of this application is shown.

[0052] Figure 2 The model automatic evaluation task orchestration method shown can be applied to Figure 1 The terminal device or server shown. This process S200 includes at least the following steps S201-S206.

[0053] Step S201: Obtain the evaluation task specification for the current evaluation task.

[0054] In this embodiment, the current evaluation task can be an evaluation processing task initiated for the agent under test. The evaluation task specification is used to describe the agent under test and the evaluation process required to perform the evaluation on the agent under test in a structured form.

[0055] In some optional implementations, the evaluation task specification includes configuration information for the agent under test and configuration information for the evaluation process. The configuration information for the agent under test identifies the agent under test and its calling environment. The configuration information for the evaluation process determines the processing objects and execution parameters of each evaluation task node, and may include, for example, evaluation data configuration, calling configuration, evaluation configuration, and result output configuration.

[0056] Evaluation task specifications can be implemented using configuration files, key-value pair collections, or other data structures that can be parsed by computers.

[0057] In some embodiments, such as Figure 3 As shown, step S201 may include the following steps S2011-S2015.

[0058] Step S2011: Obtain the evaluation requirements for the tested intelligent agent.

[0059] In this embodiment, evaluation requirements can be submitted by users through terminal devices, or triggered by application programming interfaces (APIs), scheduled tasks, or development pipelines. Evaluation requirements can take the form of natural language text, forms, or structured documents, and are used to describe the intelligent agent to be evaluated and the evaluation conditions required for this evaluation.

[0060] Step S2012: Based on the evaluation requirements, analyze the configuration information of the tested intelligent agent and the evaluation process configuration information to obtain the task configuration information.

[0061] Specifically, the evaluation requirements can be analyzed by extracting fields or performing semantic parsing to identify the configuration content used to determine the agent under test, thus obtaining the agent under test configuration information. Furthermore, the configuration content used to control the processing of evaluation data, the invocation of the agent under test, the evaluation of invocation results, and the output of evaluation results can be identified to obtain the evaluation process configuration information. The agent under test configuration information and the evaluation process configuration information are then correlated to obtain the task configuration information corresponding to the current evaluation task.

[0062] Step S2013: Perform integrity verification on the task configuration information, complete any missing configuration items in the task configuration information or identify the missing configuration items as configuration items to be confirmed, and identify any conflicting configuration items in the task configuration information as configuration items to be confirmed.

[0063] Integrity checks can be used to determine whether all configuration items required to generate the evaluation task specifications are complete. For missing configuration items in the task configuration information, they can be supplemented based on preset configurations or stored configurations. If supplementation is impossible or requires user confirmation, the missing configuration item is designated as a pending confirmation item. For configuration items with multiple inconsistent values ​​or mutual constraints in the task configuration information, conflicting configuration items are designated as pending confirmation items.

[0064] Step S2014: If the task configuration information meets the integrity requirements and there are no configuration items to be confirmed, generate the evaluation task specification based on the task configuration information.

[0065] Specifically, the configuration information of the tested intelligent agent and the evaluation process configuration information can be organized according to a preset standardized data structure to obtain an evaluation task specification that can be read by the subsequent evaluation task orchestration process.

[0066] In step S2015, if there are configuration items to be confirmed, output the configuration items to be confirmed, and in response to receiving confirmation information for the configuration items to be confirmed, update the task configuration information according to the confirmation information, and generate the evaluation task specification according to the updated task configuration information.

[0067] For example, the configuration items to be confirmed and their corresponding candidate values ​​can be sent to the terminal device. The user can select a candidate value or enter a corrected value, and the terminal device will return confirmation information accordingly. Based on the confirmation information, the corresponding configuration items are updated, and the evaluation task specification is generated after the updated task configuration information meets the integrity requirements.

[0068] Therefore, by analyzing, verifying, and confirming evaluation requirements in different forms, evaluation requirements can be transformed into evaluation task specifications with a unified structure and complete configuration, providing standardized input for the subsequent automatic generation of evaluation task dependency graphs.

[0069] Step S202: Generate an evaluation task dependency graph according to the evaluation task specification.

[0070] In this embodiment, the evaluation task dependency graph is used to represent the data transfer relationships and execution dependencies between the evaluation task nodes in the current evaluation task. The evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependency relationships, and the multiple evaluation task nodes include at least a node for the agent under test to call and a node for evaluating the call result.

[0071] In this embodiment, based on the evaluation process configuration information, at least one of the following can be set in the evaluation task dependency graph: evaluation data processing node, evaluation result generation node, and reflection decision generation node.

[0072] Specifically, the evaluation process configuration information can be read from the evaluation task specification. Based on the evaluation process configuration information, the required evaluation task nodes for this evaluation can be determined, along with the inputs, outputs, processing objects, and execution parameters of each evaluation task node. For any two evaluation task nodes, when the evaluation product generated by one evaluation task node is used as the input of another evaluation task node, a connection relationship is established between the two evaluation task nodes, pointing from the former evaluation task node to the latter, thereby generating an evaluation task dependency graph.

[0073] For example, the calling node of the agent under test can invoke the agent under test based on the evaluation set and generate a model invocation result, while the invocation result evaluation node uses this model invocation result as input. Therefore, an input-output dependency can be established between the calling node of the agent under test and the invocation result evaluation node. If the evaluation task dependency graph also includes an evaluation data processing node, an input-output dependency can be established between the evaluation data processing node and the calling node of the agent under test based on the relationship between the evaluation data processing node outputting the evaluation set and the calling node of the agent under test using the evaluation set.

[0074] The evaluation task dependency graph can be implemented using a directed acyclic graph or other data structures that can represent node dependencies. By generating the evaluation task dependency graph, the previously disjointed evaluation processing steps can be organized into an evaluation flow with clear input-output relationships. This allows subsequent steps to determine the evaluation task nodes affected by changes in evaluation conditions based on node dependencies and to run the determined evaluation task nodes in the corresponding execution order.

[0075] Step S203: Query the reference evaluation task information corresponding to the current evaluation task to obtain the query results.

[0076] In this embodiment, the reference evaluation task information can be information formed by storing historical evaluation tasks, which is used to describe the evaluation conditions, execution status and evaluation products that can be referenced by the current evaluation task.

[0077] The reference evaluation task information may include at least one of the following: task configuration information of historical evaluation tasks, execution status of each evaluation task node, existing evaluation products generated by each evaluation task node, and reference evaluation results. It can also be associated with the corresponding historical evaluation tasks through index information such as task identifier and the identifier of the tested intelligent agent.

[0078] Specifically, matching information for querying reference evaluation task information can be extracted from the evaluation task specification of the current evaluation task, and a query can be performed in the evaluation task database or evaluation knowledge base based on this matching information. The matching information may include at least one of the following: the identifier of the agent under test, the evaluation task type, the service space, the evaluation scenario, and the specified reference task identifier.

[0079] If one or more historical evaluation task information that meets the matching conditions is found, the historical evaluation task information that matches the current evaluation task to a certain degree with the preset conditions can be identified as reference evaluation task information, and query results will be generated to indicate that reference evaluation task information has been found.

[0080] The query results may also include the storage location or information identifier of the reference evaluation task information, so that the recorded task configuration information and its associated existing evaluation products can be retrieved later. If no historical evaluation task information that meets the matching conditions is found, query results indicating that no reference evaluation task information was found are generated.

[0081] By querying reference evaluation task information, it can be determined whether there is a historical execution basis for the current evaluation task that can be compared and reused, providing a basis for subsequent identification of changes in evaluation conditions and determination of whether existing evaluation products can be reused.

[0082] Step S204: Determine the changes in evaluation conditions based on the evaluation task specifications and query results.

[0083] In this embodiment, the evaluation condition change information is used to indicate the configuration items that have changed in the current evaluation task relative to the reference evaluation task, and the evaluation task nodes affected by the changed configuration items.

[0084] In some embodiments, the association between configuration items and evaluation task nodes can be pre-established. This association can be determined based on the input, processing object, and execution parameters of each evaluation task node, and is used to indicate the evaluation task nodes that may be affected when the configuration items change.

[0085] In some implementations, the node configuration information of each evaluation task node can record the configuration items that the evaluation task node depends on. After determining the configuration items that have changed, the evaluation task nodes whose node configuration information records the configuration items can be identified as the directly affected evaluation task nodes. Then, along the output direction of the evaluation task dependency graph, the downstream evaluation task nodes that use the output products of the directly affected evaluation task nodes or other affected nodes as inputs can be identified as the affected evaluation task nodes.

[0086] In some embodiments, such as Figure 4 As shown, step S204 may include the following steps S2041-S2042.

[0087] Step S2041: If the query result indicates that reference evaluation task information has been found, compare the task configuration information corresponding to the current evaluation task with the task configuration information recorded in the reference evaluation task information, determine the configuration items that have changed, determine the evaluation task nodes affected by the changed configuration items, and generate evaluation condition change information based on the evaluation task nodes affected by the changes.

[0088] Specifically, the task configuration information corresponding to the current evaluation task can be aligned with the task configuration information recorded in the reference evaluation task information according to the type or identifier of the configuration item, and the corresponding configuration items can be compared. If the value, version information, or consistency verification information of the configuration item is different, it indicates that the configuration item has changed; configuration items added or deleted in the current evaluation task can also be identified as changed configuration items.

[0089] After identifying the changed configuration item, the evaluation task nodes whose inputs, processing objects, or execution parameters are affected by the configuration item can be identified as the affected evaluation task nodes based on the association between the configuration item and the evaluation task nodes. If a change in the output of an evaluation task node may affect the input of downstream evaluation task nodes, the corresponding downstream evaluation task nodes can also be identified as affected evaluation task nodes according to the input-output dependencies in the evaluation task dependency graph.

[0090] For example, when a changed configuration option is used to determine the agent under test invoked by the invoker node, the invoker node can be identified as an affected evaluation task node. Since the model invocation result generated by the invoker node serves as input to the invocation result evaluation node, the invocation result evaluation node can also be identified as an affected evaluation task node.

[0091] Step S2042: If the query results indicate that no reference evaluation task information is found, generate evaluation condition change information to indicate that multiple evaluation task nodes are affected.

[0092] When no reference evaluation task information is found, there is no historical execution basis for configuration comparison with the current evaluation task, and it is impossible to determine whether the evaluation task nodes in the evaluation task dependency graph have existing evaluation products that can be directly reused. Therefore, multiple evaluation task nodes in the evaluation task dependency graph can be identified as affected evaluation task nodes, so that the current evaluation task is executed according to the complete evaluation process.

[0093] Therefore, the information on changes in evaluation conditions can convert changes at the task configuration level into the scope of impact at the evaluation task node level. This allows subsequent steps to determine the target evaluation task node that needs to be re-executed based on the scope of impact, reducing the need for redundant processing of evaluation task nodes that are not affected by changes in evaluation conditions.

[0094] Step S205: Based on the evaluation condition change information, the affected evaluation task nodes are determined as target evaluation task nodes. If the query results indicate that reference evaluation task information is found and there are unaffected evaluation task nodes, the existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes are reused.

[0095] In this embodiment, the target evaluation task node is the evaluation task node that needs to be executed or re-executed in the current evaluation task. Existing evaluation products are data generated and stored from evaluation task nodes in historical evaluation tasks, and may include at least one of evaluation sets, model call results, evaluation record sets, and evaluation results. Existing evaluation products can be stored in association with the evaluation task node that generated them and the corresponding reference evaluation task information.

[0096] Specifically, based on the node identifiers recorded in the evaluation condition change information, the affected evaluation task nodes can be identified from the evaluation task dependency graph, and these affected evaluation task nodes can be designated as target evaluation task nodes. If the evaluation condition change information indicates that multiple evaluation task nodes are affected, all of these nodes in the evaluation task dependency graph can be designated as target evaluation task nodes.

[0097] If the query results indicate that reference evaluation task information is found, multiple evaluation task nodes in the evaluation task dependency graph can be compared with the target evaluation task node. Evaluation task nodes that do not belong to the target evaluation task node are identified as unaffected evaluation task nodes. Based on the node identifier of the unaffected evaluation task nodes, existing evaluation products generated by the corresponding evaluation task nodes can be found from the existing evaluation products associated with the reference evaluation task information.

[0098] Existing evaluation products can be associated with reference evaluation task information and the evaluation task node that generated the existing evaluation product through at least one of the following: evaluation product identifier, storage address, and version information. When reusing an existing evaluation product, the existing evaluation product can be associated with the corresponding evaluation task node in the current evaluation task, and the existing evaluation product can be configured as the input of the corresponding downstream evaluation task node in the current evaluation task, without having to re-execute the evaluation task node that generated the existing evaluation product.

[0099] For example, if the evaluation data processing node is not affected by changes in evaluation conditions, but the called node of the tested agent is affected, an existing evaluation set generated by the evaluation data processing node and associated with the reference evaluation task information can be obtained, and this existing evaluation set can be used as the input of the called node of the tested agent in the current evaluation task, so that the evaluation data processing node does not need to be re-executed.

[0100] If the query results indicate that no reference evaluation task information was found, since there is no reference evaluation task information that can be used as a basis for reuse, multiple evaluation task nodes in the evaluation task dependency graph can be identified as target evaluation task nodes based on the evaluation condition change information, and the reuse of existing evaluation products will not be performed. If the query results indicate that reference evaluation task information was found but no unaffected evaluation task nodes are found, the reuse of existing evaluation products may also not be performed.

[0101] By determining the target evaluation task nodes that need to be re-executed based on the information of changes in evaluation conditions, the range of changes in evaluation conditions can be converted into the execution range of evaluation task nodes. By reusing the existing evaluation products corresponding to the unaffected evaluation task nodes, the repeated execution of operations such as evaluation data processing, calling the tested intelligent agent, and evaluating the calling results can be reduced, and the reused existing evaluation products can be used as inputs for subsequent target evaluation task nodes.

[0102] Step S206: Execute the target evaluation task node according to the evaluation task dependency graph, and obtain and output the current evaluation result based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

[0103] Specifically, the execution order of each target evaluation task node can be determined according to the input-output dependencies in the evaluation task dependency graph. When the input of a target evaluation task node is generated by other target evaluation task nodes, the evaluation product generated by the corresponding upstream target evaluation task node is obtained as the input of the current target evaluation task node after the upstream target evaluation task node has completed its execution; when the input of a target evaluation task node corresponds to a reused existing evaluation product, the existing evaluation product is directly read. After the target evaluation task node has completed its execution, it can store the evaluation product it generated and provide the evaluation product to the corresponding downstream target evaluation task node.

[0104] The target evaluation task node may include at least one of the following: evaluation data processing node, tested agent invocation node, invocation result evaluation node, evaluation result generation node, and reflection decision generation node.

[0105] The execution methods for different evaluation task nodes are explained below.

[0106] In some embodiments, the target evaluation task node includes an evaluation data processing node, and the evaluation products generated by the evaluation data processing node include an evaluation set.

[0107] When executing the evaluation data processing node, the evaluation set source parameters can be read from the evaluation task specification, and an initial evaluation set can be obtained or generated based on these parameters. If the evaluation set source parameters point to an existing evaluation set, the existing evaluation set is determined as the initial evaluation set. If the evaluation set source parameters point to a seed evaluation set, extended evaluation samples corresponding to at least one of the boundary problems, query variations, and combined scenarios can be generated based on the evaluation scenarios, scenario descriptions, and user query examples in the seed evaluation set. The initial evaluation set is then determined based on the seed evaluation set and the extended evaluation samples.

[0108] Among them, boundary problems can be user queries generated based on the boundary conditions of the evaluation scenario; query variants can be user queries obtained by changing the expression, query conditions, or query content of user query examples; and combined scenarios can be evaluation scenarios formed by combining multiple evaluation scenarios or multiple query conditions. By generating extended evaluation samples based on the seed evaluation set, the coverage of evaluation samples with different expression methods, boundary conditions, and scenario combinations can be increased.

[0109] After obtaining the initial evaluation set, the evaluation samples in the initial evaluation set can be cleaned, deduplicated, and quality checked to obtain the final evaluation set.

[0110] Cleaning can include processing field formats, invalid content, or abnormal characters; deduplication can be used to delete duplicate evaluation samples with the same or similar content that meet preset conditions; quality verification can be used to determine whether the necessary fields in the evaluation samples are complete and whether the field content meets the corresponding data requirements.

[0111] By parsing the fields of each evaluation sample in the evaluation set, the user query and at least one of the expected execution steps, invocation constraints, and result assertions associated with the user query can be obtained. The fields corresponding to the user query are mapped to the input fields required for the tested agent to call the interface, enabling the tested agent calling node to invoke the tested agent based on the mapped input fields. At least one of the expected execution steps, invocation constraints, and result assertions associated with the user query is determined as the evaluation criterion for evaluating the model invocation results generated by the tested agent in response to the user query, for use by the invocation result evaluation node.

[0112] In some embodiments, the target evaluation task node includes the node invoked by the tested intelligent agent.

[0113] When executing the node call of the agent under test, the business space corresponding to the agent under test and the evaluation set used to call the agent under test can be obtained, and the template call can be queried based on the business space and the agent under test.

[0114] The model call template is used to describe the interface call relationships, request parameters, and parameter mapping relationships required to call the tested intelligent agent.

[0115] The business space is used to characterize the business environment or call domain to which the tested intelligent agent belongs, and can be used together with the tested intelligent agent identifier to query the corresponding model call template.

[0116] If an existing model call template corresponding to the business space and the agent under test is found, the existing model call template is determined as the target model call template. If no existing model call template is found, the call chain document or call log corresponding to the agent under test is obtained. The interface call relationship and request parameters are parsed from the call chain document or call log. A model call template is generated based on the parsing results, and the generated model call template is determined as the target model call template.

[0117] Based on the target model call template, debug call flow and full call flow can be generated respectively.

[0118] The debugging call process is used to verify, using some evaluation samples, whether the target model can call the template and evaluation task parameters to call the tested intelligent agent normally.

[0119] The full call process is used to execute the calls of the tested agent required for the current evaluation task using the evaluation samples in the evaluation set.

[0120] Select a portion of the evaluation samples from the evaluation set, call the tested intelligent agent according to the debugging and calling process, and set the current evaluation task to a pending confirmation state.

[0121] The call status and model call results generated during debugging can be output so that users or other confirmation entities can determine whether to continue execution.

[0122] Upon receiving the continuation execution message, the system calls the tested agent using the evaluation set according to the full call process, and obtains the model call results and corresponding execution trajectory identifiers generated by the tested agent for each evaluation sample.

[0123] Among them, the execution trajectory identifier is used to locate the execution trajectory generated when the tested intelligent agent processes the corresponding evaluation sample.

[0124] In some optional implementations, when calling the tested agent according to the full call process, the call can be initiated according to a preset number of concurrent calls, or at a call rate not exceeding a preset limit on the number of queries per minute, so that the call process is adapted to the interface processing capability of the tested agent.

[0125] Upon receiving correction information, the corresponding evaluation task parameters can be updated based on the correction information, and the evaluation task node using the updated evaluation task parameters can be re-executed.

[0126] For example, when the request parameters in the model call template need to be adjusted, the corresponding request parameters can be updated, and the call node of the tested agent can be re-executed.

[0127] In some embodiments, the target evaluation task node includes a call result evaluation node.

[0128] When executing the evaluation node for the call result, the system obtains the evaluation samples used by the called node of the tested agent, the model call result output by the called node of the tested agent, and the execution trajectory identifier corresponding to the model call result. Based on the execution trajectory identifier, the system obtains the model execution process information, tool call information, and interface execution information generated when the tested agent processes the corresponding evaluation sample. The system then associates the evaluation sample, model call result, model execution process information, tool call information, and interface execution information to generate an evaluation record.

[0129] If no existing evaluation record set exists for the current evaluation task, create an evaluation record set based on multiple evaluation records, and designate the created evaluation record set as the target evaluation record set. If an existing evaluation record set exists for the current evaluation task, update the existing evaluation record set based on multiple evaluation records to generate a new version of the evaluation record set, and designate the new version of the evaluation record set as the target evaluation record set.

[0130] The existing evaluator is read according to the evaluation task specification and the existing evaluator is determined as the target evaluator, or an evaluator is generated according to the evaluation requirement document and evaluation rules corresponding to the current evaluation task and the generated evaluator is determined as the target evaluator.

[0131] When generating an evaluator based on the evaluation requirements document and evaluation rules, the evaluation object, the input fields of the evaluation record, the evaluation judgment conditions and the evaluation result output format can be extracted from the evaluation requirements document and evaluation rules. Based on the extraction results, the evaluator configuration or evaluation prompt information can be generated, and the target evaluator can be generated based on the evaluator configuration or evaluation prompt information.

[0132] Determine the evaluator identifier and evaluator version of the target evaluator in order to record the specific evaluator used in this evaluation.

[0133] The evaluation requirements document can be a document that records the evaluation objectives, evaluation objects, and evaluation requirements of the current evaluation task. Evaluation rules can be used to specify the conditions or processing methods for judging the model call results.

[0134] The target evaluation record set, evaluator identifier, and evaluator version are submitted as an evaluation task. The running status of the evaluation task is polled until the evaluation task reaches the completion state, and the evaluation status of each evaluation record in the target evaluation record set is obtained. Based on the evaluation status of each evaluation record, the evaluation records that have successfully completed the evaluation and those that have not successfully completed the evaluation are determined, and the evaluation results corresponding to the evaluation records that have successfully completed the evaluation are obtained.

[0135] If no evaluation records fail to complete, the evaluation results for the successfully completed records are set as the current evaluation result. If evaluation records fail to complete, the target evaluator is used to re-evaluate these records, obtaining a retry evaluation result. The evaluation results for the successfully completed records are then combined with the retry evaluation results to obtain the current evaluation result. By re-evaluating only the records that failed to complete, redundant processing of successfully completed records is reduced.

[0136] In some implementations, if an evaluation record that failed to complete the evaluation still fails to complete the evaluation after being re-evaluated, the evaluation record can be marked as an evaluation failure, and the failure status and reason of the evaluation record can be included as part of the retry evaluation result. The failure status and reason of the failure can be recorded in the current evaluation result, or evaluation anomaly information corresponding to the evaluation record can be output.

[0137] In some embodiments, the target evaluation task node further includes an evaluation result generation node.

[0138] When executing the evaluation result generation node, the current evaluation result and the evaluation samples corresponding to the current evaluation task can be obtained, and the sample-level evaluation results corresponding to each evaluation sample can be obtained from the current evaluation result. Multiple sample-level evaluation results are aggregated to obtain the aggregated evaluation result.

[0139] The evaluation details are generated by integrating the sample-level evaluation results, aggregated evaluation results, and at least one of the expected execution steps, invocation constraints, and result assertions in the evaluation samples. Evaluation metrics are then calculated based on these metrics to characterize the execution status of the current evaluation task or the quality status of the tested agent. Finally, failed samples are identified based on the evaluation metrics and the sample-level evaluation results.

[0140] Obtain the execution trajectory and execution trajectory identifier corresponding to the failed sample, and perform failure cause analysis on the failed sample based on the execution trajectory to obtain the failure analysis results.

[0141] Specifically, failure analysis can be performed on failed samples based on the model execution process, tool call process, or interface execution process recorded in the execution trajectory, and the failure analysis results can be obtained.

[0142] In some implementations, the error type, severity, and failure node corresponding to the failed sample can be determined based on the execution trajectory, and the execution trajectories corresponding to multiple failed samples can be comprehensively analyzed to determine the common failure reasons of multiple failed samples.

[0143] An evaluation report is generated based on the evaluation details, evaluation indicators, and failure analysis results. The relationship between the failure analysis results and the corresponding execution trajectory identifiers is established so that the execution process information supporting the failure analysis results can be queried through the execution trajectory identifiers.

[0144] The quality access control conclusion is generated based on the evaluation indicators and preset quality judgment conditions.

[0145] The quality access control conclusion can be used to indicate whether the tested agent corresponding to the current evaluation task meets the preset quality requirements, and the specific evaluation indicators and quality judgment conditions can be set according to the evaluation needs.

[0146] In some embodiments, the target evaluation task node further includes a reflection decision generation node.

[0147] When executing the reflection decision generation node, evaluation anomaly information can be determined based on at least one of the current evaluation results, failure samples, evaluation indicators, and quality gate conclusions.

[0148] Evaluation anomaly information is used to indicate abnormal situations in evaluation results, evaluation samples, or evaluation indicators that require further analysis.

[0149] Obtain the execution trajectory corresponding to the evaluation anomaly information and the historical evaluation information corresponding to the tested agent. Based on the evaluation anomaly information, execution trajectory and historical evaluation information, determine the failure type and failure root cause.

[0150] Among them, the execution trajectory is used to provide execution information of each processing step in this evaluation process, and the historical evaluation information is used to provide the evaluation status of the tested intelligent agent in historical evaluation tasks.

[0151] Failure types can include at least one of the following: incorrect response from the tested agent, misjudgment by the evaluator, insufficient coverage of the evaluation scenario, and abnormal model invocation chain. Improvement targets can include at least one of the following: evaluation set, evaluator, tested agent configuration, and model invocation template. Different failure types can correspond to different improvement targets.

[0152] Specifically, failure types can be distinguished based on model call results, evaluation criteria, and execution trajectories. For example, if the model execution process and call chain are normal, but the model call result does not satisfy the result assertion, the failure type can be identified as an incorrect response from the tested agent; if the model call result satisfies the evaluation criteria, but the evaluator outputs a failure result, the failure type can be identified as an evaluator misjudgment; if the failure sample set contains scenario features not covered by existing evaluation scenarios, the failure type can be identified as insufficient coverage of evaluation scenarios; and if the execution trajectory contains interface call failures, abnormal request parameter mappings, or missing execution trajectories, the failure type can be identified as an abnormal model call chain.

[0153] Based on the failure type, identify the corresponding improvement targets, generate failure evidence based on the root cause and execution trajectory, and generate reflective decisions including failure evidence, root cause, and improvement actions for the improvement targets. Write the reflective decisions into the evaluation knowledge base.

[0154] The reflective decisions stored in the evaluation knowledge base can be queried and used in subsequent evaluation tasks, evaluation configuration adjustments, or the improvement process of the tested intelligent agent.

[0155] By executing the aforementioned target evaluation task nodes according to the evaluation task dependency graph, each evaluation task node can deliver evaluation products based on explicit input-output dependencies. Expanding the evaluation sample improves coverage of boundary problems, query variations, and combined scenarios; associating model call results with execution process information provides a basis for failure cause analysis; and generating reflective decisions and writing them into the evaluation knowledge base allows for the continuous accumulation and reuse of failure causes and improvement actions obtained during the evaluation process.

[0156] After the target evaluation task nodes are completed, the evaluation products generated by each target evaluation task node can be obtained. Based on the evaluation task dependency graph, the evaluation products generated by the upstream target evaluation task nodes are passed to the corresponding downstream target evaluation task nodes. For evaluation task nodes that are not affected by changes in evaluation conditions, the existing evaluation products corresponding to the evaluation task node determined in step S205 can be obtained. The existing evaluation products can be used as input to the corresponding downstream target evaluation task node or as the basis for generating the current evaluation result.

[0157] If no reference evaluation task information is found, or if multiple evaluation task nodes are affected, the current evaluation result can be obtained based on the newly generated evaluation products from executing multiple target evaluation task nodes. If reference evaluation task information is found and only some evaluation task nodes are affected, the newly generated evaluation products from the target evaluation task nodes can be combined with the existing evaluation products corresponding to the unaffected evaluation task nodes according to the input-output relationship in the evaluation task dependency graph, and the current evaluation result can be obtained based on the combined evaluation product. If no evaluation task nodes need to be re-executed, the current evaluation result can also be obtained based on the reused existing evaluation products.

[0158] For example, when both the agent under test calling node and the calling result evaluation node are target evaluation task nodes, the calling result evaluation node can be executed based on the newly generated model calling result from the agent under test calling node, and the current evaluation result can be obtained based on the newly generated evaluation result from the calling result evaluation node. When the evaluation data processing node is unaffected, but the agent under test calling node and the calling result evaluation node are affected, the existing reused evaluation set can be used as input to the agent under test calling node, and the current evaluation result can be obtained based on the newly generated model calling result and evaluation result.

[0159] The current evaluation results can include sample-level evaluation results and aggregated evaluation results obtained by aggregating sample-level evaluation results. When executing the evaluation result generation node, the output can also include at least one of the following: evaluation details, evaluation metrics, failure analysis results, evaluation report, and quality gate conclusions generated based on the current evaluation results; when executing the reflection decision generation node, it can also output reflection decisions corresponding to the current evaluation task. The current evaluation results and their associated outputs can be sent to terminal devices or stored in the evaluation task database or evaluation knowledge base.

[0160] By executing the target evaluation task node according to the evaluation task dependency graph and connecting the newly generated evaluation products of the target evaluation task node with the reused existing evaluation products, the current evaluation result can be obtained while ensuring the integrity of the input-output relationship between the evaluation task nodes, thereby reducing the repeated execution of evaluation task nodes that are not affected by changes in evaluation conditions.

[0161] In some embodiments, if the query results indicate that reference evaluation task information has been found, version comparison can also be performed based on the evaluation image, which may specifically include the following A1-A4.

[0162] A1 generates or loads the evaluation image specification based on the reference evaluation task information.

[0163] The evaluation mirror specification records the non-tested evaluation conditions associated with the reference evaluation task information, excluding the tested version parameters and the environment parameters used to locate the tested version, as well as the version information or consistency verification information corresponding to the non-tested evaluation conditions. Non-tested evaluation conditions are evaluation conditions other than the tested version parameters and environment parameters that may affect the execution process or evaluation results of the evaluation task.

[0164] In some implementations, evaluation conditions and their version information or consistency verification information associated with the reference evaluation task information can be read, and evaluation image specifications can be generated based on the read information. If the corresponding evaluation image specifications have already been stored, the corresponding evaluation image specifications can also be loaded based on the image specification identifier or storage address in the reference evaluation task information.

[0165] A2. According to the evaluation mirror specification, perform consistency verification on the non-tested evaluation conditions corresponding to the current evaluation task.

[0166] Specifically, the version information of the non-tested evaluation conditions corresponding to the current evaluation task can be compared with the corresponding version information recorded in the evaluation image specification; or, consistency verification information can be generated based on the non-tested evaluation conditions corresponding to the current evaluation task, and the generated consistency verification information can be compared with the corresponding information recorded in the evaluation image specification to determine whether each non-tested evaluation condition corresponding to the current evaluation task has passed the consistency verification.

[0167] A3. If the non-tested evaluation conditions pass the consistency check, and the evaluation condition change information indicates that the changed evaluation conditions only include at least one of the tested version parameters and environment parameters, then perform a version comparison based on the evaluation mirror specification.

[0168] Specifically, the non-tested evaluation conditions recorded in the evaluation mirror specification are determined as the non-tested evaluation conditions of the current evaluation task, and at least one of the changed tested version parameters and environment parameters associated with the reference evaluation task information is replaced with the corresponding parameters in the current evaluation task to obtain the replaced evaluation conditions.

[0169] The target evaluation task node is determined based on the information about changes in evaluation conditions. If there are unaffected evaluation task nodes, existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes are reused. According to the evaluation task dependency graph, the target evaluation task node is executed based on the replaced evaluation conditions, and the current evaluation result is obtained based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

[0170] Obtain the reference evaluation results associated with the reference evaluation task information, compare the current evaluation results with the reference evaluation results, determine the evaluation differences between different tested versions based on the comparison results, and output the version comparison results.

[0171] A4. If at least one non-tested evaluation condition fails the consistency check, execute the evaluation task according to the evaluation condition corresponding to the current evaluation task.

[0172] Specifically, based on the non-tested evaluation conditions that failed the consistency check, the evaluation task nodes whose inputs, processing objects, or execution parameters were affected are identified, and the evaluation condition change information is updated according to the identified evaluation task nodes. The target evaluation task node is then re-determined based on the updated evaluation condition change information, and executed according to the evaluation task dependency graph and the evaluation conditions corresponding to the current evaluation task.

[0173] For nodes that are re-identified as target evaluation task nodes because they are not part of the evaluation criteria and fail the consistency check, the existing evaluation products corresponding to those nodes are not reused. Based on at least one of the evaluation products generated by the re-identified target evaluation task nodes and reusable existing evaluation products, the current evaluation result is obtained and output.

[0174] Therefore, by utilizing the evaluation mirroring specification to perform consistency checks on non-tested evaluation conditions, evaluations and result comparisons between different tested versions can be performed while maintaining consistency in non-tested evaluation conditions, reducing the interference of changes in non-tested factors on version comparison results. When non-tested evaluation conditions change, by expanding the scope of evaluation task nodes that need to be re-executed, the situation where inaccurate evaluation results are caused by reusing existing evaluation products that are no longer applicable can be reduced.

[0175] In one specific embodiment, the pre-release evaluation of the used car search and push intelligent agent is used as an example to illustrate the automatic evaluation task arrangement method of the model provided in this application.

[0176] Users submit the following evaluation requests through their terminal devices: evaluate the Used Car Finder and Pusher Smart Agent V2.3, with evaluation scenarios including budgets of 100,000 to 150,000 yuan, first-time buyers, users who strongly negotiate prices, trade-in inquiries, and financial solution inquiries. The evaluation sample size is 500 entries, and the evaluation results are compared with the evaluation results of the tested Smart Agent V2.2.

[0177] The server obtains the aforementioned evaluation requirements and parses them to obtain the configuration information of the agent under test and the evaluation process configuration information. The agent under test configuration information is used to determine the agent under test V2.3 and its corresponding business space and calling environment; the evaluation process configuration information is used to determine the scope of the evaluation scenario, the number of evaluation samples, the source of the evaluation set, the model calling method, the evaluation method, the version comparison requirements, and the result output method.

[0178] The server performs a completeness check on the parsed task configuration information. If any configuration item, such as the evaluator or quality assessment criteria, is missing and cannot be completed based on the preset configuration, the corresponding configuration item is identified as a pending confirmation item and output. After the user confirms the pending confirmation item via their terminal device, the server updates the task configuration information based on the confirmation and generates the evaluation task specification for the current evaluation task based on the updated task configuration information.

[0179] The server generates an evaluation task dependency graph according to the evaluation task specifications. In this embodiment, the evaluation task dependency graph sequentially includes an evaluation data processing node, a tested agent invocation node, an invocation result evaluation node, an evaluation result generation node, and a reflective decision generation node. The evaluation set output by the evaluation data processing node serves as the input to the tested agent invocation node; the model invocation result and execution trajectory identifier output by the tested agent invocation node serve as the input to the invocation result evaluation node; the current evaluation result output by the invocation result evaluation node serves as the input to the evaluation result generation node; and at least one of the failure samples, evaluation metrics, and quality gate conclusions output by the evaluation result generation node serves as the input to the reflective decision generation node.

[0180] The server queries reference evaluation task information corresponding to the current evaluation task based on the identifier of the agent under test, the service space, and the version comparison requirements. In this embodiment, the reference evaluation task information corresponding to agent V2.2 under test is retrieved. The reference evaluation task information records the task configuration information corresponding to V2.2, the existing evaluation products generated by each evaluation task node, and the reference evaluation results.

[0181] The server compares the task configuration information corresponding to the current evaluation task with the task configuration information recorded in the reference evaluation task information. Assuming the comparison results indicate that only the parameters of the tested version and the environmental parameters used to locate the tested version have changed, while non-tested evaluation conditions such as the evaluation set, evaluator, model invocation template, and quality judgment conditions remain unchanged, the calling node of the tested agent, along with the evaluation nodes for invocation results dependent on its output, the evaluation result generation nodes, and the reflective decision generation nodes, are identified as the affected evaluation task nodes and designated as the target evaluation task nodes. Since the evaluation data processing nodes are unaffected, the server reuses the existing evaluation set generated by the evaluation data processing nodes, which is associated with the reference evaluation task information.

[0182] During version comparison, the server generates or loads the evaluation image specification based on the reference evaluation task information, and performs consistency verification on the non-tested evaluation conditions corresponding to the current evaluation task according to the evaluation image specification. If the non-tested evaluation conditions pass the consistency verification, the server determines the non-tested evaluation conditions recorded in the evaluation image specification as the non-tested evaluation conditions of the current evaluation task, and replaces the tested version parameters and environment parameters with the parameters corresponding to the tested agent V2.3, thus obtaining the replaced evaluation conditions.

[0183] The server queries the model invocation template based on the business space and the agent under test. If an existing model invocation template corresponding to the business space and the agent under test is found, the existing model invocation template is determined as the target model invocation template, and a debug invocation process and a full invocation process are generated based on the target model invocation template.

[0184] The server selects a subset of evaluation samples from a reused existing evaluation set, invokes the tested agent V2.3 according to the debug invocation process, and sets the current evaluation task to a pending confirmation state. The server outputs the invocation status and model invocation results generated by the debug invocation to the terminal device. When the user confirms that the invocation results and parameter mappings meet expectations, the terminal device returns a continue execution message. The server then invokes the tested agent V2.3 using 500 evaluation samples according to the full invocation process, obtaining the model invocation results and execution trajectory identifiers corresponding to each evaluation sample.

[0185] If the debugging results indicate an anomaly in the request parameters or field mapping, the user can return correction information via their terminal device. The server updates the corresponding evaluation task parameters based on this correction information and re-executes the tested agent's calling node using the updated evaluation task parameters until it receives a continue execution request.

[0186] Based on each execution trajectory identifier, the server obtains the model execution process information, tool call information, and interface execution information generated by the tested agent V2.3 when processing the corresponding evaluation samples. It then associates the evaluation samples, model call results, and corresponding execution process information to generate evaluation records. A target evaluation record set is created based on multiple evaluation records, or an existing evaluation record set is updated and the new version is designated as the target evaluation record set.

[0187] The server determines the evaluator identifier and evaluator version of the target evaluator according to the evaluation task specifications, submits the target evaluation record set, evaluator identifier, and evaluator version as an evaluation task, and polls the running status of the evaluation task. After the evaluation task reaches the completion state, the server retrieves the evaluation results corresponding to the evaluation records that have been successfully evaluated based on the evaluation status of each evaluation record. For evaluation records that have not been successfully evaluated, the server re-evaluates them using the target evaluator and merges the retry evaluation results with the evaluation results of the successfully evaluated records to obtain the current evaluation result.

[0188] The server retrieves the sample-level evaluation results corresponding to each evaluation sample from the current evaluation results, and aggregates multiple sample-level evaluation results to obtain the aggregated evaluation result. The sample-level evaluation results, the aggregated evaluation result, and the evaluation criteria corresponding to each evaluation sample are integrated to generate evaluation details. Based on the evaluation details, evaluation indicators are calculated and failed samples are identified.

[0189] For failed samples, the server analyzes the execution process, tool call process, and interface execution process based on the corresponding execution trajectory analysis model to obtain failure analysis results. An evaluation report is generated based on the evaluation details, evaluation indicators, and failure analysis results, and a quality access control conclusion is generated based on the evaluation indicators and preset quality judgment conditions.

[0190] The server determines evaluation anomaly information based on at least one of the following: current evaluation results, failure samples, evaluation metrics, and quality gate conclusions. Combining this with the corresponding execution trajectory and the tested agent's historical evaluation information, it determines the failure type and root cause. Based on the failure type, it identifies improvement targets, generates failure evidence based on the root cause and execution trajectory, and produces a reflective decision including the failure evidence, root cause, and improvement actions. The server writes the reflective decision into the evaluation knowledge base for subsequent evaluation tasks or the tested agent's improvement process to query and use.

[0191] The server compares the current evaluation result of the tested agent V2.3 with the reference evaluation result of the tested agent V2.2 associated with the reference evaluation task information, and generates and outputs the version comparison result. The version comparison result, evaluation report, quality gate conclusion, and reflection decision can be sent to the terminal device and can be stored in association with the current evaluation task.

[0192] Through the above embodiments, when the evaluation set and other non-testing evaluation conditions remain unchanged, existing evaluation sets can be reused, and only the evaluation task nodes affected by changes in the tested version can be re-executed, thereby reducing redundant processing of evaluation data. Simultaneously, maintaining consistency of non-testing evaluation conditions through evaluation mirroring can reduce the interference of changes in evaluation conditions on version comparison results. By associating failure samples, execution trajectories, and historical evaluation information and generating reflective decisions, the reasons for failures and improvement actions obtained during the evaluation process can be stored in the evaluation knowledge base.

[0193] In another embodiment, when no reference evaluation task information corresponding to the current evaluation task is found, multiple evaluation task nodes in the evaluation task dependency graph can be determined as target evaluation task nodes, existing evaluation products are not reused, and multiple target evaluation task nodes are executed according to the evaluation task dependency graph to complete the first evaluation.

[0194] In another embodiment, when reference evaluation task information is found, but the evaluation set version, evaluator version, or other non-tested evaluation conditions fail the consistency check, the evaluation condition change information can be updated based on the non-tested evaluation conditions that failed the consistency check. This expands the range of affected evaluation task nodes, and the target evaluation task node is redefined. For nodes redefined as target evaluation task nodes, their corresponding existing evaluation products are not reused to reduce inaccurate evaluation results caused by reusing no longer applicable evaluation products.

[0195] In another embodiment, when the evaluation conditions of the current evaluation task have not changed relative to the evaluation conditions recorded in the reference evaluation task information, the evaluation task node can be obtained and output directly based on the existing evaluation products associated with the reference evaluation task information without re-executing the evaluation task node.

[0196] The automatic model evaluation task orchestration method provided in this application first obtains the evaluation task specification for the current evaluation task and generates an evaluation task dependency graph based on the specification. The evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependencies, and these nodes include at least a node for the agent under test to invoke the evaluation task and a node for evaluating the invocation result. Then, it queries reference evaluation task information corresponding to the current evaluation task, obtains the query results, and determines evaluation condition change information based on the evaluation task specification and the query results. Next, based on the evaluation condition change information, it identifies the affected evaluation task nodes as target evaluation task nodes. If reference evaluation task information is found and unaffected evaluation task nodes exist, it reuses existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes. Finally, it executes the target evaluation task nodes according to the evaluation task dependency graph and obtains and outputs the current evaluation result based on at least one of the evaluation products generated by the target evaluation task nodes and the reused existing evaluation products.

[0197] This application generates an evaluation task dependency graph based on the evaluation task specification and connects different evaluation task nodes using input-output dependencies. This organizes interconnected processing steps such as calling the tested agent and evaluating the call results into a unified evaluation execution flow, reducing the fragmentation of different evaluation processing steps and the reliance on manual connection. By determining the evaluation condition change information based on the query results of the evaluation task specification and reference evaluation task information, and identifying the affected evaluation task nodes based on the evaluation condition change information, the execution scope of the current evaluation task can be adapted to the actual changes in the evaluation conditions. This reduces the need to re-execute all evaluation processing steps due to changes in some evaluation conditions, improving the adaptability of the evaluation process to task changes. When reference evaluation task information is found and there are unaffected evaluation task nodes, the existing evaluation products corresponding to the unaffected evaluation task nodes are reused. This reduces the need for repeated data processing, tested agent calling, and result evaluation of evaluation content that has not undergone substantial changes, thereby reducing the number of model interface calls and the computational resources consumed during the evaluation process. By executing the target evaluation task node according to the evaluation task dependency graph, and combining the newly generated evaluation products of the target evaluation task node with the reused existing evaluation products to obtain the current evaluation result, the current evaluation task can be completed while maintaining the data connection between each evaluation processing link, reducing the processing time added by repeated execution, thereby improving the orchestration efficiency and execution efficiency of the model evaluation task.

[0198] Further reference Figure 5 As an implementation of the methods shown in the above figures, this application provides an embodiment of a model automatic evaluation task orchestration device, which is similar to... Figure 2Corresponding to the method embodiments shown, this device can be specifically applied to various terminal devices and servers.

[0199] like Figure 5 As shown, the model automatic evaluation task orchestration device of this embodiment includes: The system includes a specification acquisition unit 501, a dependency graph generation unit 502, an information query unit 503, a change information determination unit 504, a node arrangement unit 505, and a task execution unit 506.

[0200] The specification acquisition unit 501 is used to acquire the evaluation task specification for the current evaluation task. The dependency graph generation unit 502 is used to generate an evaluation task dependency graph based on the evaluation task specification, wherein the evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependency relationships, and the multiple evaluation task nodes include at least a tested agent invocation node and an invocation result evaluation node. The information query unit 503 is used to query reference evaluation task information corresponding to the current evaluation task and obtain query results. The change information determination unit 504 is used to determine evaluation condition change information based on the evaluation task specification and the query results. The node orchestration unit 505 is used to determine the affected evaluation task nodes as target evaluation task nodes based on the evaluation condition change information, and, if the query result indicates that the reference evaluation task information has been found and there are unaffected evaluation task nodes, reuse the existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes. The task execution unit 506 is used to execute the target evaluation task node according to the evaluation task dependency graph, and obtain and output the current evaluation result based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

[0201] In this embodiment, the specific processing of the specification acquisition unit 501, dependency graph generation unit 502, information query unit 503, change information determination unit 504, node orchestration unit 505, and task execution unit 506, and the resulting technical effects, can be found in reference to [the relevant documentation]. Figure 2 The relevant descriptions of steps S201 to S206 in the corresponding embodiments will not be repeated here.

[0202] In some optional implementations, the evaluation task specification includes configuration information of the tested intelligent agent and evaluation process configuration information. The evaluation process configuration information is used to determine the processing object and execution parameters of each evaluation task node. The specification acquisition unit 501 is specifically used for: Obtain the evaluation requirements for the tested intelligent agent; Based on the evaluation requirements, the configuration information of the tested intelligent agent and the evaluation process configuration information are analyzed to obtain the task configuration information; Perform integrity verification on task configuration information, complete missing configuration items in task configuration information or identify missing configuration items as configuration items to be confirmed, and identify conflicting configuration items in task configuration information as configuration items to be confirmed. If the task configuration information meets the integrity requirements and there are no configuration items to be confirmed, the evaluation task specification is generated based on the task configuration information. If there are configuration items to be confirmed, output the configuration items to be confirmed, and in response to receiving confirmation information for the configuration items to be confirmed, update the task configuration information according to the confirmation information, and generate the evaluation task specification according to the updated task configuration information.

[0203] In some optional implementations, the change information determination unit 504 is specifically used for: If the query results indicate that reference evaluation task information is found, the task configuration information corresponding to the current evaluation task is compared with the task configuration information recorded in the reference evaluation task information to determine the configuration items that have changed. Based on the changed configuration items, the affected evaluation task nodes are determined, and evaluation condition change information is generated based on the affected evaluation task nodes. If the query results indicate that no reference evaluation task information was found, evaluation condition change information is generated to indicate that multiple evaluation task nodes are affected.

[0204] In some optional implementations, the query results indicate that reference evaluation task information has been found, and the apparatus also includes an evaluation mirror processing unit (not shown).

[0205] The evaluation image processing unit is used to: generate or load evaluation image specifications based on reference evaluation task information, wherein the evaluation image specifications record non-tested evaluation conditions associated with the reference evaluation task information, excluding the tested version parameters and environment parameters used to locate the tested version, as well as version information or consistency verification information corresponding to the non-tested evaluation conditions; perform consistency verification on the non-tested evaluation conditions corresponding to the current evaluation task according to the evaluation image specifications; and, if the non-tested evaluation conditions pass the consistency verification, and the evaluation condition change information indicates that the changed evaluation conditions only include at least one of the tested version parameters and environment parameters, determine the non-tested evaluation conditions recorded in the evaluation image specifications as non-tested evaluation conditions for the current evaluation task. The evaluation conditions involve replacing at least one of the changed parameters of the tested version and environment associated with the reference evaluation task information with the corresponding parameters in the current evaluation task. Based on the replaced evaluation conditions, the target evaluation task node is executed to obtain the current evaluation result. The current evaluation result is then compared with the reference evaluation result associated with the reference evaluation task information, and a version comparison result is output. If at least one non-tested evaluation condition fails the consistency check, the evaluation condition change information is updated based on the non-tested evaluation condition that failed the consistency check. The target evaluation task node is then redefined based on the updated evaluation condition change information, and the redefined target evaluation task node is executed based on the evaluation conditions corresponding to the current evaluation task.

[0206] In some optional implementations, the task execution unit 506 includes an evaluation data processing unit, which is used to execute evaluation data processing nodes, and the evaluation products generated by the evaluation data processing nodes include an evaluation set.

[0207] The evaluation data processing unit is specifically used for: reading the evaluation set source parameters from the evaluation task specification; determining the existing evaluation set as the initial evaluation set when the evaluation set source parameters point to an existing evaluation set; generating extended evaluation samples corresponding to at least one of boundary problems, query variants, and combined scenarios based on the evaluation scenarios, scenario descriptions, and user query examples in the seed evaluation set when the evaluation set source parameters point to a seed evaluation set, and determining the initial evaluation set based on the seed evaluation set and the extended evaluation samples; cleaning, deduplicating, and quality-checking the evaluation samples in the initial evaluation set to obtain the evaluation set; parsing the fields of each evaluation sample in the evaluation set to obtain the user query, and at least one of the expected execution steps, calling constraints, and result assertions associated with the user query; mapping the fields corresponding to the user query to the input fields required for the tested intelligent agent to call the interface, and determining at least one of the expected execution steps, calling constraints, and result assertions associated with the user query as the evaluation basis for evaluating the model call results generated by the tested intelligent agent in response to the user query.

[0208] In some optional implementations, the task execution unit 506 includes a test agent invocation unit, which is used to execute the test agent invocation node.

[0209] The tested agent invocation unit is specifically used for: obtaining the business space corresponding to the tested agent and the evaluation set used to invoke the tested agent; querying the model invocation template based on the business space and the tested agent; if an existing model invocation template corresponding to the business space and the tested agent is found, determining the existing model invocation template as the target model invocation template; if no existing model invocation template is found, obtaining the call chain document or call log corresponding to the tested agent, parsing the interface call relationship and request parameters based on the call chain document or call log, generating the model invocation template based on the parsing result, and invoking the generated model invocation template. The template is determined as the target model invocation template; a debug invocation process and a full invocation process are generated based on the target model invocation template; a portion of the evaluation samples are selected from the evaluation set, and the tested agent is invoked according to the debug invocation process, and the current evaluation task is set to a pending confirmation state; in response to receiving the continue execution information, the tested agent is invoked using the evaluation set according to the full invocation process, and the model invocation results and corresponding execution trajectory identifiers generated by the tested agent for each evaluation sample are obtained; in response to receiving the correction information, the corresponding evaluation task parameters are updated according to the correction information, and the evaluation task node with the updated evaluation task parameters is re-executed.

[0210] In some optional implementations, the task execution unit 506 includes a call result evaluation unit, which is used to execute the call result evaluation node.

[0211] The call result evaluation unit is specifically used for: obtaining the evaluation samples used by the calling node of the tested agent, the model call results output by the calling node of the tested agent, and the execution trajectory identifiers corresponding to the model call results; obtaining the model execution process information, tool call information, and interface execution information generated when the tested agent processes the corresponding evaluation samples based on the execution trajectory identifiers; associating the evaluation samples, model call results, model execution process information, tool call information, and interface execution information to generate evaluation records; if there is no existing evaluation record set corresponding to the current evaluation task, creating an evaluation record set based on multiple evaluation records and determining the created evaluation record set as the target evaluation record set; if there is an existing evaluation record set corresponding to the current evaluation task, updating the existing evaluation record set based on multiple evaluation records to generate a new evaluation record set version and determining the new evaluation record set version as the target evaluation record set; reading the existing evaluator according to the evaluation task specification and determining the existing evaluator as the target evaluator, or based on the current evaluation task specification... The evaluation requirements document and evaluation rules corresponding to the test task are used to generate an evaluator, which is then designated as the target evaluator. The evaluator identifier and version of the target evaluator are determined. The target evaluation record set, evaluator identifier, and evaluator version are submitted as an evaluation task, and the running status of the evaluation task is polled until the evaluation task reaches its completion state, obtaining the evaluation status of each evaluation record in the target evaluation record set. Based on the evaluation status of each evaluation record, the evaluation records that have successfully completed the evaluation and those that have not, are identified, and the evaluation results corresponding to the successfully completed evaluation records are obtained. If there are no unsuccessful evaluation records, the evaluation results corresponding to the successfully completed evaluation records are determined as the current evaluation result. If there are unsuccessful evaluation records, the target evaluator is used to re-evaluate the unsuccessful evaluation records to obtain the retry evaluation results. The evaluation results corresponding to the successfully completed evaluation records are then merged with the retry evaluation results to obtain the current evaluation result.

[0212] In some optional implementations, the task execution unit 506 further includes an evaluation result generation unit, which is used to execute the evaluation result generation node.

[0213] The evaluation result generation unit is specifically used for: obtaining the current evaluation result and the evaluation sample corresponding to the current evaluation task, and obtaining the sample-level evaluation result corresponding to each evaluation sample from the current evaluation result; aggregating the sample-level evaluation results to obtain the aggregated evaluation result; integrating the sample-level evaluation result, the aggregated evaluation result, and at least one of the expected execution steps, call constraints, and result assertions in the evaluation sample to generate the evaluation details; calculating the evaluation indicators based on the evaluation details, and determining the failed samples based on the evaluation indicators and the sample-level evaluation results; obtaining the execution trajectory and execution trajectory identifier corresponding to the failed sample, and performing failure cause analysis on the failed sample based on the execution trajectory to obtain the failure analysis result; generating the evaluation report based on the evaluation details, evaluation indicators, and failure analysis result, and establishing the correlation between the failure analysis result and the corresponding execution trajectory identifier; and generating the quality access control conclusion based on the evaluation indicators and preset quality judgment conditions.

[0214] The evaluation metrics may include at least one of the following: task completion rate, response completion rate, assessment task execution completion rate, score distribution, and failure sample ratio. Different evaluation tasks can select corresponding evaluation metrics and their calculation methods according to the evaluation requirements.

[0215] In some optional implementations, the task execution unit 506 further includes a reflective decision generation unit for executing reflective decision generation nodes.

[0216] The reflection decision generation unit is specifically used for: determining evaluation anomaly information based on at least one of the current evaluation results, failure samples, evaluation indicators, and quality gate conclusions; acquiring the execution trajectory corresponding to the evaluation anomaly information and the historical evaluation information corresponding to the tested agent; determining the failure type and root cause based on the evaluation anomaly information, execution trajectory, and historical evaluation information; determining the improvement target based on the failure type; generating failure evidence based on the failure root cause and execution trajectory, and generating a reflection decision including the failure evidence, failure root cause, and improvement actions for the improvement target; and writing the reflection decision into the evaluation knowledge base.

[0217] It should be noted that the implementation details and technical effects of each unit in the automatic model evaluation task orchestration device provided in the embodiments of this application can be referred to the descriptions of other embodiments in this application, and will not be repeated here.

[0218] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing the electronic devices of the present application embodiments. Figure 6 The computer system 600 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0219] In some alternative embodiments, this application also provides an electronic device including one or more processors and a storage device storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement any of the foregoing method embodiments.

[0220] Specifically, one or more processors can correspond to Figure 6 The processing device 601 shown may include a central processing unit, a graphics processing unit, an artificial intelligence accelerator, a tensor processor, or other general-purpose or special-purpose processors. Different processors may respectively undertake at least one of the following processing steps: obtaining evaluation task specifications, generating evaluation task dependency graphs, querying reference evaluation task information, determining evaluation condition change information, determining target evaluation task nodes, reusing existing evaluation products, executing target evaluation task nodes, and generating and outputting current evaluation results.

[0221] The storage device may include at least one of read-only memory (ROM) 602, random access memory (RAM) 603, and storage device 608. ROM 602 may store a boot program or a firmware program; RAM 603 may store programs and data used during the operation of the electronic device; storage device 608 may include a disk, solid-state drive, flash memory device, or other non-volatile storage device, and may store one or more programs for implementing a method for orchestrating automatic model evaluation tasks.

[0222] In some embodiments, RAM 603 or storage device 608 may also store at least one of the following data: evaluation requirements, configuration information of the agent under test, evaluation process configuration information, task configuration information, evaluation task specifications, evaluation task dependency graph, reference evaluation task information, query results, evaluation condition change information, node information of evaluation task nodes, existing evaluation products, and current evaluation results.

[0223] RAM 603 or storage device 608 may also store at least one of the following data: evaluation set, evaluation samples, model call template, model call result, execution trajectory identifier, execution trajectory, evaluation record, evaluation record set, evaluator, evaluation result, evaluation details, evaluation index, failure analysis result, evaluation report, quality gate conclusion, reflective decision, and evaluation mirror specification. Different data can be associated through task identifier, node identifier, evaluation product identifier, version information, or storage address.

[0224] like Figure 6As shown, the processing device 601, ROM 602, and RAM 603 can be connected to each other via bus 604. Input / output interface (I / O interface) 605 can also be connected to bus 604. When the electronic device is running, the processing device 601 can execute the processing corresponding to the automatic model evaluation task arrangement method according to the program stored in ROM 602 or the program loaded from storage device 608 into RAM 603.

[0225] I / O interface 605 can connect to input device 606, output device 607, storage device 608, and communication device 609. Input device 606 may include a keyboard, mouse, touch device, microphone, data access interface, or other devices used for inputting evaluation requirements, confirmation information, continuation information, correction information, and evaluation task parameters. Input device 606 can also be used to receive data such as evaluation sets, evaluation requirement documents, evaluation rules, call chain documents, and call logs.

[0226] Output device 607 may include a display, audio output device, printing device, data output interface, or other output device. Output device 607 may be used to output at least one of the following: configuration items to be confirmed, debug call results, current evaluation results, evaluation details, evaluation indicators, failure analysis results, evaluation reports, quality gate conclusions, version comparison results, and reflective decisions.

[0227] Communication device 609 enables electronic devices to conduct wired or wireless communication with terminal devices, servers, tested intelligent agent service devices, evaluation data storage devices, evaluation service devices, model service devices, evaluation knowledge bases, or other electronic devices. Electronic devices can receive evaluation requests, evaluation data, reference evaluation task information, model call results, execution trajectories, and evaluation results through communication device 609, and can provide terminal devices or other callers with current evaluation results, evaluation reports, quality gate conclusions, version comparison results, or reflective decisions.

[0228] In some embodiments, the processing device 601 may include multiple processors. One processor may perform evaluation requirement parsing, task configuration information verification, and evaluation task specification generation; another processor may perform evaluation task dependency graph generation, reference evaluation task information query, evaluation condition change information determination, and target evaluation task node determination; and yet another processor may perform at least one of the following processes: evaluation data processing, invocation of the tested agent, evaluation of invocation results, generation of evaluation results, and generation of reflective decisions.

[0229] Multiple processors can share RAM603 or have their own device memory, and transmit data such as evaluation task specifications, evaluation task dependency graphs, evaluation condition change information, evaluation sets, model call results, execution trajectories, evaluation records, evaluation results, and evaluation reports via high-speed interconnect. By having different processors perform different processes, it can adapt to different types of computing needs, such as evaluation data processing, batch invocation of tested agents, execution trajectory acquisition, and evaluation task processing.

[0230] In some implementations, the electronic device can be a terminal device, an evaluation task orchestration server, a tested intelligent agent service device, a model evaluation server, an edge computing device, or other computing devices with data processing capabilities, or it can be a distributed electronic device composed of multiple computing nodes.

[0231] When the electronic device is a distributed electronic device, different functional units in the automatic model evaluation task orchestration device can be deployed on the same computing node or on different computing nodes that can communicate with each other. For example, the functional unit for obtaining evaluation requirements and confirmation information can be deployed on the terminal device; the functional unit for generating evaluation task specifications, evaluation task dependency graphs, and determining target evaluation task nodes can be deployed on the evaluation task orchestration server; the functional unit for executing the call node of the agent under test can be deployed on the agent under test service device; and the functional unit for executing the call result evaluation node, evaluation result generation node, and reflective decision generation node can be deployed on the model evaluation server.

[0232] Components in electronic devices are not required to all conform to Figure 6 The setup is as shown. Depending on the specific implementation needs, the electronic device may omit some input or output devices, or it may add a graphics processor, artificial intelligence accelerator, task scheduler, data cache device, high-speed network interface, or other components suitable for performing evaluation task orchestration, evaluation of the results of the intelligent agent under test and the model call.

[0233] In some alternative embodiments, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the aforementioned method embodiments.

[0234] Specifically, a computer-readable storage medium can be a tangible medium capable of containing or storing computer programs. Computer-readable storage media can include portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, solid-state drives, optical storage devices, magnetic storage devices, or any combination of the above media.

[0235] The computer-readable storage medium may be disposed within the aforementioned electronic device or independently of the aforementioned electronic device. The computer program stored in the computer-readable storage medium may include program instructions for causing the processor to execute evaluation task specification acquisition, evaluation task dependency graph generation, reference evaluation task information query, evaluation condition change information determination, target evaluation task node determination, existing evaluation product reuse, target evaluation task node execution, and current evaluation result generation and output.

[0236] When a computer program is executed by one or more processors, the one or more processors may perform all or part of the steps in the foregoing method embodiments. When executed jointly by multiple processors, different processors may respectively perform different processes in evaluation requirement parsing, evaluation task dependency graph generation, evaluation condition change analysis, evaluation data processing, invocation of the tested intelligent agent, evaluation of invocation results, generation of evaluation results, and generation of reflective decisions.

[0237] In some alternative implementations, this application also provides a computer program product, including a computer program or computer instructions, which, when executed by a processor, implement any of the aforementioned method embodiments.

[0238] Specifically, computer program products may include computer programs or computer instructions for implementing methods for orchestrating automated model evaluation tasks. These computer programs or instructions may be downloaded from a network and installed on an electronic device, or loaded onto an electronic device from a computer-readable storage medium.

[0239] When executed by the processor, the computer program or computer instructions cause the electronic device to obtain the evaluation task specification of the current evaluation task; generate an evaluation task dependency graph based on the evaluation task specification; query the reference evaluation task information corresponding to the current evaluation task and obtain the query results; determine the evaluation condition change information based on the evaluation task specification and the query results; determine the target evaluation task node based on the evaluation condition change information, and reuse existing evaluation products if reuse conditions are met; execute the target evaluation task node according to the evaluation task dependency graph, and obtain and output the current evaluation result based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

[0240] Computer programs or instructions can be written in one or more programming languages ​​and can be executed entirely on a single electronic device, or they can be executed separately on terminal devices, evaluation task orchestration nodes, evaluation data processing nodes, tested agent invocation nodes, invocation result evaluation nodes, evaluation result generation nodes, and reflective decision generation nodes. The program parts executed at different nodes can exchange data such as evaluation task specifications, evaluation task dependency graphs, evaluation condition change information, evaluation sets, model invocation results, execution trajectories, evaluation results, evaluation reports, and reflective decisions via wired or wireless networks.

[0241] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, apparatuses, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a functional unit, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function.

[0242] It should be noted that in some alternative implementations, the functions marked in the boxes may be executed in a different order than that shown in the accompanying drawings. For example, two consecutively shown boxes may actually be executed substantially in parallel, or in reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as combinations of boxes in the block diagram and / or flowchart, may be implemented by a dedicated hardware system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0243] The units involved in the embodiments of this application can be implemented in software, in hardware, or in a combination of both. The name of a unit does not necessarily limit the unit itself. For example, a task execution unit can also be described as "a unit that executes the target evaluation task node according to the evaluation task dependency graph, and obtains and outputs the current evaluation result based on at least one of the evaluation products generated by the target evaluation task node and reused existing evaluation products."

[0244] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with, but not limited to, technical features disclosed in this application that have similar functions.

Claims

1. A method for arranging automatic model evaluation tasks, characterized in that, The method includes: Obtain the evaluation task specifications for the current evaluation task; An evaluation task dependency graph is generated according to the evaluation task specification, wherein the evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependency relationships, and the multiple evaluation task nodes include at least a called node of the tested intelligent agent and a called result evaluation node; Query the reference evaluation task information corresponding to the current evaluation task to obtain the query results; Based on the evaluation task specifications and the query results, determine the evaluation condition change information; Based on the evaluation condition change information, the affected evaluation task nodes are identified as target evaluation task nodes. If the query result indicates that the reference evaluation task information is found and there are unaffected evaluation task nodes, the existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes are reused. The target evaluation task node is executed according to the evaluation task dependency graph, and the current evaluation result is obtained and output based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

2. The method according to claim 1, characterized in that, The evaluation task specification includes configuration information of the tested intelligent agent and evaluation process configuration information. The evaluation process configuration information is used to determine the processing object and execution parameters of each evaluation task node. Obtaining the evaluation task specification of the current evaluation task includes: Obtain the evaluation requirements for the tested intelligent agent; Based on the evaluation requirements, the configuration information of the tested intelligent agent and the evaluation process configuration information are analyzed to obtain the task configuration information; The task configuration information is verified for completeness. Missing configuration items in the task configuration information are completed or the missing configuration items are identified as configuration items to be confirmed. Conflicting configuration items in the task configuration information are also identified as configuration items to be confirmed. If the task configuration information meets the integrity requirements and there are no configuration items to be confirmed, the evaluation task specification is generated based on the task configuration information. If the configuration item to be confirmed exists, the configuration item to be confirmed is output, and in response to receiving confirmation information for the configuration item to be confirmed, the task configuration information is updated according to the confirmation information, and the evaluation task specification is generated according to the updated task configuration information.

3. The method according to claim 2, characterized in that, The step of determining the evaluation condition change information based on the evaluation task specifications and the query results includes: If the query result indicates that the reference evaluation task information is found, the task configuration information corresponding to the current evaluation task is compared with the task configuration information recorded in the reference evaluation task information to determine the configuration items that have changed, the affected evaluation task nodes are determined based on the changed configuration items, and evaluation condition change information is generated based on the affected evaluation task nodes. If the query results indicate that the reference evaluation task information is not found, evaluation condition change information is generated to indicate that all of the multiple evaluation task nodes are affected.

4. The method according to claim 1, characterized in that, The query result indicates that the reference evaluation task information has been found, and the method further includes: The evaluation image specification is generated or loaded according to the reference evaluation task information. The evaluation image specification records the non-tested evaluation conditions associated with the reference evaluation task information, excluding the tested version parameters and the environmental parameters used to locate the tested version, as well as the version information or consistency verification information corresponding to the non-tested evaluation conditions. According to the evaluation mirror specification, the consistency of the non-tested evaluation conditions corresponding to the current evaluation task is checked. If the non-tested evaluation conditions pass the consistency check, and the evaluation condition change information indicates that the changed evaluation conditions only include at least one of the tested version parameters and the environment parameters, then the non-tested evaluation conditions recorded in the evaluation mirror specification are determined as the non-tested evaluation conditions of the current evaluation task. At least one of the tested version parameters and the environment parameters that have changed and are associated with the reference evaluation task information is replaced with the corresponding parameters in the current evaluation task. Based on the replaced evaluation conditions, the target evaluation task node is executed to obtain the current evaluation result. The current evaluation result is then compared with the reference evaluation result corresponding to the reference evaluation task, and a version comparison result is output. If at least one of the non-tested evaluation conditions fails the consistency check, the evaluation condition change information is updated based on the non-tested evaluation condition that failed the consistency check. The target evaluation task node is then re-determined based on the updated evaluation condition change information, and the re-determined target evaluation task node is executed based on the evaluation conditions corresponding to the current evaluation task.

5. The method according to claim 1, characterized in that, The target evaluation task node includes an evaluation data processing node, and the evaluation products generated by the evaluation data processing node include an evaluation set. Executing the evaluation data processing node includes: Read the evaluation set source parameters from the evaluation task specifications; If the evaluation set source parameter points to an existing evaluation set, the existing evaluation set is determined as the initial evaluation set; When the evaluation set source parameter points to the seed evaluation set, based on the evaluation scenarios, scenario descriptions and user query examples in the seed evaluation set, an extended evaluation sample corresponding to at least one of boundary problems, query variants and combined scenarios is generated, and the initial evaluation set is determined based on the seed evaluation set and the extended evaluation sample. The evaluation samples in the initial evaluation set are cleaned, deduplicated, and quality checked to obtain the evaluation set. The fields of each evaluation sample in the evaluation set are parsed to obtain the user query, and at least one of the expected execution steps, calling constraints and result assertions associated with the user query; The fields corresponding to the user query are mapped to the input fields required for the tested agent to call the interface, and at least one of the expected execution steps, calling constraints, and result assertions associated with the user query is determined as the evaluation basis for evaluating the model calling results generated by the tested agent in response to the user query.

6. The method according to claim 1, characterized in that, The target evaluation task node includes the tested intelligent agent invocation node, and executing the tested intelligent agent invocation node includes: Obtain the business space corresponding to the tested intelligent agent and the evaluation set used to call the tested intelligent agent; The template is invoked based on the business space and the intelligent agent under test query model; If an existing model invocation template corresponding to the business space and the tested intelligent agent is found, the existing model invocation template is determined as the target model invocation template; If no existing model call template is found, obtain the call chain document or call log corresponding to the tested agent, parse the interface call relationship and request parameters according to the call chain document or call log, generate a model call template according to the parsing result, and determine the generated model call template as the target model call template. Based on the target model, a debug call process and a full call process are generated using the call template. Select a portion of the evaluation samples from the evaluation set, call the tested intelligent agent according to the debugging call process, and set the current evaluation task to a waiting confirmation state. In response to receiving the continue execution information, the test agent is invoked using the evaluation set according to the full call process, and the model call results and corresponding execution trajectory identifiers generated by the test agent for each evaluation sample are obtained. In response to receiving correction information, the corresponding evaluation task parameters are updated according to the correction information, and the evaluation task node using the updated evaluation task parameters is re-executed.

7. The method according to claim 1, characterized in that, The target evaluation task node includes the call result evaluation node. Executing the call result evaluation node includes: Obtain the evaluation samples used by the calling node of the tested intelligent agent, the model calling results output by the calling node of the tested intelligent agent, and the execution trajectory identifier corresponding to the model calling results; Based on the execution trajectory identifier, obtain the model execution process information, tool call information, and interface execution information generated when the tested intelligent agent processes the corresponding evaluation sample; The evaluation samples, model call results, model execution process information, tool call information, and interface execution information are associated to generate an evaluation record; If there is no existing evaluation record set corresponding to the current evaluation task, an evaluation record set is created based on multiple evaluation records, and the created evaluation record set is determined as the target evaluation record set. If there is an existing evaluation record set corresponding to the current evaluation task, update the existing evaluation record set according to multiple evaluation records, generate a new evaluation record set version, and determine the new evaluation record set version as the target evaluation record set; The existing evaluator is read according to the evaluation task specification and the existing evaluator is determined as the target evaluator, or an evaluator is generated according to the evaluation requirement document and evaluation rules corresponding to the current evaluation task and the generated evaluator is determined as the target evaluator; Determine the evaluator identifier and evaluator version of the target evaluator; The target evaluation record set, the evaluator identifier, and the evaluator version are submitted as an evaluation task, and the running status of the evaluation task is polled until the evaluation task reaches the end state, and the evaluation status of each evaluation record in the target evaluation record set is obtained. Based on the evaluation status of each evaluation record, determine the evaluation records that have been successfully evaluated and the evaluation records that have not been successfully evaluated, and obtain the evaluation results corresponding to the evaluation records that have been successfully evaluated. In the absence of any evaluation records that have not been successfully completed, the evaluation result corresponding to the evaluation record that has been successfully completed shall be determined as the current evaluation result; In the case of evaluation records that have not been successfully evaluated, the target evaluator is used to re-evaluate the evaluation records that have not been successfully evaluated to obtain the retry evaluation result. The evaluation result corresponding to the evaluation record that has been successfully evaluated is then merged with the retry evaluation result to obtain the current evaluation result.

8. The method according to claim 1, characterized in that, The target evaluation task node also includes an evaluation result generation node. Executing the evaluation result generation node includes: Obtain the current evaluation result and the evaluation sample corresponding to the current evaluation task, and obtain the sample-level evaluation result corresponding to each evaluation sample from the current evaluation result; The sample-level evaluation results are aggregated to obtain aggregated evaluation results; The evaluation details are generated by integrating the sample-level evaluation results, the aggregated evaluation results, and at least one of the expected execution steps, invocation constraints, and result assertions in the evaluation samples. The evaluation indicators are calculated based on the evaluation details, and the failed samples are determined based on the evaluation indicators and the sample-level evaluation results. Obtain the execution trajectory and execution trajectory identifier corresponding to the failed sample, and perform failure cause analysis on the failed sample based on the execution trajectory to obtain the failure analysis result; An evaluation report is generated based on the evaluation details, the evaluation indicators, and the failure analysis results, and an association is established between the failure analysis results and the corresponding execution trajectory identifiers. A quality access control conclusion is generated based on the evaluation indicators and preset quality judgment conditions.

9. The method according to claim 8, characterized in that, The target evaluation task node also includes a reflection and decision generation node. Executing the reflection and decision generation node includes: Based on at least one of the current evaluation results, the failed samples, the evaluation indicators, and the quality access control conclusion, determine the evaluation anomaly information; Obtain the execution trajectory corresponding to the evaluation anomaly information and the historical evaluation information corresponding to the tested intelligent agent; Based on the evaluation anomaly information, the execution trajectory, and the historical evaluation information, determine the failure type and the root cause of the failure; The improvement targets are determined based on the failure types; Based on the root cause of failure and the execution trajectory, failure evidence is generated, and a reflective decision is generated including the failure evidence, the root cause of failure, and improvement actions for the improvement target. The reflected decisions will be incorporated into the evaluation knowledge base.

10. A model automatic evaluation task scheduling device, characterized in that, The device includes: The specification acquisition unit is used to acquire the evaluation task specification for the current evaluation task; The dependency graph generation unit is used to generate an evaluation task dependency graph according to the evaluation task specification, wherein the evaluation task dependency graph includes multiple evaluation task nodes connected based on input-output dependency relationships, and the multiple evaluation task nodes include at least a called node of the tested intelligent agent and a called result evaluation node; The information query unit is used to query reference evaluation task information corresponding to the current evaluation task and obtain query results. The change information determination unit is used to determine the change information of the evaluation conditions based on the evaluation task specifications and the query results; The node orchestration unit is used to determine the affected evaluation task nodes as target evaluation task nodes based on the evaluation condition change information, and to reuse the existing evaluation products associated with the reference evaluation task information and corresponding to the unaffected evaluation task nodes when the query result indicates that the reference evaluation task information is found and there are unaffected evaluation task nodes. The task execution unit is used to execute the target evaluation task node according to the evaluation task dependency graph, and obtain and output the current evaluation result based on at least one of the evaluation products generated by the target evaluation task node and the reused existing evaluation products.

11. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When one or more programs are executed by one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images