An agent-oriented evaluation task execution method and device, equipment and medium

Abstract

The application discloses an agent-oriented evaluation task execution method and device, equipment and a medium. The method comprises the following steps: generating each instance identifier corresponding to the evaluation task identifier and each test instance corresponding to each instance identifier according to the obtained evaluation request data; determining the parent arrangement corresponding to each test instance according to the parent arrangement idempotent identity of the test instance; generating an execution plan node sequence through the parent arrangement, wherein the execution plan node sequence comprises at least one to-be-executed node; screening a current execution node in each to-be-executed node; generating an execution attempt identifier and a frozen running load for the current execution node, so that a sub-execution performs a corresponding operation according to the frozen running load to obtain sub-execution information; and generating evaluation task information corresponding to the evaluation task identifier according to each sub-execution information of each test instance. The embodiment of the application can plan the evaluation task execution progress through the parent arrangement, thereby improving the accuracy of the agent-oriented evaluation task execution.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for performing evaluation tasks for intelligent agents. Background Technology

[0002] Agent evaluation systems typically require running the agent under test in an isolated execution environment and evaluating the results using scripts, models, or manually defined evaluators after execution. Compared to ordinary batch processing tasks, agent evaluation has the following characteristics: First, a single evaluation is usually not a single command, but includes multiple steps such as environment preparation, agent execution, execution of multiple evaluators, conditional advancement, failure retries, and resource cleanup; Second, some resources have experimental lifecycles, such as the service environment started for a test instance, the working directory or sandbox retained for reuse by the evaluators after the test execution; Third, evaluation results need to be stored long-term and traceable, and old results should not be overwritten by retries or repeated dispatches.

[0003] To simplify scheduling, existing systems often treat each test execution or evaluation execution as an independent queue task. After the queue task is completed, the result writing logic creates the next batch of evaluation tasks or the next round of test tasks. This approach works for simple, one-off tasks, but in multi-round, multi-evaluator, and retryable agent evaluations, it is prone to problems such as scattered execution plans, inconsistent failure paths, missed resource cleanup, and mutual contamination of retry results. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for performing evaluation tasks for intelligent agents, in order to improve the accuracy of performing evaluation tasks for intelligent agents.

[0005] In a first aspect, embodiments of the present invention provide a method for performing an evaluation task for an intelligent agent, the method comprising:

[0006] Based on the evaluation request data corresponding to the target user, generate an evaluation task identifier, at least one instance identifier corresponding to the evaluation task identifier, and test instances corresponding to each instance identifier;

[0007] For each test instance, the parent arrangement corresponding to the test instance is determined based on the idempotency of the parent arrangement.

[0008] Based on the test cases and evaluation request data, the parent orchestration generates an execution plan node sequence, which includes at least one node to be executed.

[0009] Filter through the nodes to be executed to obtain the current execution node;

[0010] An execution attempt identifier and a frozen running payload are generated for the current execution node and sent to the sub-execution corresponding to the current execution node, so that the sub-execution can perform corresponding operations according to the frozen running payload and obtain sub-execution information;

[0011] Based on the sub-execution information corresponding to each test instance, generate the evaluation task information corresponding to the evaluation task identifier.

[0012] Secondly, embodiments of the present invention also provide an evaluation task execution device for intelligent agents, the device comprising:

[0013] The instance generation module is used to generate an evaluation task identifier, at least one instance identifier corresponding to the evaluation task identifier, and test instances corresponding to each instance identifier based on the evaluation request data corresponding to the target user.

[0014] The parent arrangement determination module is used to determine the parent arrangement corresponding to each test instance based on the idempotency of the parent arrangement of the test instance.

[0015] The sequence generation module is used to generate an execution plan node sequence based on the test cases and evaluation request data through the parent orchestration. The execution plan node sequence includes at least one node to be executed.

[0016] The node filtering module is used to filter the nodes to be executed and obtain the currently executing node;

[0017] The information determination module is used to generate an execution attempt identifier and a frozen running payload for the current execution node, and send them to the sub-execution corresponding to the current execution node, so that the sub-execution can perform corresponding operations according to the frozen running payload and obtain sub-execution information;

[0018] The information generation module is used to generate evaluation task information corresponding to the evaluation task identifier based on the sub-execution information corresponding to each test instance.

[0019] Thirdly, embodiments of the present invention also provide an evaluation task execution device for intelligent agents, the evaluation task execution device for intelligent agents comprising:

[0020] At least one processor; and

[0021] A memory that is communicatively connected to at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the agent-oriented evaluation task execution method according to any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute an agent-oriented evaluation task execution method according to any embodiment of the present invention.

[0024] The technical solution of this invention generates an evaluation task identifier and at least one instance identifier corresponding to the evaluation task identifier, as well as test instances corresponding to each instance identifier, based on the evaluation request data corresponding to the target user. For each test instance, the parent orchestration corresponding to the test instance is determined according to the idempotent status of the parent orchestration. The parent orchestration generates an execution plan node sequence based on the test instance and the evaluation request data, and the execution plan node sequence includes at least one node to be executed. Each node to be executed is filtered to obtain the current execution node. An execution attempt identifier and a frozen running payload are generated for the current execution node and sent to the sub-execution corresponding to the current execution node so that the sub-execution performs corresponding operations according to the frozen running payload to obtain sub-execution information. Based on the sub-execution information corresponding to each test instance, evaluation task information corresponding to the evaluation task identifier is generated. By uniformly holding the complete execution plan through the test-level parent orchestration, the inconsistency problem caused by the creation of multiple scattered side effects in subsequent tasks is reduced, and the accuracy of the evaluation task execution for intelligent agents is improved.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an evaluation task execution method for intelligent agents according to an embodiment of the present invention;

[0028] Figure 2 This is a flowchart of an evaluation task execution method for intelligent agents according to an embodiment of the present invention;

[0029] Figure 3 This is a structural diagram of an agent-oriented evaluation task execution device according to an embodiment of the present invention;

[0030] Figure 4This is a schematic diagram of the structure of an evaluation task execution device for intelligent agents provided in an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] In the technical solutions of the embodiments of the present invention, the acquisition, storage and application of evaluation request data, etc., all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] Example 1

[0035] Figure 1 This is a flowchart illustrating an agent-oriented evaluation task execution method according to Embodiment 1 of the present invention. This embodiment of the invention is applicable to agent-oriented evaluation task execution. The method can be executed by an agent-oriented evaluation task execution device, which can be implemented in hardware and / or software.

[0036] See Figure 1 The agent-oriented evaluation task execution method shown includes:

[0037] S101. Based on the evaluation request data corresponding to the target user, generate an evaluation task identifier, at least one instance identifier corresponding to the evaluation task identifier, and test instances corresponding to each instance identifier.

[0038] The evaluation request data can be a complete request message and configuration parameter set submitted by the target user when initiating an agent evaluation service through the control plane application programming interface (API). It may include: the version identifier of the agent under test, the evaluation dataset, the evaluation rule configuration, the maximum number of service retries, the sandbox resource specifications, the evaluation scheduling mode, the evaluation termination conditions, and other service parameters. This serves as the original input data source for the evaluation task. The evaluation task identifier can be a globally unique identifier, serving as the top-level aggregated primary key for a complete user evaluation request. It uniquely binds all evaluation-related data submitted by the user, enabling unified collection, querying, and status summarization of batch test instances. Only one evaluation task identifier is generated for each user's single evaluation request. The instance identifier can be a secondary identifier belonging to the evaluation task identifier. Multiple independent instance identifiers exist under one evaluation task, with each instance identifier corresponding to a single test instance. Test instances can be the smallest independent evaluation execution unit in the agent evaluation system; they are composed of a single sample to be evaluated and a specified version of the agent under test, and have independent execution processes, isolated operating resources, and independent audit evidence chains; test instances run without interfering with each other and have independent parent orchestration lifecycles.

[0039] Specifically, the system receives evaluation request data submitted by the target user and performs parameter validity verification. A globally unique string is assigned as the evaluation task identifier, which serves as the top-level index to carry the global configuration of the evaluation task. The evaluation dataset within the evaluation request data is parsed and split according to the data sample granularity. For each sample to be evaluated, a unique instance identifier is generated, establishing a one-to-many association between the evaluation task identifier and the instance identifier. For each instance identifier, the corresponding sample to be evaluated, the tested agent information, and the common configuration parameters inherited from the top-level evaluation task are bound to construct an independent test instance. The association between the evaluation task identifier, instance identifier, and test instance is persistently written to the business storage. All test instances can be managed in batches through the upper-level evaluation task; each test instance is isolated from each other and can independently start its own parent orchestration to carry out the evaluation process. The running status, resources, and outputs of each test instance do not interfere with each other, and the failure of a single test instance will not affect the normal operation of other test instances under the same task.

[0040] S102. For each test instance, determine the parent arrangement corresponding to the test instance based on the idempotent status of the parent arrangement.

[0041] The parent orchestration's idempotent identity can be a globally unique identifier bound to a single test instance, used to uniquely identify the complete evaluation control process of that test instance; it is generated by combining the instance identifier and the evaluation process type. This identity serves as the idempotency verification primary key, used to distinguish between repeatedly initiated process start requests, ensuring that the same test instance and the same type of evaluation process correspond to only one set of control processes. The parent orchestration can be a persistent control process unit running at the scheduling layer, bound to a single test instance; the parent orchestration does not directly perform agent operations or evaluation operations, but is only responsible for pre-generating the execution plan node sequence, controlling node flow, executing retry / skip branch decisions, reading persistent execution results to advance the evaluation process, and coordinating the entire lifecycle of the test instance from startup to resource reclamation.

[0042] Specifically, after generating and persistently storing test instances, each test instance to be launched is traversed; the instance identifier carried by the test instance and the evaluation process type corresponding to this evaluation are extracted, and a unique parent orchestration idempotent identity for that test instance is generated; the parent orchestration idempotent identity is used as the primary key to query the business storage and perform duplicate trigger verification: if a valid parent orchestration created based on the idempotent identity already exists in the business storage, the existing parent orchestration is reused directly without creating a new process; if no corresponding record exists in the business storage, a new parent orchestration is created and bound to the test instance, and the relationship between the parent orchestration and the parent orchestration idempotent identity is established and written to the business storage; finally, the matching is completed through the idempotent identity to determine the unique parent orchestration corresponding to the current test instance. Relying on the parent orchestration idempotent identity to complete duplicate trigger verification, in the face of scenarios such as repeated front-end requests, scheduling replay, and service crash restarts, the parent orchestration will not be repeatedly created for the same test instance, preventing the repeated launch of the entire evaluation process, saving sandbox and computing resources, and avoiding multiple sets of conflicting evaluation results.

[0043] S103. Based on the test cases and evaluation request data, the parent orchestration generates an execution plan node sequence, which includes at least one node to be executed.

[0044] The execution plan node sequence can be an ordered set of business steps with sequential dependencies, constructed by the parent orchestration based on business rules, fully defining the entire business process skeleton of the test instance from startup to resource recovery. Nodes within the sequence are arranged according to business time order, solidifying the prerequisite dependencies, execution conditions, skip rules, and retry limits between nodes. Execution nodes can be basic business units in the execution plan node sequence, representing a type of operation that needs to be completed at the business level. The types of execution nodes can include environment preparation nodes, test execution nodes, evaluation nodes, condition advancement nodes, and resource cleanup nodes. Each execution node records metadata such as node type, prerequisite dependencies, resource requirements, evaluation thresholds, number of business retry attempts, and skip trigger conditions.

[0045] Specifically, after the parent orchestration completes the binding of the test instance and confirms the idempotency verification, it loads the sample and tested agent information that come with the current test instance, and simultaneously reads the global evaluation configuration from the top-level evaluation request data. Following the agent evaluation standard business sequence, and combining the number of evaluations, multi-round iteration switches, and resource reuse strategies within the configuration, the parent orchestration sequentially generates ordered execution nodes, assembling them into a complete execution plan node sequence. When generating each execution node, node metadata is synchronously written: unique node number, node type, pre-dependent node ID, business retry limit, resource request specifications, evaluation binding relationship, and skip judgment conditions. The parent orchestration persistently writes the entire execution plan node sequence to the business storage. Execution nodes that do not meet the pre-running conditions are uniformly marked as pending scheduling. The execution plan node sequence is generated completely at once, and all business steps are pre-fixed, preventing downstream nodes from being dynamically created based on the side effects after sub-execution. By pre-generating a complete execution plan node sequence at the beginning of the process, all business steps are pre-fixed, ensuring that the evaluation business intent can be fully restored even after any node failure or service restart, significantly improving process consistency. Furthermore, after the execution plan node sequence is persistently stored, external systems can directly read all nodes to be executed, fully displaying the entire evaluation process steps, node dependencies, and retry and skip rules for each node. Users can intuitively view the complete evaluation plan without waiting for the task to run in order to predict subsequent steps.

[0046] S104. Filter each node to be executed to obtain the current execution node.

[0047] The current execution node can be a single execution node that has been screened and determined by the parent orchestration, meets all pre-run constraints, and is waiting to be created for execution and sent to the child execution. At any given time, there is only one current execution node as the scheduling target, which is the only business step object for the parent orchestration to initiate the child execution task.

[0048] Specifically, after the parent orchestration schedules time or completes all rounds of execution, it loads the complete execution plan node sequence from the business storage. It then iterates through all nodes to be executed within the sequence, removing those marked as skipped, having exhausted their retries, or with a final state of success / failure. Only nodes in their initial scheduling state and those not yet completed are retained. The parent orchestration reads the IDs of the preceding dependent nodes configured for each remaining node, queries the business storage for the final states of all execution attempts for the corresponding preceding node, and verifies whether all preceding dependent nodes have been completed (successfully or after exhausting retries). Only nodes with all preceding dependencies closed loops are retained. Following the preset business sequence order of the execution plan node sequence, the nodes that meet the filtering criteria are sorted, and the first node in the sorted list is taken as the current execution node. The parent orchestration writes the lock flag of this current execution node into the business storage to prevent concurrent scheduling from repeatedly selecting the same node, and outputs a unique current execution node for creating execution attempts. If no node meets the criteria after iterating through all nodes to be executed, the entire execution plan is considered closed loop, and the process directly enters the final state aggregation process for the experimental instance.

[0049] S105. Generate an execution attempt identifier and a frozen running payload for the current execution node, and send them to the sub-execution corresponding to the current execution node, so that the sub-execution can perform corresponding operations according to the frozen running payload and obtain sub-execution information.

[0050] The execution attempt identifier can be a globally unique identification code generated for a single actual run, serving as the core primary key to distinguish multiple retries of the same logical node. Each time a sub-execution is started, a new execution attempt identifier is generated. Multiple retries of the same logical node correspond to multiple non-repeating execution attempt identifiers, and each identifier is independently bound to a specific runtime artifact, log, and final state record. The frozen runtime payload can be a static read-only runtime parameter data package encapsulated by the parent orchestration based on the current test instance, the current execution node metadata, and the top-level evaluation request configuration. The payload is frozen before being issued to the sub-execution, and the sub-execution cannot modify the configuration within the payload during operation. It can include all runtime constraint parameters such as task input data, the version of the tested agent / evaluator, environmental resource policies, inheritable resource references, independent namespaces for artifacts, access permissions for reading evaluation evidence, and the maximum retry limit for a single node. The sub-execution can be a low-level runtime unit bound to a single execution attempt identifier, carrying out only a single independent run operation without global process scheduling capabilities. It only reads the frozen runtime payload to complete environmental operations, agent operation, evaluation calculations, or resource reclamation, and all runtime output data is only written to the independent storage domain corresponding to the current execution attempt identifier. Sub-execution information can be a complete set of execution results output and persisted after the sub-execution has completed all operations. It can include all single-run evidence such as environment configuration records, execution logs, session traces, artifact file indexes, evaluation scores, resource operation records, execution final state (success / failure / cancellation), error stacks, and resource reference handles.

[0051] Specifically, after the parent orchestration determines the current execution node, it combines the test instance identifier, the current execution node number, the timestamp, and a random checksum to generate a unique string, which serves as the unique primary key for this run. The parent orchestration reads the basic information of the test instance, the global configuration of the evaluation request, the metadata specific to the current execution node, and the inheritable resource references produced by the upstream node, integrating them into a complete runtime parameter package. The parameter package is then marked with a fixed tag, forming an immutable frozen runtime payload to prevent parameters from being tampered with during runtime. The execution attempt identifier is bound to the current execution node and the frozen runtime payload, and written to the business storage for pre-persistence. The parent orchestration delivers the execution attempt identifier and frozen runtime payload to the corresponding child execution process / task queue, specifying that the child execution only handles the runtime task corresponding to the execution attempt. After receiving the payload, the child execution strictly performs the corresponding business operation according to the constraint parameters in the frozen runtime payload, such as environment preparation, agent inference, multi-judgment scoring, or resource reclamation. After the child execution completes all computations and resource operations, it collects all logs, products, scores, resource operation records, and execution result status of this run, summarizes and generates complete child execution information, and binds all child execution information with this execution attempt identifier and writes it to the business storage.

[0052] S106. Based on the sub-execution information corresponding to each test instance, generate the evaluation task information corresponding to the evaluation task identifier.

[0053] The evaluation task information can be a top-level summary result dataset generated by aggregating all the underlying data with the evaluation task identifier as the unique primary key. It includes the overall task overview, the summary status of each test instance, the details of multiple execution attempts at each node, the full evaluation score statistics, the resource usage summary, the overall evaluation conclusion, and the failure / skip details list, providing a complete, displayable, and exportable full picture of the evaluation task data.

[0054] Specifically, using the target evaluation task identifier as the search condition, the system batch queries all instance identifiers associated with that evaluation task identifier from the business storage, and then retrieves the complete test instance data corresponding to all instance identifiers. Next, it iterates through each test instance, reading the sub-execution information corresponding to all its subordinate execution attempts. This involves summarizing the sub-execution information corresponding to all environment preparation, tested execution, evaluation, and resource cleanup under a single test instance; statistically analyzing the number of successful / failed execution attempts for each logical node, historical evaluation scores, resource creation and reclamation records, node skipping reasons, and the overall final state of the single instance (complete / failed / cancelled), generating summary data for a single test instance; and aggregating the summary data of all test instances under this task to uniformly calculate global metrics: total number of task samples, number of successful samples, number of failed samples, average score for each evaluation dimension, total resource consumption, details of failed samples, list of skipped evaluation processes, and summary of all retry records. The basic configuration of the task, the details of all test instances, global statistical indicators, the summary of evaluation scores, the resource usage ledger, the list of abnormal records, and the overall final state of the task are uniformly encapsulated into structured evaluation task information, and a unique binding relationship is established with the evaluation task identifier, which is persistently written to the business storage. External front-ends and auditing systems can read the complete evaluation task information at once through the evaluation task identifier, without having to traverse the scattered test instances, execution attempts, and sub-execution information layer by layer.

[0055] The technical solution of this invention generates an evaluation task identifier and at least one instance identifier corresponding to the evaluation task identifier, as well as test instances corresponding to each instance identifier, based on the evaluation request data corresponding to the target user. For each test instance, the parent orchestration corresponding to the test instance is determined according to the idempotent status of the parent orchestration. The parent orchestration generates an execution plan node sequence based on the test instance and the evaluation request data, and the execution plan node sequence includes at least one node to be executed. Each node to be executed is filtered to obtain the current execution node. An execution attempt identifier and a frozen running payload are generated for the current execution node and sent to the sub-execution corresponding to the current execution node so that the sub-execution performs corresponding operations according to the frozen running payload to obtain sub-execution information. Based on the sub-execution information corresponding to each test instance, evaluation task information corresponding to the evaluation task identifier is generated. By uniformly holding the complete execution plan through the test-level parent orchestration, the inconsistency problem caused by the creation of multiple scattered side effects in subsequent tasks is reduced, and the accuracy of the evaluation task execution for intelligent agents is improved.

[0056] Example 2

[0057] Figure 2 This is a flowchart illustrating an agent-oriented evaluation task execution method according to Embodiment 2 of the present invention. Based on the above embodiments, this embodiment optimizes and improves the agent-oriented evaluation task execution operation.

[0058] Furthermore, the step of "determining the parent orchestration corresponding to the test instance based on the idempotent identity of the parent orchestration" is refined into "determining the evaluation process type corresponding to the test instance based on the evaluation request data; determining the idempotent identity of the parent orchestration corresponding to the test instance based on the instance identifier and evaluation process type corresponding to the test instance; repeatedly triggering verification of the idempotent identity of the parent orchestration to obtain the verification result; and determining the parent orchestration corresponding to the test instance based on the verification result," in order to improve the operation of the evaluation task execution for intelligent agents.

[0059] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments.

[0060] See Figure 2 The agent-oriented evaluation task execution method shown includes:

[0061] S201. Based on the evaluation request data corresponding to the target user, generate an evaluation task identifier, at least one instance identifier corresponding to the evaluation task identifier, and test instances corresponding to each instance identifier.

[0062] S202. For each test instance, determine the corresponding evaluation process type based on the evaluation request data.

[0063] Among them, the evaluation process type can be a standardized classification identifier used to distinguish the evaluation business logic of different intelligent agents, representing a set of execution plan node sequence templates with a fixed structure; different evaluation process types correspond to different node arrangements, number of evaluators, multi-round iteration rules, and resource reuse logic, which are used to generate differentiated parent orchestration idempotent identities and realize the isolated operation of multiple evaluation schemes for the same test instance.

[0064] Specifically, after the system completes the evaluation task and generates and persistently stores each test instance, it iterates through each test instance under the current evaluation task, executing the process type determination logic independently for each individual test instance. It extracts the complete evaluation request data bound to the current evaluation task identifier and parses the key configuration fields used to distinguish process types. These fields may include: whether to enable multi-round progressive evaluation, the number of evaluators, the evaluation scheduling mode, whether to reuse the test sandbox for the evaluators, whether to include a manual review node, and whether to execute only the test without evaluation. The system has multiple built-in preset judgment rules, which match the corresponding judgment process type code based on the parsed configuration field combinations: if the configuration only includes single test execution, single-group parallel judgment, no multi-round iteration, and sandbox reusability, it matches the single-round standard judgment process; if the configuration enables multi-round iteration and intermediate judgment thresholds control the next round of test execution, it matches the multi-round progressive judgment process; if the configuration only executes the tested agent and has no judgment nodes, it matches the pure test execution process; if the configuration includes both manual review nodes and automated judgment dual links, it matches the manual-assisted judgment process; if the configuration only executes resource baseline environment verification and has no test and judgment logic, it matches the environment baseline test process. The matched judgment process type code is written into the metadata of the current test instance and persistently stored in the business storage; this process type will be used as a component of the parent orchestration idempotent identity for subsequent generation of a unique parent orchestration idempotent identity identifier.

[0065] S203. Determine the parent arrangement idempotency of the test instance based on the instance identifier and evaluation process type corresponding to the test instance.

[0066] Specifically, for each test instance to be initiated as the parent orchestration, the system reads the instance identifier stored in the metadata of that test instance from the business storage, as well as the judgment process type code that is determined and persisted, to complete the value retrieval of these two types of fields. The system predefines a unified composite concatenation rule to combine the instance identifier and the judgment process type, adding a fixed separator between them to avoid identity confusion caused by field content conflicts. The concatenated string is uniformly formatted as a fixed-length / variable-length unique identifier, serving as the parent orchestration idempotent identity corresponding to the current test instance. The generated parent orchestration idempotent identity is persistently stored in the metadata field of that test instance in the business storage, establishing a binding relationship between the test instance, instance identifier, judgment process type, and parent orchestration idempotent identity.

[0067] S204. Repeatedly trigger verification of the parent idempotent identity to obtain the verification result.

[0068] Specifically, the parent orchestration idempotency identity is used as the unique search keyword and passed to the duplicate trigger verification logic. A database / distributed storage index is created using the parent orchestration idempotency identity, and an exact match query is executed to filter parent orchestration records in storage that are completely identical to the current idempotency identity and are in a valid state (running / paused / incomplete). Parent orchestrations that have been normally terminated or archived are not included in the duplicate detection scope. If a valid parent orchestration record is found: the verification result is marked as a duplicate trigger, and the retrieved existing parent orchestration entity is appended to the verification result; if no valid parent orchestration record is found: the verification result is marked as a first trigger, and a placeholder identifier is appended to the verification result, indicating that no existing parent orchestration can be reused. The verification result, including the judgment type and existing parent orchestration entities (if any), is output to the subsequent parent orchestration determination step as the basis for determining whether to create or reuse a parent orchestration.

[0069] S205. Based on the verification results, determine the parent arrangement corresponding to the test instance.

[0070] Specifically, the system extracts the judgment flags stored internally in the verification results, distinguishes between two branches: repeated triggers and first-time triggers, and executes differentiated processing logic. When the verification result is a repeated trigger, the system extracts the persisted, valid, and unarchived existing parent orchestration entity from the verification result's supplementary data; establishes a runtime binding relationship between this existing parent orchestration and the current test instance; directly uses the existing parent orchestration as the parent orchestration corresponding to the current test instance, skipping the logic of creating a new parent orchestration and generating an execution plan, and reuses the existing process state to continue the evaluation. When the verification result is a first-time trigger, based on the current test instance, evaluation request data, and the parent orchestration's idempotent identity, a new persistent parent orchestration entity is initialized; the parent orchestration's idempotent identity, instance identifier, evaluation process type, and test instance primary key association are written into the metadata of the new parent orchestration; the new parent orchestration entity is persistently stored, establishing a unique binding relationship between the test instance and the new parent orchestration; this new parent orchestration is determined as the parent orchestration corresponding to the current test instance, and enters the downstream process of generating the execution plan node sequence. Whether reusing existing resources or creating new ones, the final output is a unique parent orchestration bound to the current test instance, which serves as the unified scheduling entity for subsequent generation execution plans.

[0071] S206. Based on the test cases and evaluation request data, the parent orchestration generates an execution plan node sequence, which includes at least one node to be executed.

[0072] S207. Filter each node to be executed to obtain the current execution node.

[0073] S208. Generate an execution attempt identifier and a frozen running payload for the current execution node, and send them to the sub-execution corresponding to the current execution node, so that the sub-execution can perform corresponding operations according to the frozen running payload and obtain sub-execution information.

[0074] S209. Based on the sub-execution information corresponding to each test instance, generate the evaluation task information corresponding to the evaluation task identifier.

[0075] This invention, in its embodiments, determines the evaluation process type corresponding to the test instance based on the evaluation request data; determines the parent orchestration idempotent identity corresponding to the test instance based on the instance identifier and evaluation process type; performs repeated trigger verification on the parent orchestration idempotent identity to obtain the verification result; and determines the parent orchestration corresponding to the test instance based on the verification result. Redundant requests are uniformly verified through the parent orchestration idempotent identity, intercepting invalid and repetitive tasks from the source, reducing the load on the message queue, database, and execution environment, and minimizing resource waste.

[0076] Optionally, after generating an execution attempt identifier and freezing runtime payload for the current execution node and sending them to the sub-execution corresponding to the current execution node so that the sub-execution performs the corresponding operation according to the freezing runtime payload and obtains the sub-execution information, the method further includes: when the sub-execution result included in the sub-execution information is execution failure, determining the number of retries based on the evaluation request data; when the number of retries is not zero, generating a current attempt identifier so that the sub-execution re-executes the corresponding operation according to the freezing runtime payload and obtains the sub-execution information; when the number of retries is zero, generating a non-execution identifier for at least one associated node corresponding to the current execution node through the parent orchestration and determining the resource reclamation node as the current execution node; when the sub-execution result included in the sub-execution information is execution success, storing the sub-execution information and execution attempt identifier together.

[0077] The sub-execution result can be a field within the sub-execution information that marks the final state of a single run, containing only two standardized values: execution successful and execution failed. Execution failure indicates business-level failures such as environmental anomalies, agent errors, evaluation timeouts, or resource creation failures during the sub-execution process. The number of retry attempts can be the remaining unused business retry quota for the current execution node, calculated by subtracting the number of execution attempts already generated by the current logical node from the maximum number of retry attempts preset by the evaluation request data; it represents how many more independent execution attempts the current node can create and rerun. Associated nodes can be all downstream nodes that have a prerequisite dependency on the current execution node and need to wait for the current node to complete before being scheduled for execution, such as the associated nodes of the tested execution node being all evaluation nodes or multiple rounds of tested nodes. The non-execution flag can be a standardized flag written into the metadata of downstream associated nodes by the parent orchestration, marking that the node skipped execution due to the exhaustion of retries by the upstream node; the blocking reason and the failed execution attempt flag are recorded synchronously and permanently persisted. Resource reclamation nodes can be fixed closing logic nodes in the execution plan node sequence. They are used to uniformly release resources such as experimental sandboxes, background services, temporary working directories, and storage mounts. They are the final business nodes for the unified convergence of all successful and failed branches.

[0078] Specifically, after the parent orchestration reads the child execution information, it first extracts the child execution result field. If the child execution result is an execution failure, the parent orchestration reads the current node's preset maximum number of business retries from the evaluation request data, queries the total number of execution attempts created by this logical node in the business storage, and calculates the remaining number of retries based on the difference between the two. If the number of retries is greater than 0, the parent orchestration generates a new and independent current attempt identifier for the same logical node; it reuses the original frozen running payload and sends the new current attempt identifier and frozen running payload to the corresponding child execution; the child execution re-completes the environment, agent, or evaluation operations according to the frozen running payload, generating new child execution information; the logs, errors, and artifacts of the old failed execution attempts are completely preserved and will not be overwritten by the data generated by this retry. If the number of retries is 0 (retry quota exhausted), the parent orchestration traverses all downstream related nodes of the current execution node, writes a non-execution identifier to each related node in batches, and persistently stores the blocking reason and the index of failed execution attempts; it locks the resource reclamation node in the execution plan node sequence as the new current execution node and enters the resource cleanup scheduling process to prevent experimental resource leakage. When the sub-execution result is successful, the parent orchestration establishes a strong binding relationship between the complete sub-execution information output by this run and the corresponding execution attempt identifier; the bound related data is persistently written to the dedicated namespace of the business storage, and all evidence such as logs, artifacts, scores, and environment records of this run are permanently archived for subsequent evaluation threshold judgment and audit tracing.

[0079] When the sub-execution result included in the sub-execution information is execution failure, the number of retries is determined based on the evaluation request data. When the number of retries is not zero, a current attempt identifier is generated so that the sub-execution can be re-executed to perform the corresponding operation according to the frozen running load and obtain the sub-execution information. When the number of retries is zero, a non-execution identifier is generated for at least one associated node corresponding to the current execution node through the parent orchestration, and the resource reclamation node is determined as the current execution node. When the sub-execution result included in the sub-execution information is execution success, the sub-execution information and execution attempt identifier are stored together. Each business failure retry generates an independent current attempt identifier. All failure records, errors, and artifacts are permanently retained, which can completely reproduce each round of failure scenarios and meet the needs of intelligent agent evaluation experiment traceability, fault review, and compliance audit.

[0080] Optionally, the current execution node is the environment preparation node. The sub-execution performs corresponding operations based on the frozen running load to obtain sub-execution information, including: configuring the current execution environment based on the frozen running load to obtain the environment configuration result; determining at least one service to be started based on the test case, and starting each service to be started in the current execution environment to obtain the startup result corresponding to each service to be started; and associating and storing the environment reference corresponding to the current execution environment, the startup result corresponding to each service to be started, the environment configuration result, and the execution attempt identifier.

[0081] The current execution environment can be an isolated sandbox runtime environment created by the sub-execution based on the frozen runtime payload, serving as a dedicated isolated runtime space for the test instance. It can include container images, independent networks, dedicated storage directories, CPU / memory resource quotas, and mutually isolated independent environments corresponding to different execution attempts within the same test instance. The environment configuration result can be a structured record generated after the sub-execution completes environment initialization, containing complete environment setup credentials such as image pull status, resource quota allocation records, network port binding information, storage mount status, environment initialization error information, and environment creation time. The services to be started can be experimental, persistent background services supporting the normal operation of the tested agent and evaluator, such as code databases, cache proxies, interface gateways, and file storage services. The service lifecycle is bound to the current execution environment and is only effective during the current execution attempt. The startup result can be an independent runtime record generated for each service to be started, containing the service process PID, port listening status, startup logs, startup timeout / crash errors, health probe return values, and service resource usage information. An environment reference can be a global handle that uniquely identifies the current isolated execution environment, and can include the sandbox ID, container access address, storage mount path, and network access credentials; it can be passed as an inheritable resource to downstream test execution and evaluation nodes to reuse the current sandbox environment.

[0082] Specifically, the sub-execution reads the solidified environment configuration parameters within the frozen runtime load and creates an independent isolation sandbox according to the image address, CPU / memory quota, network isolation rules, persistent storage mount path, and security permission policies. Throughout the environment creation process, it collects full-process operation logs, resource allocation status, and error messages, and encapsulates them to generate a standardized environment configuration result. If image retrieval fails, resource quota is insufficient, or network creation fails, the environment configuration result is marked as failed, and a complete fault stack is recorded. The sub-execution reads the metadata of the test instance it is attempting to bind, parses all supporting backend services required for the evaluation business, the startup parameters of each service, dependency order, and health check rules. It then sequentially starts all services to be started within the already initialized current execution environment according to the preset dependency order. For each service started, it continuously executes the preset health probe logic. For each service to be started, it generates a separate startup result, recording the process status, health check return value, startup log, and timeout or crash exception information. After all services have started, it summarizes the set of all service startup results. The sub-execution extracts the globally unique environment reference handle corresponding to the current sandbox, and establishes a strong association mapping between the environment reference, the complete environment configuration result, and the startup results corresponding to all services to be started, and the execution attempt identifier of this run; writes the entire set of data after binding into the business storage structured data table, and archives the environment log and service startup log to the dedicated object storage path of the execution attempt identifier; after the data is written, it is fixed and archived, and new execution attempts generated by retrying on the same logical node cannot overwrite this record.

[0083] The sub-execution configures the current execution environment based on the frozen runtime load, obtaining the environment configuration result. Based on the test case, at least one service to be started is identified, and each service to be started is started in the current execution environment, obtaining the startup result corresponding to each service to be started. The environment reference corresponding to the current execution environment, the startup result corresponding to each service to be started, the environment configuration result, and the execution attempt identifier are associated and stored. The sub-execution builds a standardized isolation sandbox based on the unified frozen runtime load, synchronously starts the background services that the evaluation depends on, and persists the unique environment reference. Downstream tested and evaluated nodes can directly read the environment reference and reuse the existing sandbox without having to repeatedly create the environment or start the service, which greatly reduces the repeated overhead of environment initialization and improves the evaluation efficiency.

[0084] Optionally, the current execution node is the execution node under test. The sub-execution performs corresponding operations based on the frozen running load to obtain sub-execution information, including: generating evaluation data based on the frozen running load and storing the evaluation data in an independent storage area; loading the evaluation agent corresponding to the frozen running load in the current execution environment so that the evaluation agent can repair the evaluation data, obtain repair data and repair results, and store the repair data in an independent storage area; associating and storing the regional data, repair results, acquired collection information, and execution attempt identifier included in the independent storage area, and storing the acquired information including: repair logs, session files, and trajectory files.

[0085] The independent storage area can be an isolated storage space allocated by the system for a single execution attempt, with a unique access path, prefixed by the execution attempt identifier. Multiple retries of the same logical node will allocate completely different independent storage areas, and data within these areas will not overwrite or interfere with each other. The evaluation agent can be a test agent running a specified version of the payload. For example, it can be a code repair agent, capable of reading defective code, locating vulnerabilities, and outputting repair code, with its lifecycle limited to the current execution environment. Repair data can be standardized output files from the evaluation agent after completing the repair processing of the test data, including modified code files, patch files, change description documents, etc. Repair results can be structured state records used to mark the overall output status of the agent's repair behavior, including metadata such as repair execution success / failure, a list of modified files, the number of lines of code changed, the runtime error stack, and repair time. Area data can be a collection of all persistent files within the independent storage area, containing two main types of files: the original test data and the repair data output by the agent. The collected information can be a full-dimensional operational credential collected synchronously by the sub-execution during the complete operation cycle of the agent, and can include three types: repair logs, session files, and trajectory files. Repair logs can be the agent's standard output, standard error, and tool call logs. Session files can be a complete dialogue interaction record between the agent and the tool and environment. Trajectory files can be a full-link tracing record of the agent's reasoning steps, decision paths, and function calls.

[0086] Specifically, the sub-execution reads the original sample materials and data preprocessing rules from the frozen runtime payload, performs standardization, formatting, decompression, and directory initialization on the original samples, and generates unified format data to be evaluated. Based on the current execution attempt identifier, a dedicated independent storage area is allocated, and all data to be evaluated is written to this isolated storage space. Different execution attempts cannot read or write files in each other's storage areas, achieving input data isolation. The sub-execution reads the environment references carried by the frozen runtime payload and connects to the created current execution sandbox environment. It pulls and starts the evaluation agent according to the specified version of the payload, passing the data to be evaluated from the independent storage area as input to the agent. It drives the agent to execute code repair logic, fully capturing the entire output of the agent's operation and generating repair data. Simultaneously, it generates structured repair results, recording repair success / failure, changed files, error messages, and execution time. The repair data output by the agent is written to the same dedicated independent storage area and archived together with the original data to be evaluated. Throughout the entire operation of the intelligent agent, the sub-execution continuously collects three types of information: repair logs, session interaction files, and decision trajectory files, and stores them together in an independent storage area. The sub-execution extracts all file indexes (regional data), structured repair results, and complete collection information sets from the independent storage area, and establishes a unique binding mapping between all the above data and the execution attempt identifier of this run. The structured metadata is written to the business database, and all file materials are archived to the dedicated object storage path of the execution attempt identifier. After the data is written, it is permanently archived. New execution attempts generated by subsequent retries of the same logical node have a completely new independent storage area and will not overwrite the current run record.

[0087] The sub-execution generates evaluation data based on the frozen runtime load and stores the evaluation data in an independent storage area. The sub-execution then loads the evaluation agent corresponding to the frozen runtime load into the current execution environment, enabling the evaluation agent to repair the evaluation data, obtain repaired data and results, and store the repaired data in an independent storage area. The independent storage area includes regional data, repair results, acquired information, and execution attempt identifiers, all associated with each other. Acquired information includes repair logs, session files, and trajectory files. Multiple business retries on the same tested execution node are allocated completely independent storage spaces. The original code, repair artifacts, logs, and trajectory files from each agent run are permanently isolated and retained. This addresses the shortcomings of existing technologies that cover historical execution records during retries and cannot trace back to failed scenarios, meeting the needs of agent evaluation experiment review, fault tracing, and compliance auditing.

[0088] Optionally, the current execution node is the evaluation node. An execution attempt identifier and a frozen runtime payload are generated for the current execution node and sent to the corresponding sub-executions. This allows the sub-executions to perform corresponding operations based on the frozen runtime payloads and obtain sub-execution information, including: determining the number of evaluators and the scheduling mode based on the evaluation rules in the evaluation request data through the parent orchestration (the scheduling mode includes serial scheduling and parallel scheduling); generating evaluation execution attempts corresponding to the number of evaluators; generating frozen runtime payloads corresponding to each evaluation execution attempt and sending them to the corresponding sub-executions, so that each sub-execution evaluates the repair data according to the evaluation method corresponding to the evaluation type, according to the scheduling mode, and obtains the evaluation result. The evaluation types include: unit test evaluation and code quality evaluation; matching the evaluation result with preset threshold conditions to obtain a judgment result; and determining whether to designate the resource reclamation node as the current execution node based on the judgment result.

[0089] The evaluation rules can be a standardized set of evaluation constraints defined within the evaluation request data. These constraints govern the number of evaluators, the execution scheduling order, the calculation logic for each type of evaluation, and the result qualification criteria. They serve as the core basis for the parent orchestration to generate the evaluation process. The number of evaluators can be the total number of independent evaluation units required for this evaluation, obtained by parsing the evaluation rules. Different evaluation types correspond to independent evaluators. The scheduling mode can be the timing strategy for the parent orchestration to manage the execution of multiple evaluation sub-executors, divided into two categories: parallel scheduling: all evaluation execution attempts are issued simultaneously, and multiple sub-executors run evaluations concurrently; serial scheduling: evaluation execution attempts are issued sequentially according to a preset order, with the next one starting only after the previous one is completed. Each evaluation execution attempt can be a globally unique execution attempt identifier generated by the parent orchestration for a single evaluator. One evaluator corresponds to one independent execution attempt, and the storage domains of each evaluation attempt are isolated from each other, without overwriting evaluation logs and scoring records. The evaluation type can be a standardized evaluation category, including two types: Unit test evaluation: automating the execution of code test cases and outputting the test case pass rate; Code quality evaluation: outputting a quality score based on coding standards, vulnerability detection, and readability models. The evaluation method can be a specific scoring and calculation logic for each evaluation type. Unit test evaluation executes automated test scripts to calculate the pass rate; code quality evaluation executes static scanning and quality model scoring. The evaluation result can be structured data output after a single evaluation execution attempt, including evaluation type, score / pass rate, test case log, vulnerability list, evaluation execution attempt identifier, and execution final state. Preset threshold conditions can be preset global pass / fail criteria in the evaluation rules, determined by weighted / joint judgment of multiple evaluation results. For example, a unit test pass rate ≥ 90% and a code quality score ≥ 80 constitute an overall pass. The judgment result can be a binary classification conclusion output by the parent orchestration after comparing all evaluation results with the preset threshold, divided into two categories: overall pass and overall fail. Resource recycling nodes can be fixed closing logic nodes at the end of the execution plan sequence to uniformly clean up the experimental sandbox, services, and temporary directories. Once selected as the current execution node, the system will no longer execute subsequent rounds of testing and evaluation steps.

[0090] Specifically, the parent orchestration reads the evaluation request data bound to the current test instance and extracts the internal evaluation rule configuration. It then parses out the total number of evaluators to be launched and the evaluator scheduling mode (parallel / serial), while recording the evaluation type corresponding to each evaluator. Based on the obtained number of evaluators, the parent orchestration generates an equal number of unique evaluation execution attempt identifiers in batches. Each identifier uniquely corresponds to an independent evaluator, and each attempt is isolated from the others. For each evaluation execution attempt, the parent orchestration separately encapsulates a dedicated frozen runtime payload. The payload binds the storage index of the test execution attempt, the corresponding evaluation type, and the evaluation calculation rules. It then distributes these payloads to the corresponding evaluation sub-executions according to the scheduling mode: if parallel scheduling is used, all evaluation sub-executions start synchronously and concurrently read the repair data to perform evaluation; if serial scheduling is used, the next evaluation payload is distributed according to the preset order of the evaluation rules, after the previous evaluation sub-execution is completed and the evaluation result is persisted. Each evaluation sub-execution reads the frozen runtime load, accesses the tested independent storage area to read repair data, and performs calculations according to the evaluation method corresponding to its own evaluation type. Upon completion, it generates a structured evaluation result and associates it with its own evaluation execution attempt identifier for persistent storage. The parent orchestration waits for all evaluation execution attempts in its group to complete a closed loop, reads all persistent evaluation results, performs joint / weighted matching operations according to the preset threshold conditions in the evaluation rules, and outputs a unique judgment result (overall pass / overall fail). If the judgment result is an overall pass: the resource recycling node is not set as the current execution node, and the parent orchestration selects the next round of tested execution nodes in the execution plan as the new current execution node, initiating multiple rounds of progressive evaluation. If the judgment result is an overall fail: the resource recycling node is directly determined as the current execution node, and all unexecuted downstream multi-round tested and evaluation nodes are uniformly marked as not to execute, blocking subsequent evaluation processes.

[0091] Based on the evaluation rules in the evaluation request data, the parent orchestration determines the number of evaluators and the scheduling mode, including serial scheduling and parallel scheduling. It generates evaluation execution attempts corresponding to the number of evaluators, generates frozen runtime payloads for each evaluation execution attempt, and sends them to the corresponding sub-executions. Each sub-execution then evaluates the repaired data according to the evaluation method corresponding to the evaluation type, obtaining evaluation results. Evaluation types include unit test evaluation and code quality evaluation. The evaluation results are matched with preset threshold conditions to obtain a judgment result. Based on the judgment result, it determines whether to designate the resource reclamation node as the current execution node. Threshold matching is performed based on persistent evaluation results. After fault recovery, the judgment logic is unbiased, and there is no issue of process loss leading to erroneous process jumps, resulting in stronger fault tolerance.

[0092] Optionally, the current execution node is a resource reclamation node. The sub-execution performs corresponding operations based on the frozen running load to obtain sub-execution information, including: generating resource reclamation commands based on the frozen running load, so that the sub-execution reclaims the corresponding resources according to the resource reclamation commands, and obtaining resource reclamation results. The resource reclamation commands include: environment reclamation commands, service shutdown commands, temporary working directory reclamation commands, and temporary storage mount reclamation commands; and associating storage with the execution attempt identifier based on the resource reclamation results.

[0093] Specifically, the sub-execution reads all experimental-level resource reference handles stored within the frozen runtime payload, and matches corresponding recycling instruction templates according to resource type: generating an environment recycling command based on sandbox environment references; generating a service shutdown command based on background service process identifiers and listening ports; generating a temporary working directory recycling command based on the agent and the evaluation runtime temporary directory path; generating a temporary storage mounting recycling command based on the mounted storage address; and summarizing to form a complete set of resource recycling commands. The sub-execution executes the recycling commands one by one; for each command executed, it synchronously collects operation return codes, runtime logs, and exception reports; after all commands are executed, it summarizes the recycling status of all single-type resources, fault information, and a list of remaining unreleased resources, and encapsulates them to generate standardized resource recycling results. If the recycling of a single type of resource fails, the fault stack is fully retained without blocking the recycling process of other resources. The sub-execution establishes a unique association between the complete structured resource recycling results and the current resource recycling execution attempt identifier; the structured recycling metadata is written to the business database, and the full log of the recycling operation is archived to the dedicated storage path of the execution attempt; the data is permanently retained after being solidified, serving as audit evidence for the completion of the resource lifecycle of the experimental instance.

[0094] By generating resource reclamation commands based on the frozen running load through sub-executions, the corresponding resources are reclaimed according to the resource reclamation commands, and the resource reclamation results are obtained. The resource reclamation commands include: environment reclamation command, service shutdown command, temporary working directory reclamation command, and temporary storage mounting reclamation command. Based on the resource reclamation results and the execution attempt identifier, storage is associated. Regardless of whether the evaluation process is successful, fails, fails the evaluation, or the retry is exhausted, standardized resource release will be triggered. This solves the shortcomings of existing technologies, such as the cleanup logic being scattered across multiple business branches, the omission of sandbox release in abnormal scenarios, long-term resource occupation caused by background services, and the depletion of cluster computing power and storage.

[0095] Example 3

[0096] Figure 3 This is a schematic diagram of a device for executing an evaluation task oriented towards intelligent agents, provided in Embodiment 3 of the present invention. This embodiment of the present invention is applicable to the execution of evaluation tasks oriented towards intelligent agents. The device can execute a method for executing evaluation tasks oriented towards intelligent agents, and can be implemented in hardware and / or software.

[0097] See Figure 3 The agent-oriented evaluation task execution device shown includes: an instance generation module 301, a parent arrangement determination module 302, a sequence generation module 303, a node filtering module 304, an information determination module 305, and an information generation module 306, wherein...

[0098] The instance generation module 301 is used to generate an evaluation task identifier and at least one instance identifier corresponding to the evaluation task identifier and an experimental instance corresponding to each instance identifier based on the evaluation request data corresponding to the target user.

[0099] The parent arrangement determination module 302 is used to determine the parent arrangement corresponding to each test instance based on the idempotency of the parent arrangement of the test instance.

[0100] The sequence generation module 303 is used to generate an execution plan node sequence based on the test cases and evaluation request data through the parent orchestration. The execution plan node sequence includes at least one node to be executed.

[0101] The node filtering module 304 is used to filter each node to be executed to obtain the currently executing node;

[0102] The information determination module 305 is used to generate an execution attempt identifier and a frozen running payload for the current execution node, and send them to the sub-execution corresponding to the current execution node, so that the sub-execution can perform corresponding operations according to the frozen running payload and obtain sub-execution information;

[0103] The information generation module 306 is used to generate evaluation task information corresponding to the evaluation task identifier based on the sub-execution information corresponding to each test instance.

[0104] The technical solution of this invention generates an evaluation task identifier and at least one instance identifier corresponding to the evaluation task identifier, as well as test instances corresponding to each instance identifier, based on the evaluation request data corresponding to the target user. For each test instance, the parent orchestration corresponding to the test instance is determined according to the idempotent status of the parent orchestration. The parent orchestration generates an execution plan node sequence based on the test instance and the evaluation request data, and the execution plan node sequence includes at least one node to be executed. Each node to be executed is filtered to obtain the current execution node. An execution attempt identifier and a frozen running payload are generated for the current execution node and sent to the sub-execution corresponding to the current execution node so that the sub-execution performs corresponding operations according to the frozen running payload to obtain sub-execution information. Based on the sub-execution information corresponding to each test instance, evaluation task information corresponding to the evaluation task identifier is generated. By uniformly holding the complete execution plan through the test-level parent orchestration, the inconsistency problem caused by the creation of multiple scattered side effects in subsequent tasks is reduced, and the accuracy of the evaluation task execution for intelligent agents is improved.

[0105] Optionally, the parent arrangement determination module 302 is specifically used for:

[0106] Based on the evaluation request data, determine the evaluation process type corresponding to the test case;

[0107] Based on the instance identifier and evaluation process type corresponding to the test instance, determine the idempotent status of the parent arrangement corresponding to the test instance;

[0108] Repeatedly trigger the verification of the parent's idempotent identity to obtain the verification result;

[0109] Based on the verification results, determine the parent arrangement corresponding to the test instance.

[0110] Optionally, the agent-oriented evaluation task execution device is also specifically used for:

[0111] After generating an execution attempt identifier and freezing runtime payload for the current execution node and sending them to the corresponding sub-execution of the current execution node, so that the sub-execution can perform corresponding operations according to the freezing runtime payload and obtain the sub-execution information, if the sub-execution result included in the sub-execution information is execution failure, the number of retries is determined according to the evaluation request data. If the number of retries is not zero, a current attempt identifier is generated so that the sub-execution can re-execute the corresponding operations according to the freezing runtime payload and obtain the sub-execution information. If the number of retries is zero, a non-execution identifier is generated for at least one associated node corresponding to the current execution node through the parent orchestration, and the resource reclamation node is determined as the current execution node.

[0112] When the sub-execution result included in the sub-execution information is successful, the sub-execution information and execution attempt identifier are stored together.

[0113] Optionally, the information determination module 305 is specifically used for:

[0114] The current execution node is the environment preparation node. The sub-execution configures the current execution environment based on the frozen running load and obtains the environment configuration result.

[0115] Based on the experimental examples, at least one service to be started is identified, and each service to be started is started in the current execution environment to obtain the startup results of each service to be started.

[0116] The environment reference corresponding to the current execution environment, the startup results of each service to be started, the environment configuration results, and the execution attempt identifier are associated and stored.

[0117] Optionally, the information determination module 305 is specifically used for:

[0118] The current execution node is the execution node under test. The sub-execution generates the data to be evaluated based on the frozen running load and stores the data to be evaluated in an independent storage area.

[0119] By sub-execution, the evaluation agent corresponding to the frozen running payload is loaded in the current execution environment so that the evaluation agent can repair the data to be evaluated, obtain the repaired data and repair results, and store the repaired data in an independent storage area;

[0120] The independent storage area contains region data, repair results, acquired information, and execution attempt identifiers, which are stored together. The acquired information includes repair logs, session files, and trajectory files.

[0121] Optionally, the information determination module 305 is specifically used for:

[0122] The current execution node is the judging node. The number of judges and the scheduling mode are determined by the parent orchestration based on the judging rules in the judging request data. The scheduling modes include serial scheduling and parallel scheduling.

[0123] Generate a number of evaluation execution attempts corresponding to the number of evaluators;

[0124] Generate a frozen runtime payload corresponding to each evaluation execution attempt and send it to the sub-execution corresponding to each evaluation execution attempt, so that each sub-execution evaluates the repair data according to the evaluation method corresponding to the evaluation type according to the scheduling mode, and obtains the evaluation result. The evaluation types include: unit test evaluation and code quality evaluation.

[0125] The evaluation results are matched with preset threshold conditions to obtain the final judgment result;

[0126] Based on the judgment result, determine whether to designate the resource recycling node as the current execution node.

[0127] Optionally, the information determination module 305 is specifically used for:

[0128] The current execution node is a resource reclamation node. It generates resource reclamation commands through sub-executions based on the frozen running load, so that the sub-executions can reclaim the corresponding resources according to the resource reclamation commands and obtain the resource reclamation results. The resource reclamation commands include: environment reclamation command, service shutdown command, temporary working directory reclamation command, and temporary storage mount reclamation command.

[0129] The storage is associated with the resource recycling results and the execution attempt identifier.

[0130] The agent-oriented evaluation task execution device provided in this embodiment of the invention can execute the agent-oriented evaluation task execution method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the agent-oriented evaluation task execution method.

[0131] Example 4

[0132] Figure 4A schematic diagram of an agent-oriented evaluation task execution device 400, which can be used to implement embodiments of the present invention, is shown.

[0133] like Figure 4 As shown, the agent-oriented evaluation task execution device 400 includes at least one processor 401 and a memory, such as a read-only memory (ROM) 402 and a random access memory (RAM) 403, communicatively connected to the at least one processor 401. The memory stores computer programs executable by the at least one processor. The processor 401 can perform various appropriate actions and processes based on the computer program stored in the ROM 402 or loaded from storage unit 408 into the RAM 403. The RAM 403 can also store various programs and data required for the operation of the agent-oriented evaluation task execution device 400. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0134] Multiple components in the agent-oriented evaluation task execution device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, optical disk, etc.; and a communication unit 409, such as a network interface card, modem, wireless transceiver, etc. The communication unit 409 allows the agent-oriented evaluation task execution device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] Processor 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 401 performs the various methods and processes described above, such as methods for performing evaluation tasks for intelligent agents.

[0136] In some embodiments, the agent-oriented evaluation task execution method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the agent-oriented evaluation task execution device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by processor 401, one or more steps of the agent-oriented evaluation task execution method described above can be performed. Alternatively, in other embodiments, processor 401 can be configured to execute the agent-oriented evaluation task execution method by any other suitable means (e.g., by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on an agent-oriented evaluation task execution device, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the agent-oriented evaluation task execution device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0141] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0142] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0143] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0144] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for performing evaluation tasks for intelligent agents, characterized in that, The method includes: Based on the evaluation request data corresponding to the target user, an evaluation task identifier and at least one instance identifier corresponding to the evaluation task identifier and an experimental instance corresponding to each instance identifier are generated. For each of the aforementioned test instances, the parent arrangement corresponding to the test instance is determined based on the idempotent status of the parent arrangement of the test instance; Based on the test instance and the evaluation request data, the parent orchestration generates an execution plan node sequence, which includes at least one node to be executed. By filtering the nodes to be executed, the current execution node is obtained; An execution attempt identifier and a frozen running payload are generated for the current execution node and sent to the sub-execution corresponding to the current execution node, so that the sub-execution performs corresponding operations according to the frozen running payload and obtains sub-execution information; Based on the sub-execution information corresponding to each of the aforementioned test instances, the evaluation task information corresponding to the evaluation task identifier is generated.

2. The method according to claim 1, characterized in that, The step of determining the parent arrangement corresponding to the test instance based on the idempotent status of the parent arrangement includes: Based on the evaluation request data, determine the evaluation process type corresponding to the test instance; Based on the instance identifier corresponding to the test instance and the evaluation process type, determine the parent arrangement idempotent status corresponding to the test instance; Repeatedly trigger verification of the parent orchestration's idempotent identity to obtain the verification result; Based on the verification results, determine the parent arrangement corresponding to the test instance.

3. The method according to claim 1, characterized in that, After generating an execution attempt identifier and a frozen runtime payload for the current execution node, and sending them to the sub-execution corresponding to the current execution node so that the sub-execution performs corresponding operations based on the frozen runtime payload and obtains the sub-execution information, the method further includes: When the sub-execution result included in the sub-execution information is execution failure, the number of retry attempts is determined according to the evaluation request data. When the number of retry attempts is not zero, a current attempt identifier is generated so that the sub-execution can re-execute and perform the corresponding operation according to the frozen running load to obtain the sub-execution information. When the number of retry attempts is zero, a non-execution identifier is generated for at least one associated node corresponding to the current execution node through the parent orchestration, and the resource reclamation node is determined as the current execution node. When the sub-execution result included in the sub-execution information is successful, the sub-execution information and the execution attempt identifier are stored together.

4. The method according to claim 1, characterized in that, The current execution node is the environment preparation node. The sub-execution performs corresponding operations based on the frozen running load to obtain sub-execution information, including: The sub-execution configures the current execution environment based on the frozen running load, and obtains the environment configuration result. Based on the experimental example, at least one service to be started is determined, and each service to be started is started in the current execution environment to obtain the startup result corresponding to each service to be started. The environment reference corresponding to the current execution environment, the startup results of each service to be started, the environment configuration results, and the execution attempt identifier are associated and stored.

5. The method according to claim 4, characterized in that, The current execution node is the execution node under test. The sub-execution performs corresponding operations based on the frozen running load to obtain sub-execution information, including: The sub-execution generates the evaluation data based on the frozen running load and stores the evaluation data in an independent storage area; The evaluation agent corresponding to the frozen running load is loaded in the current execution environment by the sub-execution, so that the evaluation agent can repair the data to be evaluated, obtain the repaired data and the repair results, and store the repaired data in the independent storage area; The independent storage area includes regional data, repair results, acquired collection information, and execution attempt identifiers, which are associated and stored together. The acquired information includes: repair logs, session files, and trajectory files.

6. The method according to claim 5, characterized in that, The current execution node is the evaluation node. An execution attempt identifier and a frozen runtime payload are generated for the current execution node and sent to the sub-execution corresponding to the current execution node. This allows the sub-execution to perform corresponding operations based on the frozen runtime payload and obtain sub-execution information, including: The parent orchestration determines the number of evaluators and the scheduling mode based on the evaluation rules in the evaluation request data, wherein the scheduling mode includes serial scheduling and parallel scheduling. Generate the evaluation execution attempts corresponding to the number of evaluators; A frozen runtime payload corresponding to each evaluation execution attempt is generated and sent to the sub-execution corresponding to each evaluation execution attempt, so that each sub-execution evaluates the repair data according to the evaluation method corresponding to the evaluation type according to the scheduling mode, and obtains the evaluation result. The evaluation type includes: unit test evaluation and code quality evaluation. The evaluation result is matched with the preset threshold conditions to obtain the judgment result; Based on the judgment result, determine whether to designate the resource recycling node as the current execution node.

7. The method according to claim 1, characterized in that, The current execution node is a resource reclamation node. The sub-execution performs corresponding operations based on the frozen running load to obtain sub-execution information, including: The sub-execution generates a resource reclamation command based on the frozen running load, so that the sub-execution reclaims the corresponding resources according to the resource reclamation command and obtains the resource reclamation result. The resource reclamation command includes: environment reclamation command, service shutdown command, temporary working directory reclamation command and temporary storage mount reclamation command. The resource recovery results are associated with the execution attempt identifier and stored accordingly.

8. A device for executing evaluation tasks for intelligent agents, characterized in that, The device includes: The instance generation module is used to generate an evaluation task identifier, at least one instance identifier corresponding to the evaluation task identifier, and test instances corresponding to each instance identifier based on the evaluation request data corresponding to the target user. The parent arrangement determination module is used to determine the parent arrangement corresponding to each test instance based on the idempotency of the parent arrangement of the test instance. The sequence generation module is used to generate an execution plan node sequence based on the test instance and the evaluation request data through the parent orchestration, wherein the execution plan node sequence includes at least one node to be executed; The node filtering module is used to filter the nodes to be executed to obtain the currently executing node; The information determination module is used to generate an execution attempt identifier and a frozen running payload for the current execution node, and send them to the sub-execution corresponding to the current execution node, so that the sub-execution performs corresponding operations according to the frozen running payload to obtain sub-execution information; The information generation module is used to generate evaluation task information corresponding to the evaluation task identifier based on the sub-execution information corresponding to each of the test instances.

9. A device for executing evaluation tasks for intelligent agents, characterized in that, The device for executing the evaluation task for intelligent agents includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the agent-oriented evaluation task execution method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the agent-oriented evaluation task execution method according to any one of claims 1-7.

Images