Intelligent agent comprehensive evaluation method, device, storage medium and product

Abstract

The application discloses an agent comprehensive evaluation method and device, a storage medium and a product, relates to the technical field of agent evaluation and optimization, and the agent comprehensive evaluation method comprises the following steps: obtaining an evaluation task configuration, determining an agent to be evaluated sample and a target evaluation operator, evaluating the agent based on the target evaluation operator, an agent driving tool and the agent to be evaluated sample, and obtaining a structured evaluation result. The agent to be evaluated sample comprises running process data of the agent, and the target evaluation operator is obtained by debugging and optimizing based on the running process data, business documents and scoring items. Therefore, the running process data containing intermediate data and judgment links can be comprehensively evaluated by using the evaluation operator. The entire running process is evaluated by using the evaluation operator, and a structured evaluation result is output, so that the agent comprehensive evaluation method can obtain a comprehensive evaluation result without checking the intermediate process of the agent step by step, and the efficiency of agent evaluation can be improved.

Description

Technical Field

[0001] This application relates to the field of intelligent agent evaluation and optimization technology, and in particular to methods, devices, storage media and products for comprehensive evaluation of intelligent agents. Background Technology

[0002] As enterprise-level intelligent agent applications evolve from single-round question-and-answer to long-process planning, multi-tool invocation, and cross-system collaboration, traditional evaluation methods are no longer sufficient to support formal quality governance. Therefore, a new intelligent agent evaluation method is needed to evaluate intelligent agents.

[0003] Current agent evaluation methods typically involve manual sampling, single-prompt scoring, or evaluation using ad-hoc scripts. These methods can only determine whether the final answer is satisfactory, while quality issues in complex agents often occur in intermediate steps or decision chains. Therefore, current agent evaluation methods cannot provide a comprehensive assessment of the agent, forcing testing and development teams to review each intermediate step of the agent's process, resulting in low efficiency in agent evaluation. Summary of the Invention

[0004] The main objective of this application is to provide a method, device, storage medium, and product for comprehensive evaluation of intelligent agents, aiming to solve the technical problem of low efficiency in current intelligent agent evaluation.

[0005] To achieve the above objectives, this application proposes a comprehensive evaluation method for intelligent agents, the method comprising: Obtain the assessment task configuration; Based on the task configuration, the agent's evaluation sample and target evaluation operator are obtained, wherein the evaluation sample includes the agent's operation process data, and the target evaluation operator is obtained by debugging and optimizing the agent's operation process data, business documents, and scoring items; Based on the target evaluation operator, the preset intelligent agent driving tool, and the sample to be evaluated, the intelligent agent is evaluated to obtain a structured evaluation result, wherein the structured evaluation result includes the evaluation results of each of the operation process data.

[0006] In one embodiment, before the step of obtaining the agent's evaluation sample and target evaluation operator based on the task configuration, the method further includes: Obtain samples of the agent's operational process, business documents, and scoring items; Based on the operation process samples, the business document samples, and the scoring item samples, a training set and a validation set are generated. Under the operator creation constraints of the intelligent agent driving tool, an initial operator is created, including evaluation instructions, reference materials, tool capabilities, output protocols, and operation traceability information; Based on the training set, the validation set, and the intelligent agent driving tool, the initial operator is debugged and optimized to obtain the target evaluation operator.

[0007] In one embodiment, the step of debugging and optimizing the initial operator based on the training set, the validation set, and the intelligent agent driving tool to obtain the target evaluation operator includes: Use the initial operator as the current operator to be optimized, and obtain the debugging task for the operator to be optimized; Based on the intelligent agent driving tool, the debugging task, and the training set, the operator to be optimized is optimized to obtain the optimized operator; Based on the verification set, the optimization operator is verified, and based on the obtained verification results, it is determined whether the optimization operator meets the release conditions. If the release conditions are not met, the optimized operator is taken as a new operator to be optimized, and the process returns to the step of obtaining the debugging task for the operator to be optimized, until the optimized operator meets the release conditions and the target evaluation operator is obtained.

[0008] In one embodiment, the step of optimizing the operator to be optimized based on the intelligent agent driving tool, the debugging task, and the training set to obtain the optimized operator includes: The debugging task is performed based on the intelligent agent driving tool, and the running process samples in the training set are evaluated to obtain sample evaluation results; Based on the sample evaluation results and the real results in the training set, the source of the problem of the operator to be optimized is determined; Based on the source of the problem, the operator to be optimized is optimized in a targeted manner to obtain the optimized operator.

[0009] In one embodiment, the sample evaluation results include debugging results, failed samples, sample scores, and expected labels; the real results include real scores and real labels; and the step of determining the source of the problem of the operator to be optimized based on the sample evaluation results and the real results in the training set includes: Based on the sample scores and the actual scores, the consistency of the scores between the operator to be optimized and the actual results is determined; Based on the debugging results, the failed samples, the expected labels, and the actual labels, the sources of problems for the operator to be optimized are determined. The sources of problems include at least one of the following: unclear expression of scoring rules, insufficient reference materials, incorrect trajectory field parsing, unstable output protocol, and overly coarse error classification.

[0010] In one embodiment, the step of evaluating the agent based on the target evaluation operator, a preset agent driving tool, and the sample to be evaluated to obtain a structured evaluation result includes: Based on the aforementioned intelligent agent driving tool, reference materials are determined that are bound to the target evaluation operator; Based on the intelligent agent driving tool, the reference materials, the operation process data in the sample to be evaluated, and the version of the target evaluation operator are combined into a standard context; Based on the target evaluation operator and the standard context, the agent is evaluated to obtain multiple corresponding evaluation results, wherein the evaluation results include sample score, pass label, judgment reason, evidence citation, misclassification, running time and abnormal state; Based on the aforementioned intelligent agent driving tool, the evaluation results are combined into the structured evaluation results.

[0011] In one embodiment, after the step of evaluating the agent based on the target evaluation operator, a preset agent driving tool, and the sample to be evaluated to obtain a structured evaluation result, the method further includes: Obtain new low-scoring samples, manual calibration results, and accuracy reports generated during the formal operation of the target evaluation operator; Create a backup operator identical to the target evaluation operator; The backup operator is optimized based on the new low-scoring samples, manual calibration results, and accuracy reports to obtain an optimized new evaluation operator.

[0012] Furthermore, to achieve the above objectives, this application also proposes a comprehensive agent evaluation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the comprehensive agent evaluation method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the intelligent agent comprehensive evaluation method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the comprehensive evaluation method for intelligent agents as described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: Current agent evaluation methods can only determine whether the final answer is qualified, and cannot comprehensively evaluate the agent. This forces testing and R&D teams to review the intermediate processes of the agent step by step, resulting in low efficiency in agent evaluation. Therefore, this application obtains an evaluation task configuration, acquires the agent's evaluation sample and target evaluation operator based on the task configuration, and evaluates the agent based on the target evaluation operator, a preset agent control tool, and the evaluation sample to obtain a structured evaluation result. Since the evaluation sample in this application includes the agent's operational process data, and the target evaluation operator used for evaluation is obtained by debugging and optimizing the operational process data, business documents, and scoring items, this application can comprehensively evaluate the operational process data, including intermediate data and judgment links, through the evaluation operator.

[0016] Since this application evaluates the entire operation process of the intelligent agent through evaluation operators, Furthermore, the output is a structured evaluation result that includes the evaluation of each of the aforementioned operational process data, which allows this application to obtain a comprehensive evaluation result without having to review each of the intermediate processes of the agent, thereby improving the efficiency of agent evaluation. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the intelligent agent comprehensive evaluation method of this application. Figure 2 This is a schematic diagram of the overall process of the agent evaluation operator provided in Embodiment 1 of the agent comprehensive evaluation method of this application; Figure 3 A schematic diagram of the operator's full lifecycle process provided in Embodiment 1 of the intelligent agent comprehensive evaluation method of this application; Figure 4 This is a schematic diagram of continuous optimization of an intelligent agent provided in Embodiment 1 of the intelligent agent comprehensive evaluation method of this application; Figure 5 This is a flowchart illustrating Embodiment 2 of the intelligent agent comprehensive evaluation method of this application; Figure 6 This is a schematic diagram of the initial operator structure provided in Embodiment 2 of the intelligent agent comprehensive evaluation method of this application; Figure 7 This is a schematic diagram of operator debugging and optimization provided in Embodiment 2 of the intelligent agent comprehensive evaluation method of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the comprehensive evaluation method of intelligent agents in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or intelligent agent comprehensive evaluation device capable of the above functions. The following description uses an intelligent agent comprehensive evaluation device as an example to illustrate this embodiment and the subsequent embodiments.

[0024] As enterprise-level intelligent agent applications evolve from single-round question-and-answer to long-process planning, multi-tool invocation, and cross-system collaboration, traditional evaluation methods are no longer sufficient to support formal quality governance. Therefore, a new intelligent agent evaluation method is needed to evaluate intelligent agents.

[0025] Current agent evaluation methods typically involve manual sampling, single-prompt scoring, or evaluation using ad-hoc scripts. These methods can only determine whether the final answer is satisfactory, while quality issues in complex agents often occur in intermediate steps or decision chains. Therefore, current agent evaluation methods cannot provide a comprehensive assessment of the agent, forcing testing and development teams to review each intermediate step of the agent's process, resulting in low efficiency in agent evaluation.

[0026] Based on this, embodiments of this application provide a comprehensive evaluation method for intelligent agents, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the intelligent agent comprehensive evaluation method of this application.

[0027] In this embodiment, the comprehensive evaluation method for the intelligent agent includes steps S10 to S30: Step S10: Obtain the evaluation task configuration; It should be noted that the assessment task configuration contains all the settings required to conduct this assessment, and the subsequent assessment process will use this configuration as the unified basis to determine how to conduct the assessment.

[0028] In its implementation, when an evaluation task is initiated, the platform reads and parses the configuration corresponding to this evaluation task, extracts the various settings recorded in the evaluation task configuration, and loads them into the execution flow of this evaluation for reference in subsequent evaluation stages. The overall flow of the intelligent agent evaluation operator in this embodiment can be referred to... Figure 2 .

[0029] Step S20: Based on the task configuration, obtain the sample to be evaluated and the target evaluation operator of the agent, wherein the sample to be evaluated includes the operation process data of the agent, and the target evaluation operator is obtained by debugging and optimization based on the operation process data of the agent, business documents and scoring items; It should be noted that the sample to be evaluated is the sample data used to evaluate the agent in this evaluation, including the agent's operational process data, which can reflect the actual operation of the agent in performing tasks. The target evaluation operator is the evaluation operator used in this evaluation. It is an evaluation operator obtained after debugging and optimization based on the agent's operational process data, business documents, and scoring items, and can be used to evaluate the evaluation object.

[0030] Operational process data refers to the data generated by the agent during operation, recording the process of the agent executing tasks and reflecting the process and methods the agent went through to complete the task. Business documents are documents related to the business handled by the agent, recording the requirements and specifications of the corresponding business domain, and providing a business-level reference for evaluating the agent's performance. Scoring items are specific items used to evaluate and score the evaluation object, specifying the aspects to be examined during the evaluation and the corresponding judgment requirements.

[0031] It is understood that this embodiment obtains the evaluation sample of the intelligent agent based on task configuration, and the evaluation sample includes the intelligent agent's operation process data. This ensures that the evaluation object originates from the real data during the intelligent agent's operation, allowing the evaluation to focus on the actual operation of the intelligent agent. Furthermore, the target evaluation operator obtained in this embodiment is an evaluation operator obtained after debugging and optimization based on the intelligent agent's operation process data, business documents, and scoring items. The target evaluation operator, in its formation process, integrates the business requirements recorded in the business documents and the scoring items specified in the scoring items. This ensures that the evaluation is based on the actual operation process data of the intelligent agent, and the target evaluation operator, after debugging and optimization based on the business documents and scoring items, performs the judgment, guaranteeing that the evaluation result accurately reflects the actual performance of the intelligent agent.

[0032] Step S30: Based on the target evaluation operator, the preset intelligent agent driving tool, and the sample to be evaluated, the intelligent agent is evaluated to obtain a structured evaluation result, wherein the structured evaluation result includes the evaluation results of each of the operation process data.

[0033] It should be noted that the intelligent agent driving tool is a pre-set tool that provides operational support and normative constraints for the evaluation process when evaluating intelligent agents. It is responsible for organizing the context required for evaluation, driving the evaluation execution, and collecting the data and records generated during the evaluation process, so that the evaluation can be carried out under a unified operational framework.

[0034] Structured evaluation results are evaluation results produced after the agent has completed the evaluation, arranged in a unified data organization method. The content includes the evaluation results of each operational process data, which can be directly parsed, recorded and reused by the platform.

[0035] In its implementation, the agent-driven tool plays multiple roles in evaluating the operator's lifecycle. During the operator creation phase, it is responsible for standardizing temporary configurations into runnable artifacts. During the debugging and running phase, it serves as the actual runtime foundation, responsible for context assembly, runner selection, and tool invocation. During the result collection phase, it unifies the output, trace, error codes, and gap analysis into structured evidence. In the pre-release phase, it acts as a threshold verification layer, determining whether the configuration meets the conditions for entering the formal state. During the iterative closed-loop phase, it continues to receive new samples, new evidence, and further debugging, forming a continuous optimization loop.

[0036] Specifically, during the creation phase, SKILL (the operator) injects Skill creation specifications through Agent Harness, constraining directory organization, file naming, input / output, Reference usage, tool call boundaries, and debugging / acceptance conditions. The generated content must revolve around the evaluation task, avoiding the introduction of generalizations unrelated to scoring and refusing to rely on unregistered data sources in the formal evaluation.

[0037] During the debugging phase, Harness is responsible for organizing a debugging task into a controllable and verifiable execution process. It constructs a debugging context based on real-world samples, splits the samples into training and validation parts, generates sample selection hashes and partition seeds to ensure reproducibility, and differentiates execution flows according to the execution objectives such as training, validation, and deployment access control. Regarding sample visibility, Harness implements a hidden oracle (standard answer) strategy, masking the target score, labels, and validation set details, ensuring that Skill can only score based on visible trajectory inputs, observation summaries, and references, thus preventing answer leakage.

[0038] During runtime, Agent Harness reads the samples to be evaluated based on the task configuration, assembles the runtime trajectory, binding references, and operator versions into a standard evaluation context, and then calls the Skill to complete the scoring. The runtime results are written back in a unified structure, including sample-level scores, pass / fail labels, judgment reasons, evidence citations, error classifications, runtime, and abnormal states. Since operator versions are frozen, the same version can be reviewed for its evaluation basis under the same inputs and configurations, facilitating quality reports, issue tracking, and historical comparisons. The entire lifecycle of the agent operator in this embodiment can be referenced... Figure 3 .

[0039] It is understood that, under the operational support provided by the preset intelligent agent driving tool, this embodiment evaluates the sample to be evaluated based on the target evaluation operator. The evaluation is carried out by the debugged and optimized target evaluation operator, and the evaluation process is organized by the preset intelligent agent driving tool. Thus, the evaluation of the intelligent agent can be performed in a standardized operating environment, reducing the interference caused by temporary human intervention during the evaluation process and improving the standardization of the evaluation.

[0040] Furthermore, since the structured evaluation results include evaluation results for each of the operational process data, the structured evaluation results record evaluation conclusions for each operational process data under a unified evaluation method, so that the evaluation results for each operational process data can be viewed through the structured evaluation results.

[0041] In one feasible implementation, the specific implementation of evaluating the agent based on the target evaluation operator, a preset agent driving tool, and the sample to be evaluated to obtain a structured evaluation result can also be: Based on the aforementioned intelligent agent driving tool, reference materials are determined that are bound to the target evaluation operator; Based on the intelligent agent driving tool, the reference materials, the operation process data in the sample to be evaluated, and the version of the target evaluation operator are combined into a standard context; Based on the target evaluation operator and the standard context, the agent is evaluated to obtain multiple corresponding evaluation results, wherein the evaluation results include sample score, pass label, judgment reason, evidence citation, misclassification, running time and abnormal state; Based on the aforementioned intelligent agent driving tool, the evaluation results are combined into the structured evaluation results.

[0042] It should be noted that the reference materials are a collection of data bound to the target evaluation operator, used to provide a reference basis for the evaluation scoring judgment. This includes scoring agreements, business rules, operational trajectory structures, typical positive and negative examples, and historical review records, all of which can be used as scoring criteria. The standard context is the evaluation input set obtained by organizing the reference materials, operational process data from the sample to be evaluated, and the version of the target evaluation operator in a unified combination method. It includes all the input information required for a single evaluation, ensuring that the evaluation process is executed under a unified input structure.

[0043] Version is a number used to identify a specific release state of the target evaluation operator. Different numbers correspond to a fixed form of the target evaluation operator under a specific degree of debugging and optimization, used to specify the specific operator form referenced during evaluation. Evaluation results are the information obtained after completing an evaluation of a single sample to be evaluated, including sample score, pass / fail label, reason for judgment, evidence cited, error classification, runtime, and abnormal status.

[0044] The sample score is the numerical rating given to a single sample in the evaluation results. The label is the information used to identify whether a single sample passed the evaluation. The reasoning behind the judgment is the explanatory content in the evaluation results that explains the basis for the scoring judgment, providing for the interpretation and verification of the evaluation conclusion. Evidence citation is the citation information in the evaluation results that points to the evidence supporting the scoring judgment, making the basis for the judgment traceable and verifiable. Error classification is the information in the evaluation results that categorizes the errors found in the evaluation, indicating the category to which the problem belongs. Runtime is the time spent on this evaluation recorded in the evaluation results. Abnormal status is the status information recorded in the evaluation results regarding any abnormal situations that occurred during the evaluation execution process.

[0045] It is understood that, before the evaluation in this embodiment, the intelligent agent driving tool determines the reference materials bound to the target evaluation operator, and combines the reference materials, the running process data in the sample to be evaluated, and the version of the target evaluation operator into a standard context. The input materials required for the evaluation are uniformly included in the standard context with a fixed structure and source. The form of the target evaluation operator used during the evaluation is also clearly defined by the version. Thus, each evaluation is performed on a standard context with the same input structure and a defined basis. The same version of the target evaluation operator, combined with the same standard context, can produce a reproducible evaluation process. The evaluation input is complete and consistent, which facilitates the review and comparison of the evaluation basis.

[0046] Furthermore, since the evaluation results include sample scores, pass labels, judgment reasons, evidence citations, error classifications, running time, and abnormal states, and the intelligent agent driving tool combines the various evaluation results into a structured evaluation result, the evaluation output retains process information such as judgment reasons, evidence citations, error classifications, running time, and abnormal states while providing sample scores and pass labels. It is summarized into a structured evaluation result in a unified structure. Therefore, the evaluation conclusion can not only reflect whether the sample is qualified, but also explain the judgment basis, trace the source of evidence, indicate the error category, and reflect the running anomaly. This makes the evaluation results interpretable and verifiable, and improves the comprehensiveness of the evaluation.

[0047] In one feasible implementation, the specific implementation following the step of evaluating the agent based on the target evaluation operator, a preset agent driving tool, and the sample to be evaluated to obtain a structured evaluation result can also be: Obtain new low-scoring samples, manual calibration results, and accuracy reports generated during the formal operation of the target evaluation operator; Create a backup operator identical to the target evaluation operator; The backup operator is optimized based on the new low-scoring samples, manual calibration results, and accuracy reports to obtain an optimized new evaluation operator.

[0048] It should be noted that formal operation refers to the phase where the target evaluation operator, after leaving the debugging phase, is deployed in a released state and continuously performs evaluations in actual evaluation tasks. New low-scoring samples are low-scoring samples generated during the formal operation of the target evaluation operator; they are used to identify specific samples that performed poorly in the evaluation and constitute the targets for subsequent optimization. Manual calibration results are calibration information obtained after manual review of the evaluation judgments, recording human confirmation or correction opinions on the evaluation conclusions, used to ensure consistency between the evaluation conclusions and human judgments. The accuracy report is a report generated based on the comparison between the evaluation results and the actual results, used to summarize and reflect the accuracy of the evaluation.

[0049] A backup operator is a copy of the target evaluation operator, used to undergo subsequent optimizations and adjustments while ensuring that the operators used in actual operation remain unaffected. A new evaluation operator is an optimized version of the backup operator, used to carry the optimized evaluation capabilities and form an operator form suitable for subsequent evaluations.

[0050] In the specific implementation, targeted revisions are made based on the current Skill (operator), debugging rounds, formal running results, error examples, manual review results, and supplementary references. The system first analyzes which scoring items, trajectory stages, and error types the failure samples are concentrated in, and then determines whether the problem originates from rule boundaries, data gaps, tool parsing, output protocols, or model instability. The operator continuous optimization process in this embodiment can be referenced. Figure 4 .

[0051] It is understandable that, since the new low-scoring samples, manual calibration results and accuracy reports obtained in this embodiment are all generated during the formal operation of the target evaluation operator, the inputs on which the optimization is based come from the feedback information actually formed in the actual evaluation. Therefore, the optimization of the evaluation operator can be carried out on the low-scoring samples and their calibration opinions and accuracy exposed in the formal operation, so that the optimization direction is in line with the performance of the target evaluation operator in the actual operation.

[0052] Furthermore, in this embodiment, a backup operator identical to the target evaluation operator is first created. Then, the backup operator is optimized based on new low-scoring samples, manual calibration results, and accuracy reports. The optimization process applies to the backup operator without affecting the target evaluation operator in the actual operation. Thus, during the optimization process, the target evaluation operator used in the actual operation maintains its original form and continues to perform evaluation stably. The optimization and adjustment are performed on the backup operator, thereby enabling continuous optimization during operation without changing the actual operating operator.

[0053] In one embodiment, the step of optimizing the backup operator based on the new low-scoring samples, manual calibration results, and accuracy reports to obtain the new evaluation operator further includes: Obtain information on the recurrence of the new low-scoring samples within a preset time range; Based on the recurrence information, the common low-scoring samples that appear repeatedly are selected from the new low-scoring samples. The common low-scoring samples are matched and analyzed with the reference data bound to the backup operator to determine the missing content of the reference data; Based on the common low-scoring samples, the manual calibration results, and the gap content, the backup operator is revised in a targeted manner to obtain the new evaluation operator.

[0054] It should be noted that recurrence information reflects whether new low-scoring samples reappear in multiple formal runs and the circumstances under which they do so. This information is used to distinguish between recurring systemic defects and accidental problems. In practice, the evaluation results retained from each formal run within a preset time frame can be sampled and frequency-counted. For example, low-scoring samples of the same type can be aggregated based on sample content characteristics or sample identifiers, and the batches and frequency of their occurrence can be counted to form recurrence information summarized by scoring item and occurrence circumstances.

[0055] Common low-scoring samples are those that repeatedly appear in multiple formal runs and are selected from new low-scoring samples. They are used to identify assessment defects that are repeatedly exposed but have not yet been resolved. In practice, new low-scoring samples within a preset time range can be deduplicated and merged according to their content characteristics, and low-scoring samples whose frequency of occurrence or batches of occurrence meets preset conditions are identified as common low-scoring samples.

[0056] The gap refers to the portion of the existing reference material that is insufficient to support a correct judgment on common low-scoring samples. It is used to determine the direction of content that needs to be supplemented to the reference material. In practice, the judgment points involved in the common low-scoring samples can be searched and matched with the content of the reference material one by one. Judgment points that do not have sufficient content support are regarded as objects that need to be supplemented, and the corresponding content can be further supplemented to the reference material through knowledge base retrieval or manual compilation.

[0057] Targeted revision is an optimization process that adjusts only the local content of the backup operator to address identified issues, without rebuilding the overall structure of the backup operator. In practice, appropriate local adjustment methods can be selected based on the aspect of the problem, such as refining the rule descriptions in the evaluation specifications, supplementing missing content in the references, correcting the format conventions in the output protocol, or refining the classification criteria for error categorization.

[0058] Understandably, the implementation first filters out recurring common low-scoring samples from the new low-scoring samples based on the recurrence information, and then determines the gap content through matching analysis between the common low-scoring samples and the reference data. Recurring problems and occasional problems are treated differently, and their corresponding defects along with data gaps are identified in a centralized manner. This can avoid the repeated consumption of optimization resources for the same type of problem in different rounds, reduce the number of iterations required to reach convergence, and improve the optimization efficiency of the evaluation operator.

[0059] Because the optimization of the backup operator is implemented through targeted revisions, adjusting only local aspects corresponding to common low-scoring samples, manual calibration results, and gap content, and because this optimization is performed on the backup operator while the confirmation of the new evaluation operator can still follow the aforementioned validation set verification and release condition control process, the target evaluation operator used in the official run remains stable. Furthermore, after the gap content is supplemented, the scoring judgment of similar low-scoring samples in subsequent evaluations has verifiable reference support. Combined with structured results such as evidence citations, this can suppress the recurrence of fixed defects in new versions, enabling the evaluation operator to achieve sustainable and convergent evolution while maintaining stable operation.

[0060] In one embodiment, the step of matching and analyzing the common low-scoring samples with the reference data bound to the backup operator to determine the missing content of the reference data further includes: Based on the evaluation results of the common low-scoring samples, extract the evidence requirement information on which the scoring judgment of each common low-scoring sample is based; The evidence requirement information is matched item by item with the searchable content of the reference materials to determine the support status of each piece of evidence requirement information. Based on the aforementioned support information, statistical information on evidence requirements that are not supported or are insufficiently supported is compiled, and the statistical results are merged according to the coverage of the common low-scoring samples to form priority gap content.

[0061] It should be noted that the evidence requirement information refers to the key evidence points needed to form a correct scoring judgment for common low-scoring samples. This information indicates which content supports the scoring judgment. In practice, the key evidence points relied upon for the judgment can be extracted from the evaluation results and operational data corresponding to the common low-scoring samples. For example, analyzing the judgment reasons in the evaluation results and the tool returns, business rules, or knowledge entries referenced by the evidence can help organize missing or questionable evidence points into structured evidence requirement information.

[0062] Support information reflects the degree to which each piece of evidence requirement is supported by the reference materials. It is used to determine whether the reference materials adequately support each piece of evidence requirement. In practice, each piece of evidence requirement can be converted into search criteria and searched within the content of the reference materials, with a relevance comparison. The support status of each item is then given according to the existence and relevance of supporting content, such as being categorized as adequately supported, partially supported, or not supported.

[0063] Coverage level reflects the number of common low-scoring samples associated with a particular evidence requirement, and is used to measure the scope of benefit from supplementing that evidence requirement in addressing common deficiencies. In practice, the number and proportion of common low-scoring samples mapped to the same evidence requirement can be statistically analyzed. The more common low-scoring samples associated with it, or the higher the proportion, the higher the coverage level.

[0064] Understandably, the determination of the gap content first uses the evaluation results of common low-scoring samples as a clue to extract evidence requirement information, and then matches the evidence requirement information with the searchable content of the reference materials item by item to obtain the support information. This makes the identification of the gap based on a specific comparison of whether each evidence requirement is supported, so as to accurately locate which specific evidence support in the reference materials is insufficient, and avoid the subsequent supplementary content being misaligned with the real gap.

[0065] Furthermore, since the evidence required for support or insufficient support is not obtained in this embodiment, the information is merged according to the coverage of common low-scoring samples to form gap content with priority. Gaps that cover a wider range of common low-scoring samples are merged into higher priority. Thus, when supplementing the content based on the gap content as reference material, the part with the greatest benefit can be filled first. This allows the limited supplementation to improve the support of common low-scoring samples to the greatest extent, so that each optimization can solve a large range of defects with a small supplementation cost.

[0066] In summary, current agent evaluation methods can only determine whether the final answer is qualified, and cannot comprehensively evaluate the agent. This forces testing and development teams to review each intermediate step of the agent's process, resulting in low efficiency in agent evaluation. Therefore, this embodiment obtains the evaluation task configuration, acquires the agent's evaluation sample and target evaluation operator based on the task configuration, and evaluates the agent based on the target evaluation operator, a preset agent control tool, and the evaluation sample to obtain a structured evaluation result. Since the evaluation sample in this embodiment includes the agent's operational process data, and the target evaluation operator used for evaluation is obtained by debugging and optimizing the operational process data, business documents, and scoring items, this embodiment can comprehensively evaluate the operational process data, including intermediate data and judgment links, through the evaluation operator.

[0067] Since this embodiment evaluates the entire operation process of the agent through an evaluation operator, Furthermore, the output is a structured evaluation result that includes the evaluation of each of the aforementioned operational process data. This allows the embodiment to obtain a comprehensive evaluation result without having to review each intermediate process of the agent, thereby improving the efficiency of agent evaluation.

[0068] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Before step S20, the comprehensive evaluation method for the intelligent agent further includes steps S01 to S04: Step S01: Obtain samples of the agent's operation process, business documents, and scoring items; It should be noted that the runtime samples are sample materials used to train and adjust the agent, including the agent's behavior and state during operation, enabling the agent to learn based on real-world performance during training. Business document samples are sample materials used to train and adjust the agent, including business-level rules and knowledge content, enabling the agent to learn and master business requirements during training. Scoring item samples are sample materials used to train and adjust the agent. In this embodiment, data suitable for skill creation is selected based on the scoring item samples, automatically divided into training and validation sets. If business knowledge is selected, the reference dataset is used as a strong reference during operator creation, and knowledge details are obtained through retrieval tools.

[0069] It should also be noted that the running samples are data from the actual operation of the intelligent agent, including user input, session context, agent planning steps, tool call parameters, tool return results, observation information, final output, anomaly information, human labels, knowledge base documents, business rules, and historical scoring results. The process in this embodiment from obtaining training samples to obtaining the final optimized operator can be referred to... Figure 6 .

[0070] Understandably, the acquired operational process samples, business document samples, and scoring item samples respectively embody the agent's operational performance, business requirements, and scoring requirements. These three types of samples collectively constitute the input material for training and adjusting the agent. The training content covers real operational performance, clear business requirements, and a well-defined scoring standard. Subsequent training and adjustments to the agent can be directly based on these samples, enabling the agent to understand its actual operational performance during learning and training, while simultaneously mastering the business and scoring requirements it must follow. The training direction closely aligns with real business and evaluation scenarios, thereby improving the effectiveness of agent adjustment and optimization.

[0071] Step S02: Based on the running process data sample, the business document sample, and the scoring item sample, generate a training set and a validation set; It should be noted that the training set is a subset of samples used to train the agent, derived from process data samples, business document samples, and scoring item samples. During training, it provides the agent with the necessary input samples for learning and adjustment. The validation set is a subset of samples used to verify the agent's training effectiveness, also derived from process data samples, business document samples, and scoring item samples. After training, it is used to examine the agent's performance on the samples and determine whether it meets the expected business and scoring requirements.

[0072] It is understandable that training and validation sets are generated based on process data samples, business document samples, and scoring item samples. These three types of samples are organized into two sets with different purposes: training and validation. Thus, the adjustment and optimization of the agent can be carried out based on the training set, and the performance of the agent can be tested using the validation set after training. The validation samples were not involved in the training process, avoiding the bias caused by using training samples to test the training effect. This allows the validation results to objectively reflect the actual effectiveness of the agent's adjustment and optimization, and provides a basis for judging whether the adjusted and optimized agent meets the scoring requirements.

[0073] Step S03: Under the operator creation constraints of the intelligent agent driving tool, create an initial operator including evaluation instructions, reference materials, tool capabilities, output protocol and operation traceability information; It should be noted that operator creation constraints are normative constraints imposed by the intelligent agent driving tool during operator creation. These constraints include limitations on the organization, structure, and creation behavior of operator content, ensuring that operator creation is performed according to uniform standards. Evaluation specifications are the descriptions in the initial operator used to define the evaluation content, including the object to be evaluated, scoring items, judgment boundaries, and output format, clarifying the evaluation object and output requirements for evaluation execution. Tool capabilities are the set of capabilities in the initial operator that can be invoked during the evaluation process. These include execution capabilities such as standardizing process data, extracting evidence, retrieving reference materials, verifying output, and classifying errors, used to support auxiliary operations during the evaluation process.

[0074] The output protocol is the format convention for the evaluation output in the initial operator, including the fields contained in the evaluation result and their organization, used to ensure that the evaluation output can be parsed by the platform in a unified format. Runtime traceability information is information in the initial operator that records the operator's origin and evolution, including creation source, debugging round, release version, and formal task reference relationship, used to trace the operator's process from creation to application. The initial operator is the preliminary operator form generated during the operator creation phase, including evaluation instructions, references, tool capabilities, output protocol, and runtime traceability information, serving as the foundational object before subsequent debugging and optimization. The composition of the initial operator generated in this embodiment can be referenced... Figure 7 .

[0075] Understandably, the initial operator is created under the operator creation constraints of the intelligent agent's driving tool. The generation method of the initial operator is uniformly limited by these constraints from the moment of creation, ensuring that the structure and content of the initial operator are organized according to established specifications, avoiding arbitrary creation that deviates from these specifications. Furthermore, since the initial operator includes evaluation instructions, reference materials, tool capabilities, output protocols, and operational traceability information, the judgment criteria, reference materials, execution capabilities, output conventions, and source information required for evaluation are all incorporated into the initial operator during the operator creation phase. This ensures that the initial operator, as the starting point for subsequent debugging and optimization, already possesses a complete content structure, thereby improving the efficiency of the operator optimization process.

[0076] Step S04: Based on the training set, the validation set, and the intelligent agent driving tool, the initial operator is debugged and optimized to obtain the target evaluation operator.

[0077] Understandably, the initial operator's debugging and optimization are based on both the training set and the validation set, and are organized and executed by the agent driving tool. The training set and the validation set provide samples for debugging and optimization from the perspectives of discovering problems and verifying effects, respectively. The agent driving tool standardizes the debugging and optimization process, so that the initial operator can be continuously adjusted based on the deficiencies exposed by the training set during the iterative optimization process, and the performance of the adjusted operator on the samples can be examined using the validation set, so that the target evaluation operator is adapted to the evaluation requirements reflected by the samples.

[0078] Furthermore, since the debugging and optimization in this embodiment are carried out under the constraints of the intelligent agent driving tool, the execution method and process of debugging and optimization are uniformly managed, so that the debugging and optimization of different rounds and different tasks can be carried out in accordance with the same specifications, thereby improving the stability of the evaluation results.

[0079] In one feasible implementation, the specific implementation of debugging and optimizing the initial operator based on the training set, the validation set, and the agent driving tool to obtain the target evaluation operator can also be: Use the initial operator as the current operator to be optimized, and obtain the debugging task for the operator to be optimized; Based on the intelligent agent driving tool, the debugging task, and the training set, the operator to be optimized is optimized to obtain the optimized operator; Based on the verification set, the optimization operator is verified, and based on the obtained verification results, it is determined whether the optimization operator meets the release conditions. If the release conditions are not met, the optimized operator is taken as a new operator to be optimized, and the process returns to the step of obtaining the debugging task for the operator to be optimized, until the optimized operator meets the release conditions and the target evaluation operator is obtained.

[0080] It should be noted that the operator to be optimized is the operator object currently awaiting optimization during the debugging and optimization process. At the start of debugging, the initial operator serves as the operator to be optimized. If the release conditions are not met, the optimized operator will be updated and continue to serve as the operator to be optimized. The debugging task is a runtime task initiated for the operator to be optimized, used to drive one optimization execution, organizing the input and execution process required for this optimization. The optimized operator is the operator form obtained after optimizing the operator to be optimized, and enters the verification stage as the direct output of a single round of optimization for verification.

[0081] The validation results are information obtained after validating the optimized operator on the validation set, reflecting the performance of the optimized operator on the validation set samples. The release conditions are criteria used to determine whether the optimized operator meets the standards for use as a target evaluation operator, and the determination is based on the validation results.

[0082] In this implementation, each debugging session records the sample input, evaluation output, expected result, manual label, pass status, exception information, evidence citation, and error type. Multiple debugging tasks can be created, with each task performing three rounds of iterative optimization by default. In each round, the current operator is tested on the validation set, and the test results are output.

[0083] The specific debugging results are presented as pass rate, scoring output rate, consistency rate, number of failed samples, anomaly type distribution, and sample-level details. R&D or testing personnel can use this information to determine whether the operator meets the release requirements, or choose to submit failed samples, manual review comments, and supplementary information to the continuous optimization process.

[0084] Furthermore, when an operator is released, the platform specifies the evaluation instructions, scoring protocol, reference version, tool version, output schema, model configuration, debugging sample range, acceptance metrics, and source information. Released operators are managed by version number; formal evaluation tasks can only reference released versions and cannot directly reference drafts that are still under modification.

[0085] Versioning is used to control the drift of evaluation conclusions. Even if the skill is further optimized, scoring rules adjusted, references added, or the model upgraded, the historical version referenced by the completed task will not change. Each new round of optimization will create a new draft or version, and the differences between it and the old version will be recorded to facilitate comparison of score changes across different versions on the same batch of samples.

[0086] It is understood that in this embodiment, if the optimized operator fails to meet the release conditions after verification, it will be treated as a new operator to be optimized. The process will return to the step of obtaining the debugging task for the operator to be optimized and continue to optimize until the release conditions are met, at which point the target evaluation operator is obtained. Moreover, the results of each round of optimization are retained and carried over to the next round, so that the final target evaluation operator is an operator form that has been fully debugged and optimized and has passed the release condition verification, thereby improving the accuracy of agent evaluation.

[0087] In one feasible implementation, the specific implementation of optimizing the operator to be optimized based on the intelligent agent driving tool, the debugging task, and the training set to obtain the optimized operator can also be: The debugging task is performed based on the intelligent agent driving tool, and the running process samples in the training set are evaluated to obtain sample evaluation results; Based on the sample evaluation results and the real results in the training set, the source of the problem of the operator to be optimized is determined; Based on the source of the problem, the operator to be optimized is optimized in a targeted manner to obtain the optimized operator.

[0088] It should be noted that the sample evaluation results are obtained by evaluating the running process samples in the training set during the debugging task, and are used to reflect the evaluation of the operator to be optimized on the running process samples. The true results are the results in the training set corresponding to the running process samples, labeled with the true situation of the samples, and are used as a benchmark for comparison with the sample evaluation results. The source of the problem is the reason why the sample evaluation results of the operator to be optimized deviate from the true results. It is determined by comparing the differences between the sample evaluation results and the true results, and is used to indicate the direction for optimization of the operator to be optimized.

[0089] It is understood that the problem source of the operator to be optimized in this embodiment is determined by comparing the sample evaluation results with the real results in the training set, and the problem source is obtained by comparing the actual evaluation output of the operator to be optimized with the real situation, so that the optimization action can be aimed at the real deviation, thereby making the direction of single-round optimization clearer.

[0090] By targeting the problem source for optimization operators, the adjustments are made to correspond to the identified problem source, while other parts unrelated to the problem source are not modified unnecessarily. As a result, the optimized operators can correct existing problem sources while maintaining their original usability, without introducing new deviations due to large-scale changes, so that each round of optimization results in effective improvement.

[0091] In one feasible implementation, the sample evaluation results include debugging results, failed samples, sample scores, and expected labels, and the real results include real scores and real labels. A further implementation of determining the source of the problem for the operator to be optimized based on the sample evaluation results and the real results in the training set may be: Based on the sample scores and the actual scores, the consistency of the scores between the operator to be optimized and the actual results is determined; Based on the debugging results, the failed samples, the expected labels, and the actual labels, the sources of problems for the operator to be optimized are determined. The sources of problems include at least one of the following: unclear expression of scoring rules, insufficient reference materials, incorrect trajectory field parsing, unstable output protocol, and overly coarse error classification.

[0092] It should be noted that the debugging results are the overall results obtained after the debugging task is completed, reflecting the overall status of this debugging operation. Failure samples are those that did not perform as expected during the debugging process; they are used to expose the shortcomings of the operator to be optimized and serve as the object of problem analysis. Sample scores are the numerical scores given to each running sample in the sample evaluation results, reflecting the operator's judgment of the sample.

[0093] Expected labels are the anticipated judgment labels recorded in the sample evaluation results, corresponding to each operational process sample, and are used for comparison with the true labels. True scores are the actual score values ​​in the actual results, corresponding to each operational process sample, and are used for comparison with the sample scores. True labels are the actual judgment labels in the actual results, corresponding to each operational process sample, and are used for comparison with the expected labels. Scoring consistency is a result index that measures the degree of agreement between the sample scores and true scores of the operator to be optimized. It is determined by comparing the sample scores and true scores and reflects the reliability of the operator to be optimized at the scoring level.

[0094] Unclear scoring rules are a source of inconsistencies in the evaluation criteria, leading to inconsistent judgments of the operator to be optimized on similar samples. Insufficient reference materials result in inadequate evaluation data, making it impossible to verify key evidence. Incorrect trajectory field parsing leads to missing information in tool calls or intermediate steps due to incorrect parsing of trajectory-related fields. Unstable output protocols cause inconsistent execution of the evaluation output format, resulting in unparseable output. Overly coarse error classification fails to adequately guide subsequent remediation due to the lack of clear reasons for low scores.

[0095] Understandably, since the consistency of the scores is determined based on the sample scores and the actual scores, and the source of the problem is determined based on the debugging results, failed samples, expected labels and actual labels, the optimization of the operator to be optimized combines the comparison of the scoring dimension with the debugging process and the analysis of the label dimension. Thus, it can determine the degree of deviation between the score of the operator to be optimized and the actual situation from the score consistency, and further pinpoint the source of the problem by combining process information such as failed samples.

[0096] Furthermore, since the sources of problems include at least one of the following: unclear expression of scoring rules, insufficient reference materials, incorrect parsing of trajectory fields, unstable output protocol, and overly coarse error classification, the sources of problems are narrowed down to a limited and clear range of categories. Thus, after determining the source of the problem, optimization can be directly applied to the corresponding aspects based on the category, thereby improving the efficiency and targeting of the optimization of the operator to be optimized.

[0097] In summary, this embodiment first divides the agent's operational process samples, business document samples, and scoring item samples into training and validation sets. Then, it creates initial operators based on the operator creation constraints of the agent's driving tool, allowing subsequent debugging and optimization to proceed on these complete initial operators. Furthermore, during debugging, optimization operators that do not meet the release conditions are iterated upon as new operators to be optimized. Each round of optimization first compares the sample evaluation results with the actual results in the training set to determine scoring consistency and the source of the problem. Then, targeted optimization is implemented based on the source of the problem, thus improving the efficiency and relevance of the optimization process.

[0098] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the comprehensive evaluation method of the intelligent agent in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0099] This application provides a comprehensive evaluation device for intelligent agents, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the comprehensive evaluation method for intelligent agents in Embodiment 1 described above.

[0100] The following is for reference. Figure 8The diagram illustrates a structural schematic suitable for implementing the intelligent agent comprehensive evaluation device in the embodiments of this application. The intelligent agent comprehensive evaluation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The intelligent agent comprehensive evaluation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0101] like Figure 8 As shown, the intelligent agent comprehensive evaluation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the intelligent agent comprehensive evaluation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the intelligent agent comprehensive evaluation device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows an intelligent agent comprehensive evaluation device with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0102] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0103] The intelligent agent comprehensive evaluation device provided in this application, employing the intelligent agent comprehensive evaluation method in the above embodiments, can solve the technical problem of low efficiency in current intelligent agent evaluation. Compared with the prior art, the beneficial effects of the intelligent agent comprehensive evaluation device provided in this application are the same as those of the intelligent agent comprehensive evaluation method provided in the above embodiments, and other technical features in this intelligent agent comprehensive evaluation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0106] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the intelligent agent comprehensive evaluation method in the above embodiments.

[0107] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0108] The aforementioned computer-readable storage medium may be included in the comprehensive evaluation device for intelligent agents; or it may exist independently and not be assembled into the comprehensive evaluation device for intelligent agents.

[0109] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the intelligent agent comprehensive evaluation device, cause the intelligent agent comprehensive evaluation device to execute the aforementioned intelligent agent comprehensive evaluation method.

[0110] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0113] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described comprehensive agent evaluation method, which can solve the technical problem of low efficiency in current agent evaluation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the comprehensive agent evaluation method provided in the above embodiments, and will not be repeated here.

[0114] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the comprehensive evaluation method for intelligent agents as described above.

[0115] The computer program product provided in this application can solve the technical problem of low efficiency in current agent evaluation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the comprehensive agent evaluation method provided in the above embodiments, and will not be repeated here.

[0116] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A comprehensive evaluation method for intelligent agents, characterized in that, The method includes: Obtain the assessment task configuration; Based on the task configuration, the agent's evaluation sample and target evaluation operator are obtained, wherein the evaluation sample includes the agent's operation process data, and the target evaluation operator is obtained by debugging and optimizing the agent's operation process data, business documents, and scoring items; Based on the target evaluation operator, the preset intelligent agent driving tool, and the sample to be evaluated, the intelligent agent is evaluated to obtain a structured evaluation result, wherein the structured evaluation result includes the evaluation results of each of the operation process data.

2. The method as described in claim 1, characterized in that, Before the step of obtaining the agent's evaluation sample and target evaluation operator based on the task configuration, the method further includes: Obtain samples of the agent's operational process, business documents, and scoring items; Based on the operation process samples, the business document samples, and the scoring item samples, a training set and a validation set are generated. Under the operator creation constraints of the intelligent agent driving tool, an initial operator is created, including evaluation instructions, reference materials, tool capabilities, output protocols, and operation traceability information; Based on the training set, the validation set, and the intelligent agent driving tool, the initial operator is debugged and optimized to obtain the target evaluation operator.

3. The method as described in claim 2, characterized in that, The step of debugging and optimizing the initial operator based on the training set, the validation set, and the intelligent agent driving tool to obtain the target evaluation operator includes: Use the initial operator as the current operator to be optimized, and obtain the debugging task for the operator to be optimized; Based on the intelligent agent driving tool, the debugging task, and the training set, the operator to be optimized is optimized to obtain the optimized operator; Based on the verification set, the optimization operator is verified, and based on the obtained verification results, it is determined whether the optimization operator meets the release conditions. If the release conditions are not met, the optimized operator is taken as a new operator to be optimized, and the process returns to the step of obtaining the debugging task for the operator to be optimized, until the optimized operator meets the release conditions and the target evaluation operator is obtained.

4. The method as described in claim 3, characterized in that, The step of optimizing the operator to be optimized based on the intelligent agent driving tool, the debugging task, and the training set to obtain the optimized operator includes: The debugging task is performed based on the intelligent agent driving tool, and the running process samples in the training set are evaluated to obtain sample evaluation results; Based on the sample evaluation results and the real results in the training set, the source of the problem of the operator to be optimized is determined; Based on the source of the problem, the operator to be optimized is optimized in a targeted manner to obtain the optimized operator.

5. The method as described in claim 4, characterized in that, The sample evaluation results include debugging results, failed samples, sample scores, and expected labels; the real results include real scores and real labels; and the step of determining the source of the problem of the operator to be optimized based on the sample evaluation results and the real results in the training set includes: Based on the sample scores and the actual scores, the consistency of the scores between the operator to be optimized and the actual results is determined; Based on the debugging results, the failed samples, the expected labels, and the actual labels, the sources of problems for the operator to be optimized are determined. The sources of problems include at least one of the following: unclear expression of scoring rules, insufficient reference materials, incorrect trajectory field parsing, unstable output protocol, and overly coarse error classification.

6. The method as described in claim 1, characterized in that, The step of evaluating the agent based on the target evaluation operator, the preset agent driving tool, and the sample to be evaluated to obtain a structured evaluation result includes: Based on the aforementioned intelligent agent driving tool, reference materials are determined that are bound to the target evaluation operator; Based on the intelligent agent driving tool, the reference materials, the operation process data in the sample to be evaluated, and the version of the target evaluation operator are combined into a standard context; Based on the target evaluation operator and the standard context, the agent is evaluated to obtain multiple corresponding evaluation results, wherein the evaluation results include sample score, pass label, judgment reason, evidence citation, misclassification, running time and abnormal state; Based on the aforementioned intelligent agent driving tool, the evaluation results are combined into the structured evaluation results.

7. The method as described in claim 1, characterized in that, After the step of evaluating the agent based on the target evaluation operator, the preset agent driving tool, and the sample to be evaluated to obtain a structured evaluation result, the method further includes: Obtain new low-scoring samples, manual calibration results, and accuracy reports generated during the formal operation of the target evaluation operator; Create a backup operator identical to the target evaluation operator; The backup operator is optimized based on the new low-scoring samples, manual calibration results, and accuracy reports to obtain an optimized new evaluation operator.

8. A comprehensive evaluation device for intelligent agents, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the comprehensive evaluation method for intelligent agents as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the comprehensive evaluation method for intelligent agents as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the comprehensive evaluation method for intelligent agents as described in any one of claims 1 to 7.

Images