Evaluation method and device for intelligent agent, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the fields of financial science and technology and medical health, and discloses an evaluation method and device for an intelligent agent, equipment and a medium. Performing analysis and structured processing on the original moving trajectory data to extract an evaluation information set related to evaluation; the evaluation information set is input into an evaluation model cluster for evaluation to obtain a plurality of different evaluation results, and the evaluation model cluster comprises a plurality of special evaluation models configured to execute different evaluation dimensions; the multiple different evaluation results are subjected to fusion analysis to generate a comprehensive evaluation report, and the comprehensive evaluation report comprises quantitative scores and diagnosis information of the intelligent agent in target understanding, planning making and action execution. And the accuracy and effectiveness of evaluation are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology and can be applied to the fields of financial technology and healthcare. In particular, it relates to an evaluation method, apparatus, device, and medium for intelligent agents. Background Technology

[0002] Currently, intelligent agent technology based on large language models has shown great application potential in key areas such as fintech and healthcare, but the development of its evaluation methods has lagged significantly. Traditional evaluation paradigms are mostly limited to a simple comparison of the final results of the task, that is, judging success or failure by matching the agent's output answer with the manually labeled standard answer.

[0003] This approach suffers from particularly significant drawbacks in complex business scenarios. First, it heavily relies on large amounts of high-quality, high-cost manually labeled data. In scenarios such as financial risk control model evaluation or medical diagnostic pathway verification, the labeling costs are exorbitant, standards are difficult to unify, and scalability is poor. More importantly, it completely fails to reveal the root causes of agent failures in complex decision-making chains. For example, when a credit approval agent makes an incorrect decision, or a clinical diagnostic assistance agent provides inappropriate advice, existing methods can only conclude that the agent is "unqualified," but cannot diagnose whether the problem stems from a misunderstanding of user risk preferences (target problem), a flaw in the approval or diagnostic logic path (planning problem), or an error in calling external data interfaces (action problem). This "black box" evaluation makes it difficult for developers to perform effective optimization, resulting in a severely insufficient evaluation effectiveness.

[0004] Furthermore, existing assessment tools often employ a single, "one-size-fits-all" standard, attempting to measure an agent's performance across multiple dimensions, including risk, compliance, and efficiency, using the same set of rules. When handling complex tasks with long chains of operations and high risks, such as financial transaction audits or personalized treatment plan generation, a single evaluator struggles to simultaneously consider multiple key dimensions, including logical rigor, regulatory compliance, execution efficiency, and outcome security. This often leads to overlooking important risk points or making misjudgments, resulting in low assessment accuracy. This blind spot in assessment, disconnected from the agent's actual "perception-planning-decision-making" process, not only hinders the development of high-performance agents but also poses a substantial obstacle to their industrial deployment in high-risk fields. Summary of the Invention

[0005] This invention provides an evaluation method, apparatus, computer equipment, and medium for intelligent agents, in order to solve the technical problem that the accuracy and effectiveness of existing agent evaluations are both low.

[0006] Firstly, an evaluation method for intelligent agents is provided, including: The raw trajectory data generated after the intelligent agent executes the target task is obtained, and the raw trajectory data is parsed and structured to extract the evaluation information set related to the evaluation. The evaluation information set is input into an evaluation model cluster for evaluation to obtain multiple different evaluation results, wherein the evaluation model cluster includes multiple dedicated evaluation models configured to perform different evaluation dimensions. The multiple different evaluation results are integrated and analyzed to generate a comprehensive evaluation report, which includes quantitative scores and diagnostic information of the intelligent agent in goal understanding, planning and action execution.

[0007] Secondly, an evaluation device for intelligent agents is provided, comprising: The acquisition and processing unit is used to acquire the original running trajectory data generated after the intelligent agent executes the target task, and to parse and structure the original running trajectory data to extract the evaluation information set related to the evaluation. An evaluation unit is used to input the evaluation information set into an evaluation model cluster for evaluation to obtain multiple different evaluation results, wherein the evaluation model cluster includes multiple dedicated evaluation models configured to perform different evaluation dimensions. The generation unit is used to integrate and analyze multiple different evaluation results to generate a comprehensive evaluation report, wherein the comprehensive evaluation report includes quantitative scores and diagnostic information of the intelligent agent in goal understanding, planning and action execution.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the evaluation method for intelligent agents described above.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the evaluation method for intelligent agents described above.

[0010] The aforementioned evaluation method, apparatus, computer equipment, and storage medium for intelligent agents can acquire raw operational trajectory data generated after the intelligent agent executes a target task, and parse and structure the raw operational trajectory data to extract an evaluation information set related to the evaluation. This evaluation information set is then input into an evaluation model cluster for evaluation to obtain multiple different evaluation results. The evaluation model cluster includes multiple dedicated evaluation models configured to perform different evaluation dimensions. The multiple different evaluation results are then fused and analyzed to generate a comprehensive evaluation report. This comprehensive evaluation report includes quantitative scores and diagnostic information regarding the intelligent agent's target understanding, planning, and action execution. In this invention, by parsing the raw operational trajectory data of the intelligent agent, its internal decision-making process becomes observable and transparent. Multiple dedicated evaluation models are used to analyze the process from multiple dimensions, replacing a single final result comparison. The resulting comprehensive evaluation report not only provides quantitative scores but also pinpoints defects in specific stages, thus realizing a shift from "result judgment" to "process diagnosis," significantly improving the accuracy and effectiveness of the evaluation. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating an evaluation method for intelligent agents according to an embodiment of the present invention; Figure 2 yes Figure 1 A schematic diagram of a specific implementation of step S110; Figure 3 yes Figure 1 A schematic diagram of a specific implementation of step S120; Figure 4 yes Figure 1 A schematic diagram of a specific implementation of step S130; Figure 5 This is a schematic block diagram of an evaluation device for intelligent agents in one embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The evaluation method for intelligent agents provided in this invention can be applied to either a client or a server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. Currently, in the fintech and healthcare fields, the accuracy and effectiveness of existing agent evaluations are relatively low. To address these issues, this invention proposes an evaluation method for intelligent agents. This method achieves observability and white-boxing of the internal decision-making process by analyzing the raw operational trajectory data of the intelligent agent; it utilizes multiple dedicated evaluation models to analyze the process from multiple dimensions, replacing a single comparison of the final result; the resulting comprehensive evaluation report not only provides quantitative scores but also pinpoints defects in specific stages, thus realizing a shift from "result judgment" to "process diagnosis," significantly improving the accuracy and effectiveness of the evaluation. The invention will be described in detail below through specific embodiments.

[0015] Please see Figure 1 As shown, Figure 1 A flowchart illustrating an evaluation method for intelligent agents provided in an embodiment of the present invention includes the following steps: S110-S130.

[0016] S110. Obtain the original running trajectory data generated after the intelligent agent executes the target task, and parse and structure the original running trajectory data to extract the evaluation information set related to the evaluation.

[0017] Specifically, the original running trajectory data, recorded in chronological order, generated by the intelligent agent during the complete execution of a user task, is captured or received from the agent's operating environment or log system. This original running trajectory data includes the agent's interaction records with the environment, internal decision state outputs, tool call sequences and their parameters, and the environment's response to each action.

[0018] Among them, such as Figure 2 As shown, step S110 includes steps S111-S112: S111. Based on preset parsing rules, the original running trajectory data is identified and segmented to extract task instructions, planning text, tool call records and environmental feedback text. S112. The task instructions, planning text, tool call records, and environmental feedback text are cleaned, classified, and formatted in sequence, and then encapsulated into a structured data set to form the evaluation information set. The evaluation information set includes the task instructions, planning sequence text, final task output, actual action sequence, and environmental feedback sequence corresponding to the actual action sequence.

[0019] Specifically, the system automatically parses unstructured raw operation trajectory data according to preset parsing rules. First, it applies regular expression matching, keyword recognition, and a specific log format parser to accurately identify and separate different types of data fragments from the raw operation trajectory data. Specifically, by matching identifiers such as "User Input:", "Plan:", and "Action:" or analyzing specific fields in structured logs such as JSON / XML, it extracts task instructions (i.e., initial user requests or system instruction text). Next, it locates and extracts the step-by-step plans described in lists or natural language from the agent's internal decision-making process output, as planning text. Simultaneously, by recognizing keywords such as "Tool Call:" and "Function:" and subsequent parameter structures, it extracts detailed information for each external function or API call, forming a tool call record containing the tool name, input parameters, and call timestamp. Finally, by correlating the system return information after each tool call or action, it obtains environmental feedback text, i.e., a description of the tool's execution results or changes in the environmental state.

[0020] The task instructions, planning text, tool call records, and environmental feedback text extracted in step S111 are deeply processed to form standardized, consistent, and directly usable structured data for model evaluation. First, data cleaning is performed, including removing irrelevant debugging information, correcting coding errors, standardizing timestamp formats, and handling missing values. Then, categorization and association are performed; for example, all tool call records belonging to the same planning step and scattered across different log lines are grouped together, and each tool call record is bound to its corresponding environmental feedback text. Next, format standardization is implemented; for example, all tool call parameters are converted to standard JSON objects, steps in the planning text are uniformly numbered, and all text fields are standardized and encoded. Finally, the processed task instructions, planning sequence text, final task output, actual action sequence (converted from tool call records), and environmental feedback sequence are organized and encapsulated according to a predefined data schema (e.g., a dictionary containing specific key-value pairs or a structured JSON schema), thereby generating a complete, unambiguous set of evaluation information that can be directly consumed by subsequent specialized evaluation models.

[0021] S120. The evaluation information set is input into the evaluation model cluster for evaluation to obtain multiple different evaluation results, wherein the evaluation model cluster includes multiple dedicated evaluation models that are configured to perform different evaluation dimensions.

[0022] Specifically, the evaluation model cluster includes a first dedicated evaluation model, a second dedicated evaluation model, a third dedicated evaluation model, a fourth dedicated evaluation model, and a fifth dedicated evaluation model; the evaluation results include a first evaluation result, a second evaluation result, a third evaluation result, a fourth evaluation result, and a fifth evaluation result. It should be noted that the multiple dedicated evaluation models in the evaluation model cluster execute the evaluation tasks in parallel.

[0023] Among them, such as Figure 3 As shown, step S120 includes steps S121-S125: S121. Based on the initial quality of the task and the final output of the task, the first dedicated evaluation model is used to evaluate the degree of target achievement and generate a first evaluation result. S122. Based on the planning sequence text, the logical quality of the planning is evaluated using the second dedicated evaluation model, and a second evaluation result is generated; S123. Based on the actual action sequence and the planning sequence text, the third dedicated evaluation model is used to evaluate the plan fidelity and generate a third evaluation result. S124. Based on the actual action sequence, the effectiveness and efficiency of the action execution are evaluated using the fourth dedicated evaluation model, and a fourth evaluation result is generated. S125. Based on the evaluation information set, the fifth dedicated evaluation model is used to evaluate the logical consistency between the intelligent agent's reasoning and declaration in the original running trajectory data, and a fifth evaluation result is generated.

[0024] Specifically, step S124 includes: verifying the completeness and format compliance of the input parameters of each tool call in the actual action sequence based on a preset tool interface specification library, and evaluating whether they meet the calling preconditions, generating an effectiveness evaluation sub-result; analyzing the actual action sequence, identifying redundant calls, consecutive calls that can be merged, or inefficient execution steps with better alternatives, generating an efficiency evaluation sub-result; and generating the fourth evaluation result based on the effectiveness evaluation sub-result and the efficiency evaluation sub-result.

[0025] More specifically, by constructing a multi-dimensional, professional evaluation process closely aligned with the core decision-making cycle of intelligent agents—the "goal-planning-action" framework—a fundamental shift from black-box result verification to white-box process diagnosis has been achieved. The specific evaluation steps are as follows: First, the goal achievement assessment is performed by a dedicated evaluation model. This model receives the initial task instruction and the final task output. Its core is not superficial string matching, but rather deep semantic understanding and intent alignment analysis. It is configured to parse the user's true goals, implicit constraints, and success criteria contained in the initial instruction, and comprehensively evaluate the final output's compliance in terms of functionality, completeness, contextual relevance, and whether all constraints are met. For example, for an instruction to "plan a weekend trip to Beijing," the evaluation not only checks whether attractions are listed, but also assesses the reasonableness of the time arrangement, adherence to budget constraints, and matching of points of interest, thereby generating a first evaluation result containing a quantitative score and a specific gap description, accurately answering the ultimate question: "Did the agent truly solve the problem?"

[0026] Second, the planning quality assessment is independently handled by a second dedicated assessment model. This model focuses on the intrinsic quality of the planning sequence text itself, analyzed offline, and is completely decoupled from its subsequent implementation. Its assessment dimensions include: the overall logical rigor and coherence of the plan; the rationality, clarity, and completeness of breaking down complex tasks into sub-steps; the appropriateness of the tools, methods, or resources selected or allocated for each sub-step; and whether the plan considered potential risks or alternative solutions. This assessment step is crucial, as it clearly distinguishes whether a task is thwarted due to "strategic (planning) errors" or "tactical (execution) failures." The resulting second assessment results provide a detailed analysis of the strengths and weaknesses of the plan itself as an "action roadmap."

[0027] Third, the plan fidelity assessment is conducted using a dedicated third assessment model, which rigorously compares and analyzes the actual action sequence against the planned sequence text item by item. Its work includes: identifying whether any planned steps were omitted during actual execution; whether any unauthorized additional actions were added; whether the order of key steps was reversed; and whether any substantial modifications to the planned steps have a reasonable explanation. This step directly quantifies the agent's "execution ability" and "discipline," and the generated third assessment results accurately record all deviations and their nature, serving as a key basis for judging whether the agent is "saying one thing and doing another."

[0028] Fourth, the performance evaluation of action execution is undertaken by a dedicated fourth evaluation model, which conducts a dual review of the actual action sequence. At the effectiveness level, the model, based on a pre-defined interface specification library, performs compliance checks on each tool or API call in the sequence, including checking the syntactic correctness of input parameters, semantic validity (such as the rationality of query keywords), data type conformity, and whether the preconditions for the call are met, generating a effectiveness evaluation sub-result. At the efficiency level, the model analyzes the entire actual action sequence, identifying redundant calls (repeatedly executing the same or equivalent operations), mergeable consecutive calls (that could have been completed in one request), suboptimal choices with known better alternatives, and execution sequences that may cause unnecessary waiting or resource consumption, obtaining an efficiency evaluation result. Based on the effectiveness and efficiency evaluation sub-results, the fourth evaluation result is generated.

[0029] Fifth, the logical consistency assessment is performed by a dedicated assessment model, which acts as a global "auditor." Based on a complete assessment information set, this model conducts a cross-timestep review of all intermediate steps of reasoning, factual statements, assumptions, and final conclusions generated by the agent throughout the entire task lifecycle. Its specialty is to detect and flag various logical flaws, such as: a later assertion contradicting a previously confirmed fact or the agent's own prior statements (factual illusion or self-contradiction); unexplainable jumps in the reasoning chain from known information to the conclusion; or inconsistent interpretations of the same concept or fact in different contexts. The resulting fifth assessment results profoundly reflect the stability, reliability, and level of rationality of the agent's cognitive state.

[0030] By implementing steps S121-S125, a white-box, refined diagnosis and attribution of the intelligent agent's execution process was achieved, significantly improving the accuracy and operability of the assessment and providing a direct basis for targeted optimization of the agent.

[0031] It should be noted that the dedicated evaluation models are based on the same large language model, instantiated and optimized through carefully designed prompts. Each instance of a dedicated evaluation model is assigned a clear evaluation role and strictly adheres to the task context, structured evaluation criteria, and output format defined in the prompts. In the evaluation process, the five dedicated evaluation models each perform their respective functions: the first dedicated evaluation model evaluates the degree of goal achievement based on the task's start and end states; the second dedicated evaluation model analyzes the logical quality of the planning text; the third dedicated evaluation model compares the actual actions with the plan, evaluating the plan's fidelity; the fourth dedicated evaluation model directly evaluates the effectiveness and efficiency of the actual actions; and the fifth dedicated evaluation model integrates all information throughout the process, evaluating the consistency of the agent's statements and reasoning. This system, through prompt engineering, transforms the general large language model into a series of standardized, reusable dedicated evaluators, ensuring that the evaluation is both in-depth and standardized.

[0032] S130. The multiple different evaluation results are integrated and analyzed to generate a comprehensive evaluation report, wherein the comprehensive evaluation report includes quantitative scores and diagnostic information of the intelligent agent in goal understanding, planning and action execution.

[0033] Specifically, such as Figure 4 As shown, step S130 includes steps S131-S132: S131, according to preset weights, the quantitative scores in each of the evaluation results are weighted to obtain a comprehensive score; S132, the comprehensive score and the diagnostic information in each of the evaluation results are integrated to obtain the comprehensive evaluation report. Specifically, the quantitative scores in the first to fifth evaluation results are weighted according to preset weights. For example, the first evaluation result (goal achievement) which measures the final effect can be given a higher weight, and the fifth evaluation result (reasoning consistency) which reflects the fundamental logic can be given a key weight. Then, the quantitative scores of the second (evaluation result planning logic), the third evaluation result (plan fidelity), and the fourth evaluation result (execution efficiency) are combined using a weighted formula to synthesize a comprehensive score on a percentage scale, which serves as a benchmark for overall performance.

[0034] The generated comprehensive assessment report does not simply list five assessment results, but rather correlates and attributes their textual diagnoses, mapping them to three core dimensions: "goal understanding," "planning," and "action execution." Goal understanding dimension: mainly analyzes the diagnosis of the first assessment result (whether the correct goal was achieved) and the fifth assessment result (whether the goal cognition was consistent during the process) to determine whether the intelligent agent has any deviation or wavering in its grasp of the task intent.

[0035] Planning dimensions: The feasibility and rationality of the plan are evaluated by focusing on the diagnosis based on the second assessment result (the logical quality of the plan itself) and combining it with the analysis of the reasons for deviation in the third assessment result (whether the actual implementation follows the plan).

[0036] Action execution dimension: The core integrates the diagnosis of the third assessment result (fidelity to the plan) and the fourth assessment result (efficiency and effectiveness of the action itself) to evaluate the execution capability and resource utilization level from planning to implementation.

[0037] Ultimately, the comprehensive evaluation report will present a structured conclusion: after providing an overview of the overall score and the quantitative scores of each evaluation result, it will organically embed correlational diagnoses from the five evaluation results, with the above three-dimensional framework as the core, clearly pointing out in which link the intelligent agent has shortcomings, its manifestations, and possible causes, thus forming a decision support document that combines quantitative evaluation and qualitative attribution.

[0038] In one embodiment, such as this embodiment, step S130 further includes: identifying defective links that cause the quantitative score to be lower than a preset threshold or the diagnostic information to be incorrect, based on the quantitative score and diagnostic information in the comprehensive evaluation report; generating optimization suggestions based on the mapping relationship between the defective links and the intelligent agent; and adjusting at least one module in the intelligent agent based on the optimization suggestions. Specifically, the core of the step of generating optimization suggestions based on the comprehensive evaluation report is a precise analysis process of "diagnosis-attribution-prescription". First, the quantitative score and diagnostic information in the comprehensive report are deeply analyzed to identify defective links. This includes not only explicit problems such as quantitative scores being lower than a preset threshold (e.g., a low "goal achievement" score), but more importantly, logical attribution of error patterns revealed in the diagnostic information (e.g., "frequent deviation from the goal in planning" or "redundant loops in the action sequence") to locate the root failure node, such as a deviation in goal understanding, a contradiction in planning logic, or a decision error in the execution module. Subsequently, based on a predefined architectural mapping relationship, the defective links are associated with specific functional modules of the intelligent agent, and highly targeted optimization suggestions are generated. These recommendations aim to directly drive adjustments to the core modules of the intelligent agent: If the deficiency points to goal understanding (related to the first and fifth evaluation results), the recommendations might be "improving task parsing prompts" or "enhancing the ability to utilize historical context." If the deficiency lies in planning (related to the second and third evaluation results), the recommendations might be "introducing integrity verification rules into the planner" or "optimizing the sub-goal decomposition strategy for long-sequence tasks." If the deficiency lies in action execution (related to the third and fourth evaluation results), the recommendations might be "adjusting the action prediction model to improve the success rate" or "adding an early termination mechanism for invalid actions."

[0039] Ultimately, this process transforms the comprehensive evaluation into specific, actionable iterative instructions, thereby optimizing the capabilities of the intelligent agent.

[0040] For ease of understanding, the evaluation method for intelligent agents of the present invention will be described in detail with examples from the fields of fintech and healthcare: Examples in the fintech sector: In the fintech field, using corporate credit risk assessment as a scenario, this assessment method can systematically evaluate the entire process of an intelligent agent generating credit decision reports. The core task of the intelligent agent is to generate a structured credit risk assessment report and preliminary credit limit recommendations based on the input basic corporate information (such as industry, registered capital, and key indicators of recent financial reports) and real-time risk data (such as equity pledge, legal proceedings, and public opinion dynamics). During the assessment, the system first acquires and parses the raw operational trajectory data of the intelligent agent, extracting task instructions ("Generate a credit risk assessment report for Company A based on the provided data"), planning text ("Step 1: Verify data integrity; Step 2: Calculate debt repayment capacity indicators; Step 3: Assess operational and public opinion risks; Step 4: Integrate factors to provide risk control conclusions and credit limit recommendations"), actual action sequences (calling business information APIs, calling financial statement analysis tools, calculating current ratios, querying public opinion databases, etc.), and corresponding environmental feedback (data returned by the API, and specific calculated values), and encapsulates this into a structured assessment information set. Subsequently, the assessment model cluster will conduct a multi-dimensional specialized assessment of this information set: The first specialized assessment model is responsible for assessing the degree of goal achievement, that is, whether the final risk report covers key dimensions such as debt repayment ability, profitability, industry risk, and public opinion risk, and provides clear risk levels and limit recommendations. If the report omits core risk points or has vague conclusions, it will score lower; The second specialized assessment model independently examines the logical quality of the planning sequence text, judging whether its steps are reasonable and complete. For example, if the planning lacks an assessment step on "related party transaction risk," the logical chain is flawed; The third specialized assessment model compares the actual actions. The sequence and planning sequence text are used to assess plan fidelity. If the plan includes "calculate the Z-Score model" but is not actually executed, the fidelity is low. The fourth dedicated evaluation model analyzes the effectiveness and efficiency of the actual action sequence, checking whether the tool call parameters are correct, whether there are redundant queries (such as repeatedly calling the same API), and whether there is a better data acquisition path. The fifth dedicated evaluation model reviews the entire process and evaluates the logical consistency between the agent's reasoning and statements. For example, if an intermediate step calculates that the company has sufficient cash flow, but the final conclusion indicates "extremely high short-term debt repayment risk" without a reasonable explanation, the consistency is questionable. Finally, the quantitative scores of the five evaluation results are combined into a comprehensive score according to preset weights (such as higher weights for goal achievement and logical consistency), and diagnostic information is integrated to generate a comprehensive evaluation report. The report will pinpoint the deficiencies. For example, if the root cause is a misunderstanding of the objective (incomplete understanding of key indicators of "credit risk"), the optimization suggestion might be "optimize the task prompts and clarify the list of core risk assessment dimensions that must be included"; if the problem lies in planning (missing key analysis steps), the suggestion is "to build necessary rules such as 'related party risk verification' into the planning module"; if the problem lies in action execution (invalid calls), the suggestion is "to calibrate the tool's parameter library and add redundancy checks".

[0041] Example in the healthcare field: In the healthcare field, using clinical diagnostic assistance as a scenario, this evaluation method is used to examine the reliability and safety of intelligent agents assisting doctors in making diagnostic decisions. The task of the intelligent agent is to generate an auxiliary diagnostic report containing preliminary diagnostic opinions, differential diagnoses, and suggestions for further examinations based on simulated or desensitized patient complaints, vital signs, laboratory test results, and imaging reports. At the beginning of the evaluation, the raw operational trajectory data of the intelligent agent is acquired and parsed, extracting task instructions ("generate preliminary diagnostic suggestions based on patient data"), planning text ("Step 1: identify key abnormal indicators; Step 2: propose possible diagnoses based on the medical knowledge base; Step 3: list differential diagnoses and their basis; Step 4: suggest key follow-up or confirmatory examinations"), actual action sequences (calling the medical knowledge base to query disease symptoms, retrieving clinical guidelines, comparing examination results with typical case characteristics, etc.), and environmental feedback (information returned by the knowledge base, matching results), forming a structured evaluation information set. Next, the evaluation model cluster conducts a multi-dimensional assessment: The first dedicated evaluation model assesses the degree of goal achievement, judging whether the generated report accurately addresses the patient's core questions, whether the proposed diagnostic suggestions are based on key indicators, and whether the differential diagnosis is reasonable. If the report only lists general information without closely addressing the case characteristics, the score will be low. The second dedicated evaluation model assesses the logical quality of the plan, examining whether the plan follows clinical reasoning (e.g., prioritizing common and frequently occurring diseases before considering rare diseases), and whether the steps are comprehensive. If the plan lacks the step of "recommending key imaging re-examinations to rule out tumors," the logical chain has potential safety risks. The third dedicated evaluation model assesses the fidelity of the plan. The evaluation process involves several steps. First, it checks whether the actual execution strictly follows the plan. If the plan includes "checking for specific drug allergy history" but this is not actually performed, then the fidelity is insufficient. Second, a dedicated evaluation model assesses the effectiveness and efficiency of the action execution, checking the accuracy and completeness of the medical knowledge base search and identifying any inefficient or redundant searches. Third, a dedicated evaluation model assesses the logical consistency between reasoning and statements, ensuring that the intelligent agent's judgment of the condition and priority of diagnosis are not contradictory throughout the reasoning process. For example, if the initial analysis suggests "high probability of infection," but the subsequent recommended examination plan focuses entirely on non-infectious diseases without reasonable explanation, then the consistency is poor. Finally, the results of all evaluations are integrated to generate a comprehensive evaluation report, including a comprehensive score and integrated diagnostic information. Based on the deficiencies identified in the report, targeted optimization suggestions can be generated: If the problem lies in the understanding of the objective (inaccurate grasp of the boundaries and key points of "aided diagnosis"), it is recommended to "strengthen the prompt words, clarify the auxiliary positioning, and require that the suggestions must cite and associate with the patient's specific abnormal indicators"; if the problem lies in the planning process (inaccurate clinical decision-making logic), it is recommended to "embed 'critical value' verification and safety checkpoints in the planning module"; if the problem lies in the execution of actions (inaccurate information retrieval), it is recommended to "optimize the knowledge base query algorithm to improve retrieval accuracy".

[0042] The evaluation method for intelligent agents in this invention achieves observability and white-boxing of the internal decision-making process by analyzing the raw operational trajectory data of the intelligent agent; it uses multiple dedicated evaluation models to analyze the process from multiple dimensions, replacing the single comparison of the final result; the final comprehensive evaluation report not only provides quantitative scores, but also locates the defects in specific links, thereby realizing the transformation from "result judgment" to "process diagnosis", significantly improving the accuracy and effectiveness of the evaluation.

[0043] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0044] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0045] In one embodiment, an evaluation device 200 for intelligent agents is provided, which corresponds one-to-one with the evaluation methods for intelligent agents described in the above embodiments. For example... Figure 5 As shown, the evaluation device 200 for intelligent agents includes an acquisition and processing unit 201, an evaluation unit 202, and a generation unit 203. Detailed descriptions of each functional module are as follows: The acquisition and processing unit 201 is used to acquire the original running trajectory data generated after the intelligent agent executes the target task, and to parse and structure the original running trajectory data to extract the evaluation information set related to the evaluation. Evaluation unit 202 is used to input the evaluation information set into the evaluation model cluster for evaluation to obtain multiple different evaluation results, wherein the evaluation model cluster includes multiple dedicated evaluation models that are configured to perform different evaluation dimensions. The generation unit 203 is used to integrate and analyze multiple different evaluation results to generate a comprehensive evaluation report, wherein the comprehensive evaluation report includes quantitative scores and diagnostic information of the intelligent agent in goal understanding, planning and action execution.

[0046] In one embodiment, the acquisition processing unit 201 is specifically used for: Based on preset parsing rules, the original running trajectory data is identified and segmented to extract task instructions, planning text, tool call records and environmental feedback text; The task instructions, planning text, tool call records, and environmental feedback text are cleaned, classified, and formatted in sequence, and then encapsulated into a structured data set to form the evaluation information set. The evaluation information set includes the task instructions, planning sequence text, final task output, actual action sequence, and environmental feedback sequence corresponding to the actual action sequence.

[0047] In one embodiment, the evaluation unit 202 is specifically used for: Based on the initial quality of the task and the final output of the task, the first dedicated evaluation model is used to evaluate the degree of goal achievement and generate a first evaluation result. Based on the planning sequence text, the logical quality of the planning is evaluated using the second dedicated evaluation model, and a second evaluation result is generated. Based on the actual action sequence and the planning sequence text, the third dedicated evaluation model is used to evaluate the plan fidelity and generate a third evaluation result. Based on the actual action sequence, the effectiveness and efficiency of action execution are evaluated using the fourth dedicated evaluation model, and a fourth evaluation result is generated.

[0048] In one embodiment, the evaluation unit 202 is further configured to: Based on the preset tool interface specification library, the integrity and format compliance of the input parameters of each tool call in the actual action sequence are verified, and whether they meet the calling preconditions are evaluated to generate a validity evaluation sub-result. Analyze the actual action sequence to identify redundant calls, consecutive calls that can be merged, or inefficient execution steps with better alternatives, and generate efficiency evaluation sub-results. The fourth evaluation result is generated based on the effectiveness evaluation sub-result and the efficiency evaluation sub-result.

[0049] In one embodiment, the evaluation unit 202 is further configured to: Based on the evaluation information set, the fifth dedicated evaluation model is used to evaluate the logical consistency between the intelligent agent's reasoning and declaration in the original running trajectory data, and a fifth evaluation result is generated.

[0050] In one embodiment, the generating unit 203 is further configured to: Based on preset weights, the quantitative scores in each of the evaluation results are weighted and calculated to obtain a comprehensive score. The comprehensive assessment report is obtained by integrating the overall score and the diagnostic information from each of the assessment results.

[0051] In one embodiment, the evaluation device 200 for intelligent agents further includes: The identification unit is used to identify, based on the quantitative score and the diagnostic information in the comprehensive evaluation report, the defective link that causes the quantitative score to be lower than a preset threshold or the diagnostic information to indicate an error. An adjustment generation unit is used to generate optimization suggestions based on the mapping relationship between the defective link and the intelligent agent, and to adjust at least one module in the intelligent agent based on the optimization suggestions.

[0052] The evaluation device for intelligent agents in this invention achieves observability and white-boxing of the internal decision-making process by analyzing the original operational trajectory data of the intelligent agent; it uses multiple dedicated evaluation models to analyze the process from multiple dimensions, replacing the single comparison of the final result; the final comprehensive evaluation report not only provides quantitative scores, but also locates the defects in specific links, thereby realizing the transformation from "result judgment" to "process diagnosis", significantly improving the accuracy and effectiveness of the evaluation.

[0053] Specific limitations regarding the evaluation apparatus for intelligent agents can be found in the limitations of the evaluation methods for intelligent agents described above, and will not be repeated here. Each unit in the aforementioned evaluation apparatus for intelligent agents can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0054] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements functions or steps on the server side of an evaluation method for intelligent agents.

[0055] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements functions or steps on the client side of an evaluation method for intelligent agents.

[0056] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the evaluation method for intelligent agents described above.

[0057] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the evaluation method for intelligent agents described above.

[0058] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0059] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0060] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0061] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An evaluation method for intelligent agents, characterized in that, include: The raw trajectory data generated after the intelligent agent executes the target task is obtained, and the raw trajectory data is parsed and structured to extract the evaluation information set related to the evaluation. The evaluation information set is input into an evaluation model cluster for evaluation to obtain multiple different evaluation results, wherein the evaluation model cluster includes multiple dedicated evaluation models configured to perform different evaluation dimensions. The multiple different evaluation results are integrated and analyzed to generate a comprehensive evaluation report, which includes quantitative scores and diagnostic information of the intelligent agent in goal understanding, planning and action execution.

2. The evaluation method for intelligent agents as described in claim 1, characterized in that, The step of parsing and structuring the original trajectory data to extract the evaluation information set related to the evaluation includes: Based on preset parsing rules, the original running trajectory data is identified and segmented to extract task instructions, planning text, tool call records and environmental feedback text; The task instructions, planning text, tool call records, and environmental feedback text are cleaned, classified, and formatted in sequence, and then encapsulated into a structured data set to form the evaluation information set. The evaluation information set includes the task instructions, planning sequence text, final task output, actual action sequence, and environmental feedback sequence corresponding to the actual action sequence.

3. The evaluation method for intelligent agents as described in claim 2, characterized in that, The evaluation model cluster includes a first dedicated evaluation model, a second dedicated evaluation model, a third dedicated evaluation model, and a fourth dedicated evaluation model; the step of inputting the evaluation information set into the evaluation model cluster for evaluation to obtain multiple different evaluation results includes: Based on the initial quality of the task and the final output of the task, the first dedicated evaluation model is used to evaluate the degree of goal achievement and generate a first evaluation result. Based on the planning sequence text, the logical quality of the planning is evaluated using the second dedicated evaluation model, and a second evaluation result is generated. Based on the actual action sequence and the planning sequence text, the third dedicated evaluation model is used to evaluate the plan fidelity and generate a third evaluation result. Based on the actual action sequence, the effectiveness and efficiency of action execution are evaluated using the fourth dedicated evaluation model, and a fourth evaluation result is generated.

4. The evaluation method for intelligent agents as described in claim 3, characterized in that, The step of evaluating the effectiveness and efficiency of action execution using the fourth dedicated evaluation model based on the actual action sequence, and generating a fourth evaluation result, includes: Based on the preset tool interface specification library, the integrity and format compliance of the input parameters of each tool call in the actual action sequence are verified, and whether they meet the calling preconditions are evaluated to generate a validity evaluation sub-result. Analyze the actual action sequence to identify redundant calls, consecutive calls that can be merged, or inefficient execution steps with better alternatives, and generate efficiency evaluation sub-results. The fourth evaluation result is generated based on the effectiveness evaluation sub-result and the efficiency evaluation sub-result.

5. The evaluation method for intelligent agents as described in claim 3, characterized in that, The evaluation model cluster also includes a fifth dedicated evaluation model; the step of inputting the evaluation information set into the evaluation model cluster for evaluation to obtain multiple different evaluation results further includes: Based on the evaluation information set, the fifth dedicated evaluation model is used to evaluate the logical consistency between the intelligent agent's reasoning and declaration in the original running trajectory data, and a fifth evaluation result is generated.

6. The evaluation method for intelligent agents as described in claim 1, characterized in that, The step of fusing and analyzing multiple different evaluation results to generate a comprehensive evaluation report includes: Based on preset weights, the quantitative scores in each of the evaluation results are weighted and calculated to obtain a comprehensive score. The comprehensive assessment report is obtained by integrating the overall score and the diagnostic information from each of the assessment results.

7. The evaluation method for intelligent agents as described in any one of claims 1-6, characterized in that, The method further includes: Based on the quantitative score and diagnostic information in the comprehensive evaluation report, identify the defective links that cause the quantitative score to be lower than a preset threshold or the diagnostic information to indicate errors. Optimization suggestions are generated based on the mapping relationship between the defective link and the intelligent agent, and at least one module in the intelligent agent is adjusted according to the optimization suggestions.

8. An evaluation device for intelligent agents, characterized in that, include: The acquisition and processing unit is used to acquire the original running trajectory data generated after the intelligent agent executes the target task, and to parse and structure the original running trajectory data to extract the evaluation information set related to the evaluation. An evaluation unit is used to input the evaluation information set into an evaluation model cluster for evaluation to obtain multiple different evaluation results, wherein the evaluation model cluster includes multiple dedicated evaluation models configured to perform different evaluation dimensions. The generation unit is used to integrate and analyze multiple different evaluation results to generate a comprehensive evaluation report, wherein the comprehensive evaluation report includes quantitative scores and diagnostic information of the intelligent agent in goal understanding, planning and action execution.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the evaluation method for intelligent agents as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the evaluation method for intelligent agents as described in any one of claims 1 to 7.

Images