An agent automation testing method and system based on a Web platform

Abstract

The application discloses an agent automatic test method and system based on a Web platform, which comprises a test framework, a Web platform and a data storage layer, adopts six AI prompt words to form a string parallel pipeline, a large language model executes prompt word analysis to generate a three-section YAML test case set of an agent source code, a browser automation tool is used to create tasks in batches on the Web platform and record them to a task state list, breakpoint continuous measurement is supported, a state machine is used to poll and execute routing, intermediate reasoning and three-dimensional verification of product quality, a backend is directly connected or an authentication layer is bypassed through an interface, a session track and a product list containing tool calling and execution planning are accurately extracted, finally, a parallel execution process and product multi-dimensional quantitative evaluation are performed, a claim cross verification anti-illusion mechanism is introduced to verify claims one by one, and finally, an agent test report with eight sections is merged and exported. The application realizes full-link automatic test, and ensures that the evaluation result is real, reliable and traceable.

Description

Technical Field

[0001] This invention relates to the field of Web automated testing technology, and in particular to an automated testing method and system for intelligent agents based on a Web platform. Background Technology

[0002] Current specialized testing methods for AI agent behavior are inadequate. Traditional software testing, centered on deterministic input / output assertions, is unsuitable for the open-domain generation of outputs and autonomous decision-making behaviors of AI agents. Existing AI agent evaluation frameworks (such as AgentBench and G-Eval) are only for offline benchmarking and cannot interface with real web platforms, nor do they possess engineering capabilities such as batch execution, breakpoint resumption, conversation trajectory extraction, and anti-illusion cross-validation. In particular, the scoring conclusions of existing large-model-based evaluation paradigms (such as G-Eval) heavily rely on the output statements of the large model itself, lacking a step-by-step verification process with the actual execution records of the tested agent. This easily leads to illusory scoring that does not conform to reality, and the scoring conclusions cannot be traced back to specific original interaction evidence, resulting in unreliable and untraceable evaluation results. Given this technological gap, engineering practice often adopts the following two types of solutions: Existing Solution A: Manual Web Interface Testing; This relies on testers manually operating the web interface through a browser, and has three core drawbacks: ① The test case design lacks standardization and cannot systematically cover the three dimensions of routing correctness, inference behavior, and output products; ② It requires manual operation one by one and continuous waiting for AI responses, making batch processing impossible and lacking the ability to resume testing after a breakpoint; ③ Evaluation relies on subjective observation, making it impossible to quantify process quality and product quality, and lacking platform backend traceability capabilities.

[0003] Existing Solution B: Fixed-rule script-based automated testing; the script-based solution has fundamental limitations: ① Fixed rules cannot perform effective semantic assertions on the open-domain output of the AI ​​Agent, and the maintenance cost increases sharply with version iterations; ② It can only verify the final output of the web interface and cannot evaluate the tool call order and intermediate inference process; ③ It is completely isolated from the web platform's backend database and object storage, and cannot obtain the intermediate inference trajectory during the execution of the AI ​​Agent.

[0004] Therefore, there is an urgent need for an automated testing method for AI agents based on a web platform. This method should be able to automatically generate standardized test cases covering three dimensions—routing correctness, intermediate reasoning process, and output quality—from the agent's source code. It should support batch asynchronous execution of test tasks and breakpoint resumption testing. Furthermore, it should achieve quantifiable and traceable multi-dimensional evaluation through cross-validation with the original session trajectory, thereby filling the systematic gap in the field of automated testing of the entire AI agent chain on a web platform. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide an automated testing method and system for intelligent agents based on a Web platform.

[0006] To achieve the objectives of this invention, the technical solution adopted is as follows: An automated testing system for intelligent agents based on a web platform includes: The test framework includes a task status list and an AI-driven module group, which is used to drive the first to sixth AI prompt words to orchestrate the test pipeline. The web platform includes a web front-end, an API gateway, a business service layer, and a user container. The user container is used to carry out the toolchain execution of the intelligent agent under test and write the session messages and tool calls during the execution process into the business service layer. The data storage layer includes a session database for storing business data and session traces, and an object storage service for storing output artifact files. The testing framework interacts with the Web platform through two paths: Path 1 involves using browser automation tools to operate the Web frontend, completing task creation, status polling, and 3D verification; Path 2 involves directly accessing the data storage layer via remote connection or bypassing the Web platform authentication layer through the backend data interface to extract the session trajectory and product list containing the intermediate reasoning process.

[0007] An automated testing method for intelligent agents based on a web platform includes the following steps: Step S1: The large language model executes the first AI prompt word, parses the source code of the agent under test, and generates a three-part YAML test case set; Step S2: The large language model executes the second AI prompt word, creates test tasks in batches on the Web platform through browser automation tools, and writes the task identifier and initial state into the task status list; Step S3: The large language model executes the third AI prompt word, judges the task status through state machine polling, performs three-dimensional verification of the completed task, including routing correctness, analysis of intermediate inference process, and output product quality, and writes the verification results back to the task status list; Step S4: The large language model executes the fourth AI prompt word, uses the task identifier that passed the verification result as the index, extracts the intermediate inference trajectory to generate the session trajectory data storage, and enumerates the output files to generate the product list data storage; Step S5: The large language model executes the fifth AI prompt word, performs a full-process multi-dimensional quantitative evaluation based on the conversation trajectory data storage, and generates and stores the process evaluation results; Step S6: The large language model executes the sixth AI prompt word, performs multi-dimensional quantitative evaluation of the product based on the product list data storage, and uses the conversation trajectory data storage to perform claim cross-validation anti-illusion detection on the evaluation statement, generating and storing the product evaluation results; Step S7: Merge the process evaluation results storage and the product evaluation results storage, and export the agent test report file.

[0008] Further, step S1 specifically includes: Skill definitions and trigger conditions are extracted from the source code of the agent under test and mapped to routing verification test cases; tool call relationships and data flow structures are mapped to analysis verification test cases; and output file directories and format specifications are mapped to output verification test cases. YAML test cases are generated one by one, and each test case contains six standard fields: description, files, expected.routing, expected.analysis.behavior, expected.analysis.content, and expected.output.

[0009] Further, step S2 specifically includes: Read in the YAML test cases, and sequentially operate the Web platform interface to complete file upload, description input and submission, trigger the execution of the tested agent, obtain the task UUID returned by the platform and record the key timestamp; persist the task_id, url, status, timing and verification result fields of each task to the task status list. In the initial stage, set the status field status=created and the verification result to an empty value, as the status anchor point for resuming test from breakpoint.

[0010] Furthermore, step S3 specifically includes: State machine polling: The page state is judged sequentially at preset intervals and preset maximum number of rounds. It automatically identifies and processes pop-up interactive elements and autonomously judges the page content through semantic understanding. If an error message, abnormal interruption or task termination failure signal is detected, the polling will immediately stop. 3D Validation: Routing Validation: Reads the AI's response content and matches it with the target Skill name declared in expected.routing, counting the number of passes, failures, and skips; Analysis Validation: Constructs a structured query from the expected behavior list and expected content list and injects it into the conversation response text. The large language model returns a judgment conclusion and reason for each item according to the preset structured format, and the overall conclusion is the most severe level among all items; Output Validation: Confirms whether the actual output path and file extension match the directory structure declared in expected.output through the platform file interface; Writes the overall conclusion of the 3D validation back to the task status list and updates the task status to one of the three states: pass, fail, or partial.

[0011] Further, step S4, extracting intermediate inference trajectories, includes: The platform's session records and output paths are accurately located by directly connecting to the session database and object storage service layer via path two. The structure of the session trajectory data storage includes: basic session information, message list, tool call list containing tool name, input parameters, output results and call sequence, execution plan record, and session-level aggregate statistics.

[0012] Furthermore, steps S5 and S6 are executed in parallel, wherein: Step S5 quantifies and scores the tested agent from five dimensions: step completeness, tool usage efficiency, reasoning accuracy, error handling, and workflow consistency, and outputs the scores by weighting them according to preset weights. Step S6 quantifies and scores the output product from five dimensions: completeness, accuracy, structure, clarity, and coverage. Each product evaluation claim must be verified through the claim cross-validation anti-illusion mechanism. If the evidence is inconsistent, it is judged as an illusion evaluation and the corresponding dimension score is revoked.

[0013] Furthermore, the claim cross-validation anti-hallucination mechanism includes: (1) The criteria for determining hallucination are divided into three categories: factual inconsistency (H1), lack of supporting evidence (H2), and logical contradiction (H3); (2) Three-layer evidence retrieval method, which adopts a layered strategy for retrieval: the first layer is the message text layer, the second layer is the tool call layer, and the third layer is the product record layer; (3) The rules for canceling scores at different granularities are as follows: deducting weight for single contradictions, downgrading the quantity threshold, capping the overall score, and non-transitivity. (4) The format of the evidence appendix is ​​standardized, with the product file path as the primary key name, and the storage is expanded under each evaluation dimension.

[0014] Furthermore, the agent test report file generated in step S7 includes an evaluation report with an eight-segment structure and a structured summary file. The eight segments are: overview score, session information, process evaluation, product evaluation, key findings, problem list, improvement suggestions, and evidence appendix.

[0015] The beneficial effects of this invention are that, compared with the prior art, this invention achieves full-link automation from intelligent agent source code to evaluation report through a six-level AI prompt word string parallel pipeline, eliminating the dependence of test execution on manual operation; by remotely connecting directly to the database and object storage through path two and bypassing the Web platform authentication layer, it accurately extracts intermediate inference trajectories such as session messages, message fragments, tool calls and execution plans, solving the technical bottleneck of existing solutions that can only verify the final output of the Web interface and are isolated from the backend storage, thus failing to obtain the intermediate inference process.

[0016] This invention systematically covers the three dimensions of routing (routing correctness), analysis (intermediate reasoning process), and output (product quality) through a three-stage YAML test case structure, filling the structural gap in existing test frameworks that can only evaluate the final output; the task status list persistence mechanism supports breakpoint resumption testing after any node is interrupted, and completed tasks do not need to be executed again, significantly improving the engineering reliability of batch testing.

[0017] This invention verifies the consistency between the evaluation statement and the original conversation trajectory one by one through a claim cross-validation mechanism. It prevents the large model's hallucination scoring from contaminating the test conclusions through hallucination determination, three-layer evidence retrieval, and granular score revocation. It also ensures that each scoring conclusion can be traced back to the specific original interaction record, thus ensuring the credibility and traceability of the evaluation conclusions.

[0018] Each AI prompt word in this invention can be independently configured with a different large-scale language model, allowing implementers to flexibly adapt it according to the reasoning depth and cost requirements at each stage, thus exhibiting good scalability. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the web-based intelligent agent automated testing system described in this invention; Figure 2 This is a flowchart of the intelligent agent automated testing method based on the web platform described in this invention; Figure 3 This is the execution flowchart for the first AI prompt word-driven test case generation; Figure 4 This is the execution flowchart of the second AI prompt word-driven state machine; Figure 5 This is the execution flowchart of the third AI prompt word-driven 3D verification; Figure 6It is an execution flowchart of the dual-dimensional evaluation and cross-validation driven by the fourth, fifth, and sixth AI prompt words. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of this application.

[0021] like Figure 1 As shown, the intelligent agent automated testing system based on the web platform of this invention consists of three layers: a local testing framework, a web platform, and a data storage layer. The testing framework includes an AI-driven module group and a task status list. The AI-driven module group drives the first to sixth AI prompts. The AI-driven module group interacts with the web platform through two paths: Path one operates the web frontend via a browser automation tool to complete task creation, status polling, and 3D verification; Path two directly accesses the data storage layer via a remote connection to extract session trajectories and enumerate product files. The web platform consists of a web frontend, an API gateway, a business service layer, and a user container. The web frontend is used for task submission, status query, and file management, and performs routing scheduling through the API gateway. The user container carries the execution of the AI ​​intelligent agent's toolchain, writes session messages and tool calls to the business service layer, and uploads product files to the object storage service. The data storage layer consists of a database (storing business data and session trajectories) and an object storage service (storing output product files). The testing framework directly accesses the database and object storage service via Path two, bypassing the web platform authentication layer to achieve accurate extraction of trajectories and products.

[0022] This invention employs a parallel pipeline of six AI prompts, using a shared data medium for relay storage to achieve decoupled collaboration between prompts at each level. The first AI prompt takes the agent's source code as input and outputs a three-part YAML test case set, which serves as input for the second AI prompt. The second AI prompt creates test tasks one by one on the web platform based on the test case set, writing the task's unique identifier (UUID) into the task status list, which serves as input for the third AI prompt. The third AI prompt reads the task status list, completes three-dimensional verification, and writes the timestamp and verification result back to the task status list. The fourth AI prompt uses the task UUID in the task status list as an index to query the corresponding session identifier in the web platform's backend database, extracting dialogue messages, message fragments, execution plans, and aggregate statistics to generate session trajectory data storage and product list data storage, which serve as input for the fifth and sixth AI prompts, respectively. The fifth and sixth AI prompts execute in parallel, generating process evaluation result storage and product evaluation result storage, respectively, which are then merged to output the final evaluation report.

[0023] Each AI prompt is implemented by a large language model with long text understanding and code generation capabilities, and supports independent configuration. For example, models with stronger capabilities can be configured for the fifth and sixth AI prompts (process evaluation and claim cross-validation) which have higher reasoning requirements, and models with faster response can be configured for the second AI prompt for batch creation tasks.

[0024] The technical solution of the present invention consists of a sequence of technical actions that can be executed by a computer system, including but not limited to: browser automated operation, state machine polling, database query, object storage enumeration, file reading and writing, and structured verification, etc.; the above-mentioned sequence of technical actions is generated and arranged by a large language model driven by multi-level AI prompt words.

[0025] This invention does not limit the number or boundary division of the AI ​​driving modules (Agents) that carry these prompts. Regardless of whether the implementer integrates six AI prompts into one driving module, three driving modules, or any other combination, as long as the sequence of technical actions executed, the data flow relationship, and the input-output correspondence of each level of prompts conform to the description of this invention, they should all fall within the protection scope of this invention.

[0026] like Figure 2 The detailed technical steps of the web-based automated testing method for intelligent agents described in this invention are as follows: Step S1: The large language model executes the first AI prompt word and generates a three-part YAML test case set; Driven by the first AI prompt word, the large language model takes the agent's source code as input, extracts skill definitions, tool call relationships and output specifications, and generates a YAML format test case set with a three-part structure of routing (routing correctness), analysis (intermediate reasoning process) and output (product quality), which serves as standardized input for subsequent steps.

[0027] Step S2: The large language model executes the second AI prompt word, batches test tasks, and writes them to the status list; The system uses a second AI prompt word to drive a large language model and a browser automation tool to drive the web platform interface. It reads the description and associated files of the test cases one by one, completes the file upload, description input and submission operations, triggers the run of the Agent under test, obtains the task UUID returned by the platform, records key timestamps, writes the task identifier and initial state into the task status list, and supports breakpoint resume testing.

[0028] Step S3: The large language model executes the third AI prompt word to complete the state machine polling and three-dimensional verification; Driven by a third AI prompt word, the large language model reads the tasks to be verified from the task status list, determines the completion of the task by polling the state machine, performs three-dimensional verification of routing, analysis, and output on the completed task, and writes the verification results (pass / fail / partial) back to the task status list.

[0029] Step S4: The large language model executes the fourth AI prompt word to extract the conversation trajectory and output list; Driven by the fourth AI prompt word, the large language model uses the taskUUID with the verification conclusion of "pass" in the task status list as an index to locate the corresponding session_id in the web platform session database, extract dialogue messages, tool calls, execution plans and aggregate statistics, and generate session trajectory data storage; at the same time, it enumerates all output files in the product catalog to generate product list data storage, providing raw data for steps S5 and S6.

[0030] Step S5: The large language model executes the fifth AI prompt word to complete the process quality assessment; Driven by the fifth AI prompt word, the large language model is used to quantitatively evaluate the tested intelligent agent based on the data storage of conversation trajectory. The evaluation is carried out from five dimensions: step completeness, tool usage efficiency, reasoning accuracy, error handling, and workflow consistency. Each dimension is given a score of 0 to 10 and textual evidence. The weighted and merged results are used to generate and store the process evaluation results.

[0031] Step S6: The large language model executes the sixth AI prompt word to complete the product quality assessment (including claim cross-validation). Simultaneously, a large language model driven by a sixth AI prompt word is used to quantitatively evaluate the output products based on the product list data storage, considering five dimensions: completeness, accuracy, structure, clarity, and coverage. Each evaluation claim must be cross-validated with the conversation trajectory data storage. If the evidence is inconsistent, it is judged as a hallucination evaluation, and the corresponding dimension score is revoked, ensuring the traceability of the evaluation conclusions and generating and storing the product evaluation results. The claim cross-validation anti-hallucination mechanism includes four sub-steps: hallucination determination, three-layer evidence retrieval, granular score revocation, and structured evidence appendix.

[0032] Step S7: Merge reports and export; The process evaluation results and product evaluation results are merged to generate an evaluation report that includes an overview score, session information, process evaluation, product evaluation, key findings, issue list, improvement suggestions, and evidence appendix. Finally, an agent test report file is generated.

[0033] like Figure 3As shown, the first AI prompt word drives the large language model to parse the source code of the agent under test, extracting Skill definitions and triggering conditions, tool call relationships and data flow structure, output file directory and format specifications to form an initial set of test dimensions. Skill definitions and triggering conditions are mapped to routing verification cases (verifying whether the Skill route triggered by the AI ​​response is correct), tool call relationships and data flow are mapped to analysis verification cases (verifying intermediate tool call steps and reasoning processes), and output file structure and format specifications are mapped to output verification cases (verifying the final output path and file format), achieving systematic coverage of three-dimensional test cases. Finally, YAML format test cases are generated one by one according to the template, each test case containing six standard fields: description, files, expected.routing, expected.analysis.behavior, expected.analysis.content, and expected.output. Full Skill coverage is performed to ensure that each Skill definition corresponds to at least one routing test case. Routing robustness test cases are automatically added, such as for abnormal routing scenarios like invalid input and boundary input. Version metadata is written to the file header, generating a timestamp and the tested Agent version number, and a coverage check is performed.

[0034] like Figure 4 As shown, the second AI prompt word drives the large language model to call the browser's automation tools to operate the web platform, automatically completing task creation and state persistence. It reads the platform configuration file to obtain mapping relationships and reads the YAML test case set generated in step S1. The test environment is selected from multiple preset environments according to user coverage parameters or default strategies. Each environment can be configured with corresponding task submission parameters and test accounts as needed to support multi-user concurrency. Subsequently, tasks are executed serially test by test case, recording key timestamps and forcibly waiting at preset intervals to prevent triggering frequency limits. The task_id, url, status, timing, and verification result fields of each task are persisted to the status list file. In Phase A, status=created is written and the verification result is set to empty, providing a state anchor point for breakpoint continuation testing in step S3.

[0035] like Figure 5 As shown, the third AI prompt word drives the large language model, which reads the tasks to be verified from the task status list that have status=created and no verification result. The state machine polls to determine the completion status, performs three-dimensional verification on each completed task, and writes the verification result back to the task status list, forming a Phase B closed loop.

[0036] State machine polling phase: The page status of each task is judged sequentially at preset intervals and preset maximum number of rounds; during the polling process, various interactive elements that may pop up on the web platform, such as multiple choice questions, questionnaires, confirmation boxes, etc., are automatically identified and processed; the large language model performs semantic understanding of the page content in real time, and autonomously judges whether failure signals occur, including but not limited to error messages, abnormal interruptions and task termination flags. When a failure is determined, the task status is immediately marked as failed and the polling ends.

[0037] The three-dimensional verification phase includes: The routing verification reads the AI ​​response content and matches it with the target Skill name declared in `expected.routing`, outputting NP (passes), NF (failures), and NS (skips) counts; the analysis verification uses a third AI prompt to construct a structured query from the expected behavior list (`expected.analysis.behavior`) and expected content list (`expected.analysis.content`), injects it into the conversation response text, and drives the large language model to return each judgment result (including item description, judgment conclusion, and judgment reason) in a preset structured format; after the judgment results are parsed by the script, the conclusions of each item are summarized and written into the verification result field of the task status list, and the overall conclusion is the most severe level among all items (any failure results in an overall failure, all passes result in an overall pass); the output verification reads the output file directory structure declared in `expected.output`, confirms whether the actual output path and file extension match through the platform file interface, and outputs the overall pass / fail conclusion. The 3D verification results are written to the log in a single-line report format (task_id|url|status|routing:NP / NF / NS|analysis|output). At the same time, the verification result fields are written back to the task status list via the status list read / write module, and the task status is updated to one of the three states: pass, fail, or partial.

[0038] Supports breakpoint resumption testing: When step S3 is re-executed after any node is interrupted, the task discovery section automatically filters tasks with existing verification result fields, and only performs verification on tasks with status=created and verification result is empty, avoiding duplicate processing.

[0039] like Figure 6As shown, the fourth AI prompt uses the task_uuid obtained and written to the list when creating the task in step S2 as an index to read the task with the verification conclusion of pass, and accurately locates the platform session record and product path: locate the corresponding session_id in the session database (database) and locate the corresponding task path in the product directory (object storage service); extract the complete dialogue message sequence, message fragments (including tool call requests and responses), execution plan records and session-level aggregate statistics (total number of messages, total number of tool calls, total time consumption) and write them to the session trajectory data storage; and enumerate all output files under the task path, record the path, file type, size and generation time of each product and write them to the product list data storage.

[0040] The session trace data storage structure includes: basic session information (such as task_id, session_id, creation time), message list (such as role, content, timestamp), tool call list (such as tool name, input parameters, output results, call sequence), execution plan records and aggregate statistics, providing a complete original chain of evidence for the evaluation of steps S5 and S6.

[0041] It should be noted that the step of extracting intermediate trajectories and products by remotely connecting to the database and object storage service via path two is only a preferred implementation method. In other embodiments, equivalent session trajectories and product list data can also be extracted through the backend data interface (API), operation log interface, or server push event (SSE) stream exposed by the web platform. As long as the equivalent technical effect of bypassing the interface layer and obtaining the intermediate inference process data of the tested intelligent agent is achieved, it is an implementation method of the present invention.

[0042] like Figure 6 As shown, steps S5 and S6 are executed in parallel. The fifth AI prompt reads the conversation trajectory and provides a quantitative score of 0 to 10 and textual evidence for each of the five dimensions: step completeness (whether all necessary steps are executed), tool usage efficiency (the rationality and redundancy of tool calls), reasoning accuracy (whether intermediate reasoning conclusions are consistent with the task objectives), error handling (detection and retry of anomalies), and workflow consistency (the logical connection and reasonable order of each step). The results are then weighted and merged according to preset configurable weights (the sum of the weights of each dimension is 100%, which can be adjusted by the implementer according to the scenario) and the process evaluation results are stored.

[0043] The sixth AI prompt reads the product list and provides quantitative scores and evaluation statements for each type of output product from five dimensions: completeness (whether it covers all expected output items), accuracy (whether the content matches the task requirements), structure (organizational hierarchy and format standardization), clarity (readability of expression), and coverage (the coverage ratio of the required items). The product evaluation results are then stored after being weighted and merged according to preset configurable weights (the sum of the weights of each dimension is 100%, and the implementer can adjust them according to the product type).

[0044] LLM evaluation claim cross-validation closed loop: Each evaluation claim is cross-validated with the original dialogue data in the session trace data store to prevent hallucination scoring and ensure the credibility of the evaluation conclusions.

[0045] In one embodiment, the anti-hallucination mechanism for claim cross-validation executed by the sixth AI prompt word includes four sub-steps: hallucination determination criteria, three-layer evidence retrieval, granular score withdrawal, and structured evidence appendix, as detailed below: (1) Criteria for determining hallucinations; Hallucination assessments are categorized into three types: Category 1 (H1, Factual Inconsistency): The specific value asserted in the evaluation statement differs from the corresponding value recorded in the tool call output field of the session trajectory data storage beyond the preset tolerance. If the difference exceeds the tolerance, it is considered a contradiction; or the evaluation statement claims that a certain file has been read, but there is no corresponding read type operation record in the tool call list of the session trajectory data storage. Category 2 (H2, no supporting evidence): After three layers of searching, no original textual evidence related to the evaluation statement was found in the message list, tool call records and product list data storage of the session trajectory data storage. The facts involved in the statement cannot be verified by any original interaction records. Category 3 (H3, logical contradiction): The evaluation statement describes a certain execution step as being completed successfully, but the status field of the corresponding tool call in the session trace data storage contains unresolved error records, that is, the claimed success contradicts the actual execution status.

[0046] Clearly defined: If the wording of the assessment differs from that of the original text but is semantically consistent, it does not constitute a hallucination; if the assessment statement is a reasonable comprehensive inference rather than a specific factual assertion, it does not trigger a hallucination determination; only the above three situations, H1, H2, and H3, trigger the hallucination determination process.

[0047] (2) Three-tiered evidence retrieval method; Evidence retrieval employs a hierarchical strategy, performed sequentially at the following three levels: The first layer (message text layer): using the key entities and values ​​extracted from the evaluation statement as query items, it traverses the content fields of each message in the message list of the session trajectory data storage, locates the text fragments containing relevant content, and extracts the role identifier, content and timestamp of the matching message as evidence reference; The second layer (tool invocation layer) iterates through message fragments of tool invocation type in the session trajectory data storage, locates the corresponding input parameters and output result fields according to the tool type field and invocation sequence, and is used to verify the specific tool invocation facts (such as whether the target file was actually read and the actual output content of the script). The third layer (product record layer): In the product list data storage, the actual product record is located by file path, format field and generation time field. The assertions involving products in the evaluation statement are checked item by item using the file existence verification field, session trace declaration consistency field and difference record field.

[0048] (3) Rules for canceling scores based on granularity; Score cancellation is handled according to the following four levels of rules: Single contradictory claim deduction rule: In the accuracy dimension scoring calculation, each claim judged as H1 (contradicted) is deducted from the number of verified claims by a preset deduction coefficient. The accuracy calculation formula is: accuracy_ratio = (verified_count − contradicted_count × deduction coefficient) / total_claims; H1 claims have a higher negative contribution than H2 (unverifiable) claims, reflecting that contradictory claims are more harmful than unverifiable claims; Contradiction Quantity Threshold Downgrade Rule: When the number of H1 class declarations in a single product file exceeds the preset threshold, the accuracy dimension score of that product is directly reduced to the lowest level, and the proportional calculation is no longer performed. Overall score cap rule: When a critical issue is detected during the evaluation (such as missing core chapters or multiple contradictory H1 statements), the overall score cap is set to the preset first cap value; when a fatal issue is detected (such as all statement verification failures or unrecoverable error states in all tool calls), the overall score cap is further reduced to the preset second cap value; Non-transitive rule: The revocation of a score in a single dimension does not automatically propagate to other evaluation dimensions; the scores for process quality evaluation (the fifth AI prompt word output) and product quality evaluation (the sixth AI prompt word output) are calculated independently, and the revocation of one dimension will not affect the result of the other.

[0049] Each revocation operation writes a revocation entry to the problem record field of the product evaluation results storage, recording the path of the revoked product file, the severity level of the problem, the problem type classification, and a detailed description. The detailed description includes a specific comparison between the original statement and the original evidence value.

[0050] (4) Format guidelines for evidence appendices; The product evaluation results are stored in the following structure as an evidence appendix, with the product file path as the primary key, and expanded under each evaluation dimension: The evidence appendix for the accuracy dimension includes the following fields: score, number of verified statements, number of contradictory statements, number of unverifiable statements, and accuracy ratio. Each evidence appendix entry contains three mandatory fields: the original assessment statement field records the original assessment statement; the original evidence record field records the original tool call output or message content excerpt, and indicates the evidence source type and entry number; and the contradiction severity level field indicates the contradiction severity level. Verified statements are counted through the verification pass field, and unverifiable statements are counted through the unverifiable count field. All contradictory statements are listed in the contradictory statement list field in array form, with each entry pointing to a specific original record, ensuring the completeness and traceability of the assessment conclusions.

[0051] Step S7: Merge the process evaluation documents and product evaluation documents to generate a structure evaluation report.

[0052] Merge the process evaluation document from step S5 with the product evaluation document from step S6 to generate an evaluation report and a structured summary document with an eight-segment structure: overall score (process quality score, product quality score and comprehensive score), conversation information (task_id, conversation duration, total number of messages, etc.), process evaluation (five-dimensional score details), product evaluation (five-dimensional score details), key findings (highlighted strengths and core issues), issue list (prioritized items to be improved), improvement suggestions, and evidence appendix (original text of cross-validated dialogue evidence).

[0053] The applicant of this invention has provided a detailed description of the embodiments of the invention in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred embodiments of the invention. The detailed description is only intended to help readers better understand the spirit of the invention and is not intended to limit the scope of protection of the invention. On the contrary, any improvements or modifications made based on the inventive spirit of the invention should fall within the scope of protection of the invention.

Claims

1. An automated testing system for intelligent agents based on a Web platform, characterized in that, include: The test framework includes a task status list and an AI-driven module group, which is used to drive the first to sixth AI prompt words to orchestrate the test pipeline. The web platform includes a web front-end, an API gateway, a business service layer, and a user container. The user container is used to carry out the toolchain execution of the intelligent agent under test and write the session messages and tool calls during the execution process into the business service layer. The data storage layer includes a session database for storing business data and session traces, and an object storage service for storing output artifact files. The testing framework interacts with the Web platform through two paths: Path 1 involves using browser automation tools to operate the Web frontend, completing task creation, status polling, and 3D verification; Path 2 involves directly accessing the data storage layer via remote connection or bypassing the Web platform authentication layer through the backend data interface to extract the session trajectory and product list containing the intermediate reasoning process.

2. A web-based automated testing method for intelligent agents, based on the web-based automated testing system for intelligent agents as described in claim 1, characterized in that, Includes the following steps: Step S1: The large language model executes the first AI prompt word, parses the source code of the agent under test, and generates a three-part YAML test case set; Step S2: The large language model executes the second AI prompt word, creates test tasks in batches on the Web platform through browser automation tools, and writes the task identifier and initial state into the task status list; Step S3: The large language model executes the third AI prompt word, judges the task status through state machine polling, performs three-dimensional verification of the completed task, including routing correctness, analysis of intermediate inference process, and output product quality, and writes the verification results back to the task status list; Step S4: The large language model executes the fourth AI prompt word, uses the task identifier that passed the verification result as the index, extracts the intermediate inference trajectory to generate the session trajectory data storage, and enumerates the output files to generate the product list data storage; Step S5: The large language model executes the fifth AI prompt word, performs a full-process multi-dimensional quantitative evaluation based on the conversation trajectory data storage, and generates and stores the process evaluation results; Step S6: The large language model executes the sixth AI prompt word, performs multi-dimensional quantitative evaluation of the product based on the product list data storage, and uses the conversation trajectory data storage to perform claim cross-validation anti-illusion detection on the evaluation statement, generating and storing the product evaluation results; Step S7: Merge the process evaluation results storage and the product evaluation results storage, and export the agent test report file.

3. The automated testing method for intelligent agents based on a Web platform according to claim 2, characterized in that, Step S1 specifically includes: Skill definitions and trigger conditions are extracted from the source code of the agent under test and mapped to routing verification test cases; tool call relationships and data flow structures are mapped to analysis verification test cases; and output file directories and format specifications are mapped to output verification test cases. YAML test cases are generated one by one, and each test case contains six standard fields: description, files, expected.routing, expected.analysis.behavior, expected.analysis.content, and expected.output.

4. The automated testing method for intelligent agents based on a Web platform according to claim 2, characterized in that, Step S2 specifically includes: Read in the YAML test cases, and sequentially operate the Web platform interface to complete file upload, description input and submission, trigger the execution of the tested agent, obtain the task UUID returned by the platform and record the key timestamp; persist the task_id, url, status, timing and verification result fields of each task to the task status list. In the initial stage, set the status field status=created and the verification result to an empty value, as the status anchor point for resuming test from breakpoint.

5. The automated testing method for intelligent agents based on a Web platform according to claim 2, characterized in that, Step S3 specifically includes: State machine polling: The page state is judged sequentially at preset intervals and preset maximum number of rounds. It automatically identifies and processes pop-up interactive elements and autonomously judges the page content through semantic understanding. If an error message, abnormal interruption or task termination failure signal is detected, the polling will immediately stop. 3D Validation: Routing Validation: Reads the AI's response content and matches it with the target Skill name declared in expected.routing, counting the number of passes, failures, and skips; Analysis Validation: Constructs a structured query from the expected behavior list and expected content list and injects it into the conversation response text. The large language model returns a judgment conclusion and reason for each item according to the preset structured format, and the overall conclusion is the most severe level among all items; Output Validation: Confirms whether the actual output path and file extension match the directory structure declared in expected.output through the platform file interface; Writes the overall conclusion of the 3D validation back to the task status list and updates the task status to one of the three states: pass, fail, or partial.

6. The automated testing method for intelligent agents based on a Web platform according to claim 2, characterized in that, The extraction of intermediate inference trajectories in step S4 includes: The platform's session records and output paths are accurately located by directly connecting to the session database and object storage service layer via path two. The structure of the session trajectory data storage includes: basic session information, message list, tool call list containing tool name, input parameters, output results and call sequence, execution plan record, and session-level aggregate statistics.

7. The automated testing method for intelligent agents based on a Web platform according to claim 2, characterized in that, Steps S5 and S6 are executed in parallel, wherein: Step S5 quantifies and scores the tested agent from five dimensions: step completeness, tool usage efficiency, reasoning accuracy, error handling, and workflow consistency, and outputs the scores by weighting them according to preset weights. Step S6 quantifies and scores the output product from five dimensions: completeness, accuracy, structure, clarity, and coverage. Each product evaluation claim must be verified through the claim cross-validation anti-illusion mechanism. If the evidence is inconsistent, it is judged as an illusion evaluation and the corresponding dimension score is revoked.

8. The automated testing method for intelligent agents based on a Web platform according to claim 7, characterized in that, The claim cross-validation anti-hallucination mechanism includes: (1) The criteria for determining hallucination are divided into three categories: factual inconsistency (H1), lack of supporting evidence (H2), and logical contradiction (H3); (2) Three-layer evidence retrieval method, which adopts a layered strategy for retrieval: the first layer is the message text layer, the second layer is the tool call layer, and the third layer is the product record layer; (3) The rules for canceling scores at different granularities are as follows: deducting weight for single contradictions, downgrading the quantity threshold, capping the overall score, and non-transitivity. (4) The format of the evidence appendix is ​​standardized, with the product file path as the primary key name, and the storage is expanded under each evaluation dimension.

9. The automated testing method for intelligent agents based on a Web platform according to claim 2, characterized in that, The agent test report file generated in step S7 includes an evaluation report with an eight-segment structure and a structured summary file. The eight segments are: overview score, session information, process evaluation, product evaluation, key findings, problem list, improvement suggestions, and evidence appendix.

Images