Graphical user interface agent robustness dynamic evaluation system and method
By constructing a dynamic evaluation system, combined with a semantic rule engine and multi-layer state verification, the robustness evaluation problem of GUI agents in complex environments is solved. This enables quantitative evaluation of the controllable anomaly injection and self-recovery capabilities of GUI agents in real environments, thereby improving the authenticity and objectivity of the evaluation.
Patent Information
- Application Number
- CN202511650910.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-12
- Publication Date
- 2026-01-30
AI Technical Summary
Existing GUI agent evaluation methods lack dynamism and have incomplete anomaly modeling, failing to reflect their robustness and self-recovery capabilities in complex scenarios in real-world environments. Traditional evaluation systems rely on static datasets and single task paths, making it impossible to simulate multi-step decision-making and dynamic interactions in actual systems.
This paper provides a system and method for dynamically evaluating the robustness of a graphical user interface (GUI) intelligent agent. Through the collaborative work of a task execution environment module, an exception triggering mechanism module, a state verification module, and a data acquisition module, the system enables controllable exception injection and state verification of the GUI intelligent agent in a real environment. A semantic rule engine is used to identify triggering conditions and generate dynamic exception scenarios. Combined with multi-layer state verification and multi-source data acquisition, the system calculates indicators such as robustness success rate.
It significantly improves the authenticity and objectivity of the evaluation, can quantify the robustness and self-recovery ability of GUI agents in complex environments, supports deployment across application scenarios, and provides a unified technical platform for agent robustness research.
Smart Images

Figure CN121434033A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a system and method for dynamic evaluation of the robustness of a graphical user interface intelligent agent. Background Technology
[0002] With the rapid development of multimodal large language models (MLLM) and vision-language models (VLM), graphical user interface (GUI) agents have gradually become an important research direction in the field of artificial intelligence. GUI agents simulate human operations through visual recognition, semantic understanding, and task planning, and achieve autonomous control of computer or mobile interfaces. Their core goal is to be able to autonomously perform tasks in complex interfaces, such as application launch, information input, and page navigation, thus becoming an important foundation for the automation of human-computer interaction.
[0003] Currently, most mainstream GUI agents are trained and evaluated based on static datasets. These datasets typically consist of fixed screenshots and their corresponding operation sequences, such as Android Control, ScreenSpot, and GUI-Robust. While these evaluation environments can measure the model's operational capabilities in ideal scenarios, they lack dynamism and real-world interference factors, failing to reflect the agent's robustness in actual use. Their shortcomings are quite obvious. For example, existing static benchmarks usually assume that each task's execution path is deterministic and singular, without mid-process interference or sudden changes in interface state. However, in real-world systems, agents often encounter numerous problems, such as: System-level anomalies such as low battery warnings, network connection interruptions, and system update pop-ups; Application-level anomalies such as application crashes, permission request dialogs, and ad overlays; User experience disruptions caused by push notifications, upgrade prompts, and temporary pop-ups.
[0004] The aforementioned anomalies can significantly alter the task execution path and even lead to task failure. Existing testing frameworks lack systematic modeling and evaluation mechanisms for such situations. Although some studies have attempted to introduce "abnormal samples," such as GUI-Robust inserting warning pop-ups into screenshot sequences, they are still based on static images and cannot simulate real dynamic interaction processes and multi-step recovery paths. They also fail to reflect the interaction links, subsequent state changes, and multi-step decision-making processes in actual systems, resulting in evaluation results that are out of touch with the real environment.
[0005] Furthermore, existing evaluation systems often rely on "action accuracy" as the main indicator, only judging whether the model has performed the correct operation, without considering its adaptability and stability during the anomaly recovery process. Therefore, the current technical system has obvious limitations in dynamic evaluation, anomaly injection, recovery capability testing, and cross-application versatility.
[0006] In summary, there is an urgent need for a new dynamic evaluation framework that can automatically inject multiple types of abnormal events into a realistic simulation environment and objectively evaluate them through task result status rather than operation trajectory, so as to comprehensively reflect the robustness and self-recovery capability of the GUI agent in the real environment. Summary of the Invention
[0007] This invention provides a dynamic evaluation system and method for the robustness of graphical user interface intelligent agents. Its main purpose is to address the problems of insufficient dynamism, imperfect anomaly modeling, single task path, and limited verification methods in existing GUI intelligent agent evaluation methods. It provides a dynamic evaluation system and method for the robustness of GUI intelligent agents in real-world environments. Through controllable anomaly injection and state verification, it achieves real, comprehensive, and quantifiable robustness testing of intelligent agents.
[0008] To achieve the above objectives, the present invention provides a graphical user interface intelligent agent robustness dynamic evaluation system and method. The system includes a task execution environment module, an exception triggering mechanism module, a state verification module, a data acquisition module, and an evaluation index calculation module. The data acquisition module is communicatively connected to the task execution environment module, the exception triggering mechanism module, the state verification module, and the evaluation index calculation module. The task execution environment module is communicatively connected to the exception triggering mechanism module and the state verification module. The state verification module is communicatively connected to the exception triggering mechanism module and the evaluation index calculation module.
[0009] Furthermore, the task execution environment module includes a built-in interface state stability detection mechanism unit and a simulation framework construction unit.
[0010] Furthermore, the exception triggering mechanism module is built on the collaborative work of the semantic rule engine and the exception event database. The semantic rule engine is responsible for identifying the triggering time, determining whether the triggering conditions are met, and issuing instructions to the exception event database. The semantic rule engine is equipped with a context-aware mechanism unit.
[0011] Furthermore, the state verification module adopts a state-driven verification mechanism.
[0012] A method for dynamically evaluating the robustness of a graphical user interface (GUI) intelligent agent, applied in the GUI intelligent agent robustness dynamic evaluation system, includes the following steps: S1. Build a simulation environment, and the intelligent agent enters the simulation environment and completes the cyclical interaction process of "perception-decision-execution-feedback"; S2. Monitor abnormal events, automatically trigger them, generate abnormal scenarios, and recover from them; S3. Perform status verification and determine whether the task has been successfully completed; S4. Perform multi-source data collection for continuous recording throughout the entire operation cycle; S5. Calculate key performance indicators based on the task execution results.
[0013] Further, step S1 specifically involves the system first initializing the test scenario through the task execution environment module, loading the application to be tested and setting the task objective, generating the simulation environment. After the intelligent agent enters the simulation environment, it understands the current interface state and generates operation instructions through visual perception and structural information parsing. The system executes these instructions in real time and returns the execution results to the intelligent agent, thus forming a cyclical interactive process of "perception-decision-execution-feedback".
[0014] Further, step S2 specifically involves the following steps: during the task, the exception triggering mechanism module continuously monitors the interface semantics and context state. When a triggering condition matching a preset rule is detected, the semantic rule engine immediately calls the corresponding template in the exception event library to dynamically inject the exception into the simulation environment. At this time, the agent needs to identify the interference, make a response decision, and perform a recovery operation.
[0015] Furthermore, the abnormal dynamic injection includes a foreground interference stage and a subsequent state stage. In the foreground interference stage, the system will overlay elements such as pop-ups, prompts, or obscuring layers on the current task interface to directly interrupt the agent's operation process. In the subsequent state stage, the system changes the application state according to the agent's response action.
[0016] Further, step S3 specifically involves the following steps: after the task is completed, the status verification module automatically extracts the final interface status information and compares it with the target status in the task definition stage. The system comprehensively determines whether the task has been successfully completed through multi-level judgment of text, structure and visual features. If the agent can still recover to the target interface and achieve the expected operation after abnormal interference, the task is judged as "robust success" and its recovery path and response behavior are recorded.
[0017] Further, step S4 specifically involves continuously recording multi-source data of the task, including screenshot sequences, interface structure, operation logs, and system events, through the data acquisition module throughout the entire operation and testing cycle.
[0018] Further, step S5 specifically involves the system using the evaluation index calculation module to statistically analyze the success rate and robust success rate of the agent under normal and abnormal environments, and performing sub-analysis based on the abnormality category to generate a complete evaluation report and visualization results.
[0019] The graphical user interface intelligent agent robustness dynamic evaluation system and method provided by this invention have the following advantages: An interactive, dynamic task environment is built using an Android emulator, allowing GUI agents to perform operations and receive feedback in real time. This simulates the task execution and system response process in real-world applications, transforming the evaluation environment from static to dynamic. This significantly improves the realism and reference value of the evaluation, and solves the problems of traditional benchmarks that rely solely on static screenshots or recorded sequences and cannot simulate real-world scenarios such as system interruptions and pop-up prompts. By using a semantic rule engine to match information such as interface text and hierarchical structure, and automatically triggering corresponding abnormal events, the evaluation process can be repeated, scalable and closely resemble real human-computer interaction logic. The anomaly triggering mechanism module can systematically evaluate the agent's anomaly identification, decision-making and recovery capabilities, filling the gap in the existing evaluation in the "process robustness" dimension; The task completion status is verified by comparing the UI structure tree with semantic features, replacing the traditional method that relies on a fixed action sequence for verification. This eliminates path dependency, ensures that agents with different strategies can be correctly identified when they reach the same goal, and improves the objectivity and universality of the evaluation. The "robust success rate" metric is defined to measure the model's ability to maintain performance under abnormal conditions. It can quantitatively reflect the model's reliability in real and complex environments. A multidimensional robustness index system is established to quantify the model's robustness.
[0020] With its modular architecture design, it has good scalability and reusability. All exception rules and verification logic are declared through external configuration files, supporting user-defined exception types and task sets. This facilitates deployment across application scenarios and industry standardization, and can serve as a unified technology platform for research on the robustness of intelligent agents. Attached Figure Description
[0021] Figure 1 This is a schematic diagram of the system structure of a graphical user interface intelligent agent robustness dynamic evaluation system and method provided in an embodiment of the present invention; Figure 2 This is a schematic flowchart of a graphical user interface intelligent agent robustness dynamic evaluation system and method provided in an embodiment of the present invention.
[0022] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings.
[0023] The attached figures are labeled as follows: 1. Task execution environment module; 2. Exception triggering mechanism module; 3. Status verification module; 4. Data acquisition module; 5. Evaluation index calculation module. Detailed Implementation
[0024] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without creative effort are within the scope of protection of the invention.
[0025] This invention provides a graphical user interface intelligent agent robustness dynamic evaluation system and method. Some embodiments of this invention will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0026] In this embodiment, a dynamic benchmarking method for robust performance in a real-world environment follows a dynamic closed-loop logic of "execution—trigger—recovery—verification—evaluation." All modules operate collaboratively within a unified framework, forming an automated evaluation pipeline from task loading to robustness result output. (Refer to...) Figure 2 Specifically, it includes: S1. Set up a simulation environment. The intelligent agent enters the simulation environment and completes the cyclical interaction process of perception-decision-execution-feedback. Specifically, the system first initializes the test scenario, loads the application to be tested and sets the task objectives. After the intelligent agent enters the simulation environment, it understands the current interface state and generates operation instructions through visual perception and structural information parsing. The system executes these instructions in real time and returns the execution results to the intelligent agent, thus forming a cyclical interaction process of "perception-decision-execution-feedback". S2. Monitoring, automatic triggering, generation of abnormal scenarios and recovery of abnormal events. Specifically, during the task, the interface semantics and context state are continuously monitored. When a triggering condition matching the preset rule is detected, the semantic rule engine immediately calls the corresponding template in the abnormal event library to dynamically inject the abnormality into the simulation environment. At this time, the agent needs to identify the interference, make a response decision, and perform recovery operations. S3. Perform status verification and determine whether the task has been successfully completed. After the task is completed, the final interface status information is automatically extracted and compared with the target status in the task definition stage. The system comprehensively determines whether the task has been successfully completed through multi-level judgment of text, structure and visual features. If the agent can still recover to the target interface and achieve the expected operation after abnormal interference, the task is judged as "robust success" and its recovery path and response behavior are recorded. S4. Continuously record multi-source data of the task throughout the entire operation cycle. Specifically, continuously record multi-source data of the task throughout the entire operation test cycle, including screenshot sequences, interface structure, operation logs and system events. S5. Calculate key performance indicators based on task execution results. Specifically, the system calculates the success rate (SR) and robust success rate (RSR) of the agent under normal and abnormal environments, and performs itemized analysis according to the anomaly category to generate a complete evaluation report and visualization results.
[0027] The above testing method follows a dynamic closed-loop logic of "execution-trigger-recovery-verification-evaluation". Each module operates collaboratively under a unified framework, forming an automated evaluation pipeline from task loading to robust result output.
[0028] The above process is characterized by automated control of both test tasks and anomaly injection, and real-time linkage between verification and analysis. This constitutes a highly integrated, scalable, and reproducible dynamic evaluation system for the robustness of GUI intelligent agents. Through this method, the evaluation process can not only quantify the model's task execution capability under ideal conditions, but also reveal its adaptive and recovery capabilities when facing interference in complex real-world environments, providing systematic technical support for the robustness optimization of intelligent agents.
[0029] Reference Figure 1 A dynamic benchmark testing system for robust performance in a real-world environment includes a task execution environment module 1, an exception triggering mechanism module 2, a state verification module 3, a data acquisition module 4, and an evaluation index calculation module 5. The data acquisition module 4 is communicatively connected to the task execution environment module 1, the exception triggering mechanism module 2, the state verification module 3, and the evaluation index calculation module 5, respectively. The task execution environment module 1 is communicatively connected to the exception triggering mechanism module 2 and the state verification module 3, respectively. The state verification module 3 is communicatively connected to the exception triggering mechanism module 2 and the evaluation index calculation module 5, respectively.
[0030] In this embodiment, the task execution environment module 1 is mainly used in step S1. The system first initializes the test scenario in the task execution environment module 1, loads the application to be tested and sets the task objective. After the agent enters the simulation environment, it understands the current interface state and generates operation instructions through visual perception (screenshot) and structural information parsing (UI-level XML). The system executes these instructions in real time and returns the execution results to the agent, thus forming a cyclical interactive process of "perception-decision-execution-feedback".
[0031] The task execution environment module 1 is the underlying operating foundation of the system, which is used to provide a dynamic, controllable and highly realistic virtual operating environment for the intelligent agent. The task execution environment module 1 has a built-in interface state stability detection mechanism unit and a simulation framework building unit. The system builds a test platform based on the Android simulation framework through the simulation framework building unit, which can flexibly load different applications and reproduce the real user interaction process, thereby simulating complex interface operations and state changes without the need for physical devices. The core logic of Task Execution Environment Module 1 follows an interactive loop similar to "See-Act (Perception-Execution)". The agent first perceives the interface state through visual input (screenshot) and structured information (UI-level XML). Then, based on the understanding results, it generates operation instructions (such as click, swipe, text input, etc.) and sends them to the system for execution through the ADB (Android Debug Bridge) interface. After receiving the instructions, the system immediately executes the operation and returns the feedback result. At the same time, it records the interface changes before and after the operation, realizing a continuous closed loop of perception-action-feedback. To ensure the stability and authenticity of the test, the interface state stability detection mechanism unit only updates the screenshots and structural data after the interface is fully rendered or the animation is finished, avoiding timing deviations. Meanwhile, the task execution environment module 1 supports parallel execution of multiple tasks and automatic switching between multiple applications, and communicates bidirectionally with the agent through a unified API interface.
[0032] The task execution environment module 1 is both the "experimental scenario" of the entire system and the core of operation for realizing dynamic evaluation, ensuring that the evaluation process is reproducible, monitorable, and interventionable, providing a solid foundation for the robustness analysis of intelligent agents.
[0033] In this embodiment, the exception triggering mechanism module 2 is mainly used in step S2. When the agent encounters an interruption or system interference during task execution, the task execution environment module 1 and the exception triggering mechanism module 2 work together seamlessly to inject or restore the exception event in real time without disrupting the integrity of the task flow. Specifically: During the task, the interface semantics and context state are continuously monitored through the exception triggering mechanism module 2. When a triggering condition matching the preset rule is detected (such as permission request, network disconnection or system warning prompt), the semantic rule engine immediately calls the corresponding template in the exception event library and dynamically injects the exception into the simulation environment. At this time, the agent needs to identify the interference, make a response decision, and perform recovery operations, such as closing the pop-up window, reconnecting or renavigating. The whole process can be regarded as a "dynamic robustness interference test" to examine the agent's adaptability and recovery ability.
[0034] The working principle of the exception triggering mechanism module 2 is based on the collaborative work of the semantic rule engine and the exception event library. The semantic rule engine is responsible for identifying the triggering time. It parses the structure file (XML level) of the Android interface in real time and extracts key semantic information, such as control text, description attributes (content-desc) and their hierarchical relationship. The system will match this information with the triggering conditions defined in the rule file. When a specific keyword is detected in the interface or a certain task context is met and the corresponding semantic threshold is satisfied, the rule engine determines that the triggering condition is met and sends an instruction to the exception event library. The triggering rules adopt a declarative definition approach, allowing researchers to flexibly set conditions such as keywords, interface context, matching thresholds, and triggering actions in the configuration file. For example, a rule can be set to trigger a "permission request" exception when the interface text contains "location permission" and the current task is in a navigation application scenario. Such a rule system not only supports precise matching but also uses semantic similarity algorithms for fuzzy recognition, thereby maintaining consistency and universality across different applications or versions.
[0035] Once a rule is triggered, the exception event library is responsible for generating the corresponding exception scenario. The library has a variety of callable exception templates, covering five major categories: system resource exceptions (such as low battery, overheating), network connection exceptions (Wi-Fi disconnection, data switching), application running exceptions (crash or lag), access control exceptions (access request pop-ups), and user experience interference (advertisements, upgrade prompts, push notifications). All exceptions can be customized or extended through parameterization to ensure that the system can flexibly adapt to different tasks and applications.
[0036] The anomaly injection process follows a two-phase structure. The first phase is the foreground interference phase, where the system overlays pop-ups, prompts, or obscuring layers on the current task interface to directly interrupt the agent's operation flow. The second phase is the subsequent state phase, where the system changes the application state based on the agent's response (e.g., clicking "Allow" or "Close"), potentially navigating to a new task page, restarting the application, or resuming the original task. This two-phase design ensures that anomaly injection goes beyond surface-level interference, reflecting subsequent state changes caused by interactive choices in a real-world environment, thereby comprehensively evaluating the agent's perception, decision-making, and recovery capabilities.
[0037] The anomaly triggering mechanism module 2 realizes the intelligent and controllable anomaly triggering process through the context awareness mechanism unit of the semantic-driven engine. This allows the evaluation system to dynamically respond to task progress and agent behavior like a real device, fully record the triggering conditions, response paths and recovery results, provide accurate data support for subsequent robustness analysis, and make the robustness testing of the agent truly close to the complex interaction logic of the real world.
[0038] In this embodiment, the status verification module 3 is mainly used in step S3. After the task is completed, the status verification module 3 will automatically extract the final interface status information and compare it with the target status in the task definition stage. The system will comprehensively determine whether the task has been successfully completed through multi-level judgment of text, structure and visual features. If the agent can still recover to the target interface and achieve the expected operation after abnormal interference, the task is judged as "robust success" and its recovery path and response behavior are recorded.
[0039] The state verification module 3 is the core of the system's judgment and feedback. It is responsible for automatically judging whether the agent has truly completed the predetermined goal after the task is completed. Unlike the traditional "operation matching verification" that relies on fixed action sequences, the state verification module 3 adopts a state-driven verification mechanism. It no longer focuses on whether the agent operates according to the predetermined path, but directly examines whether the semantic and structural state of the final interface conforms to the target definition. In this way, the system can more fairly and objectively measure the agent's true ability to complete tasks in a dynamic environment.
[0040] During the task definition phase, the system presets target state conditions for each test scenario, such as interface title, core text content, key control hierarchy, or specific image elements. After the task is completed, the system automatically extracts the complete structural information (XML hierarchy tree) and screenshot content of the current interface, and compares and analyzes them against the task target. The judgment process integrates multiple strategies such as text matching, structural hierarchy analysis, and visual feature comparison. The system first identifies target keywords or interface text, then verifies its contextual logic based on the hierarchy and spatial position of controls, and combines image feature similarity calculation when necessary to ensure that the verification results are both accurate and robust. Even if the agent adopts an operation path different from that of humans, it can still be correctly identified as "completing the task", effectively avoiding the common misjudgment problem under static sequence evaluation. At the same time, the state verification module 3 introduces fuzzy matching and semantic similarity algorithms to adapt to the diversity brought about by dynamic UI interfaces or minor language changes. For example, when the button text changes from "submit" to "confirm" after the application is updated, the system can still judge it as an equivalent target through semantic similarity, thereby maintaining the consistency of the verification results.
[0041] The state verification module 3 not only provides single judgment results, but also works in conjunction with the exception triggering module to determine the success of task recovery. When the agent can still return to the correct interface and complete the goal after encountering an interruption or exception, the system will record the recovery path and include it in the robustness evaluation index. In this way, the state verification module 3 is no longer a simple "endpoint determiner", but becomes a dynamic feedback mechanism that runs through the entire process of task execution, which can measure both the degree of completion and the recovery capability.
[0042] As can be seen, the state verification module 3 achieves automation and intelligence in evaluation through structured parsing and semantic understanding of the final interface. It shifts robustness testing from static judgment based on action matching to dynamic recognition based on interface state, ensuring that the entire evaluation system can maintain objectivity, fairness and high scalability when facing complex interactions and abnormal interference.
[0043] In this embodiment, the data acquisition module 4 is mainly used in step S4. The data acquisition module 4 continuously records multi-source data of the task throughout the entire test cycle, including screenshot sequences, interface structure, operation logs and system events. This data not only supports task backtracking and error localization, but also provides objective basis for subsequent model training and evaluation index calculation. All execution information can be reproduced and exported, ensuring the transparency and traceability of the test process.
[0044] Data acquisition module 4 runs through the entire evaluation process and serves as the information hub and analysis support layer of the system. Its role is not only to "record" but also to reproduce the task execution process through structured data, providing real and traceable evidence for robustness assessment and model optimization. At each stage of the agent's task execution, the system automatically collects key data, including screenshot sequences, UI structure files (XML-level data), operation logs, system event information, etc. Through this multi-source data, the system can completely restore the agent's operation trajectory and interface state changes, providing accurate input for anomaly detection, state verification, and indicator calculation.
[0045] The design of data acquisition module 4 follows the principles of "full process, full link, and reproducibility". The collected data is not limited to the operation of the intelligent agent, but also includes the interaction records of human operators, thereby supporting the evaluation mode of "human-machine comparison". Through the replay function, researchers can visually trace back every detail of the task execution and analyze the intelligent agent's response strategy and recovery path under specific abnormal or interference situations.
[0046] In addition, the system can automatically generate standardized task samples and abnormal scenario datasets, transforming behavioral data from the real evaluation process into high-quality sample resources that can be directly used for model training and evaluation. This function makes the evaluation system not only a verification platform, but also a data generation engine for the development of robust models. Through a unified data export interface, all collected results can be output in a structured manner for external research or algorithm performance comparison. Data acquisition module 3 ensures the traceability and scientific nature of the evaluation process, providing solid data support for the continuous iteration of subsequent algorithms and the verification of model performance.
[0047] In this embodiment, the evaluation index calculation module 5 is mainly applied to step S5. Through the evaluation index calculation module 5, the system calculates the success rate (SR) and robust success rate (RSR) of the agent under normal and abnormal environments, respectively, and performs sub-analysis according to the abnormality category to generate a complete evaluation report and visualization results. The evaluation report can be used to compare the robust performance of different models or to evaluate the robustness changes of the same model in multiple scenarios.
[0048] The evaluation index calculation module 5 is the core of the system's quantitative analysis. It is responsible for converting the execution results of the entire evaluation process into comparable and statistical performance indicators. By comparing the performance of the same task under "normal environment" and "abnormal environment", it evaluates the robustness and recovery ability of the intelligent agent under real interference.
[0049] After completing multiple rounds of task execution, the system will automatically call the verification results, count the success and failure of each task, and then calculate the success rate (SR) and the robust success rate (RSR). The former reflects the agent's basic execution capability under standard conditions, while the latter measures the proportion of tasks that can still be completed after being subjected to abnormal interference, thus directly reflecting the robustness level of the model.
[0050] The calculation of evaluation index module 5 is not a simple statistical summary, but is based on a comprehensive analysis of structured logs and state verification results. The system will perform itemized statistics and visualization based on different anomaly types (such as system resources, network connection, access control, user experience interference, etc.), enabling researchers to accurately identify the weak links of the model under different types of interference. For example, when the agent's RSR is significantly lower than the average level in the network disconnection scenario, the system can indicate that this direction is a potential optimization focus.
[0051] Finally, the evaluation index calculation module 5 will generate a complete statistical report, covering the task execution results, the distribution of anomaly recovery paths, and the trend of robustness index changes, forming an automated evaluation closed loop from data to conclusions. Through the evaluation index calculation module 5, the model robustness evaluation that originally relied on human subjective judgment is transformed into a quantitative, objective, and reproducible analysis process, so that the performance optimization of the intelligent agent has clear data basis and traceable improvement direction.
[0052] like Figure 1-2 The graphical user interface intelligent agent robustness dynamic evaluation system and method shown herein have significant improvements and innovations in technical structure, functional implementation, and evaluation dimensions compared with existing static GUI agent evaluation frameworks (such as Android Control, GUI-Robust, AndroidWorld, etc.). The main advantages are as follows: The task execution environment module 1 uses an Android emulator to build an interactive dynamic task environment, which allows the GUI agent to perform operations and receive feedback in real time. It simulates the task execution and system response process in actual applications. The evaluation environment is transformed from static to dynamic, which significantly improves the authenticity and reference value of the evaluation. It solves the problem that traditional benchmarks rely only on static screenshots or recorded sequences and cannot simulate real scenarios such as system interruptions and pop-up prompts. The semantic rule engine of the exception triggering mechanism module 2 matches information such as interface text and hierarchical structure, dynamically identifies scenarios such as "permission request", "update prompt" and "network disconnection", and automatically triggers the corresponding exception events, realizing the context-aware triggering of exceptions. This makes the evaluation process repeatable, scalable and close to the real human-computer interaction logic, solving the problem that existing exception modeling methods are mostly static overlay images and cannot be dynamically triggered for specific interface states. The anomaly triggering mechanism module 2 divides anomalies into a foreground interference stage (pop-ups, prompts) and a subsequent state stage (interface jumps, state changes). It can systematically evaluate the agent's anomaly identification, decision-making and recovery capabilities, fill the gap in the "process robustness" dimension of existing evaluations, and avoid the situation that traditional anomaly samples are mostly single pop-ups or blocked screens, and cannot test the agent's recovery behavior in subsequent operations. The task completion status is verified by comparing the UI structure tree with semantic features, replacing the traditional method that relies on a fixed action sequence for verification. This eliminates path dependence, ensures that agents with different strategies can be correctly identified when they reach the same goal, improves the objectivity and universality of the evaluation, and avoids the situation where the agent may be misjudged as failing because it takes different but correct operation paths. The "Robust Success Rate (RSR)" metric is defined to measure the model's ability to maintain performance under abnormal conditions. It can quantitatively reflect the model's reliability in real and complex environments and provide a guiding evaluation standard for model optimization. Compared with traditional evaluation metrics that are singular and only measure model performance by task success rate, which cannot distinguish differences in robustness, the above metric establishes a multi-dimensional robustness index system to quantify model robustness.
[0053] Compared to existing evaluation systems that often employ hard-coded implementations and are difficult to extend to different tasks or applications, this system, through its modular architecture design, possesses excellent scalability and reusability. All exception rules and verification logic are declared through external configuration files, supporting user-defined exception types and task sets. This facilitates cross-application scenario deployment and industry standardization, making it a unified technical platform for intelligent agent robustness research.
[0054] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A graphical user interface agent robustness dynamic evaluation system, characterized in that: The task execution environment module includes a task execution environment module, an exception triggering mechanism module, a state verification module, a data acquisition module, and an evaluation index calculation module.
2. The system of claim 1, wherein: The task execution environment module includes a built-in interface state stability detection mechanism unit and a simulation framework building unit. The exception triggering mechanism module is based on the cooperative action of a semantic rule engine and an exception event library. The state verification module uses a state-driven verification mechanism.
3. A method for dynamically evaluating the robustness of a graphical user interface agent, applied to a system for dynamically evaluating the robustness of a graphical user interface agent, characterized in that, The method includes the following steps: S1, building a simulation environment, and an agent entering the simulation environment and completing a "perception - decision - execution - feedback" cycle interaction process; S2, monitoring, automatically triggering, generating an exception scenario, and recovering an exception event; S3, verifying the state and determining whether the task is successfully completed; S4, continuously recording the multi-source data acquisition of the task during the entire running period; S5, calculating the key performance indicators according to the task execution results.
4. The method of claim 3, wherein, The step S1 specifically includes: The system first initializes the test scenario through the task execution environment module, loads the application to be tested, and sets the task target, generates a simulation environment, and after the agent enters the simulation environment, understands the current interface state through visual perception and structural information analysis and generates operation instructions, the system executes these instructions in real time, and returns the execution results to the agent, thereby forming a "perception - decision - execution - feedback" cycle interaction process.
5. The method of claim 3, wherein, The step S2 specifically includes: During the task, the exception triggering mechanism module continuously monitors the interface semantics and context state, and when a trigger condition matching a preset rule is detected, the semantic rule engine immediately calls the corresponding template in the exception event library, dynamically injects an exception into the simulation environment, and the agent needs to identify the interference, make a response decision, and execute a recovery operation.
6. The method according to claim 5, wherein: The exception dynamic injection includes a foreground interference stage and a subsequent state stage, in the foreground interference stage, the system will superimpose a pop-up window, a prompt box or a shielding layer element on the current task interface, directly interrupting the operation process of the agent, in the subsequent state stage, the system changes the application state according to the response action of the agent.
7. The method of claim 3, wherein, The step S3 specifically includes: When the task is completed, the state verification module automatically extracts the final interface state information and compares it with the target state in the task definition stage. The system determines whether the task is successfully completed through multi-layer judgment of text, structure and visual features. If the agent can still recover to the target interface and achieve the expected operation after abnormal interference, the task is determined as "robust success", and its recovery path and response behavior are recorded.
8. The method of claim 3, wherein, The step S4 specifically includes: The data acquisition module continuously records the multi-source data of the task during the entire running test period, including screenshot sequences, interface structures, operation logs and system events.
9. The method of claim 3, wherein, The step S5 specifically includes: The system uses the evaluation index calculation module to respectively calculate the success rate and robust success rate of the agent in normal and abnormal environments, and performs itemized analysis according to the abnormal categories to generate a complete evaluation report and visual results.
Citation Information
Cited By
Method, system and equipment for evaluating space capability of body-equipped agent and storage medium
CN121706991A
Body-aware agent spatial ability evaluation method, system, device and storage medium
CN121706991B
