Automatic GUI (Graphical User Interface) testing method and system based on multi-modal model

Abstract

The invention discloses an automatic GUI test method and system based on a multi-modal model, and the method comprises the steps: (1) constructing a unified test benchmark environment GTArena, decomposing a GUI test process into three continuous stages of test intention generation, test task execution and GUI defect detection, and executing the processes of the three stages by adopting the multi-modal model; (2) modeling a test task execution process by using POMDP, and describing a state transition and decision process; (3) modeling an interaction state of the application program by means of the transfer tuple, and recording a state transfer sequence and an operation behavior log in a test task execution process; (4) constructing a large-scale data set by using a real application program, an application program of manual defect injection and synthetic data, and performing evaluation and training optimization on the multi-modal large language model; and (5) carrying out automatic GUI testing by utilizing the multi-modal model subjected to evaluation and training optimization. By means of the method and device, the overall efficiency and reliability of automatic GUI testing can be effectively improved.

Description

Technical Field

[0001] The invention belongs to the field of graphical user interface testing, and in particular relates to an automated GUI testing method and system based on a multimodal model. Background Art

[0002] GUI (Graphical User Interface) has become one of the most important components of interactive software and is widely used in various application software and software tools. GUI brings great convenience to user operation, but also makes GUI software more complex and more difficult to test.

[0003] With the rapid development of multimodal large language models (MLLMs), models that integrate visual and language inputs have shown potential value in a wide range of fields. However, in the scenarios of automated GUI testing and defect detection, the applicability, scalability and performance of these models still have much room for improvement.

[0004] For example, Chinese patent document with publication number CN118519916A discloses a GUI test case generation method, computer program and terminal; Chinese patent document with publication number CN119088717A discloses an automatic generation method of Android application GUI test cases based on a large language model.

[0005] However, most current research focuses on tasks in a single link (such as navigation or simple operations) and fails to provide a unified framework to support the end-to-end automated GUI testing process. In addition, the lack of standardized evaluation environments and public test benchmarks limits the performance improvement and practical application promotion of existing models.

[0006] To solve the above problems, a systematic evaluation and optimization framework based on a standardized test environment is needed. This framework should cover the entire process from test intent generation, test task execution to GUI defect detection, and provide standardized data sets and evaluation indicators. By conducting rigorous and comprehensive experiments on multimodal models in a unified environment, its applicability, stability and continuous optimization potential in real application scenarios can be objectively evaluated. Summary of the invention

[0007] The present invention provides an automated GUI testing method and system based on a multimodal model, which can objectively evaluate the comprehensive performance of MLLM in a real application environment, and optimize the weak links of the model in a targeted manner, ultimately improving the overall efficiency and reliability of automated GUI testing.

[0008] An automated GUI testing method based on a multimodal model comprises the following steps:

[0009] (1) Build a unified test benchmark environment GTArena, which decomposes the automated GUI testing process into three consecutive stages: test intent generation, test task execution, and GUI defect detection;

[0010] Among them, in the test intent generation stage, the multimodal large language model is used to generate test operation intentions to simulate the interaction path of real users using the application; in the test task execution stage, the multimodal large language model performs corresponding GUI operations based on the generated test intent, and records and analyzes the state transition and operation behavior logs generated during the execution process; in the GUI defect detection stage, the multimodal large language model uses the recorded state transition and operation logs to detect and evaluate defects in the GUI interface;

[0011] (2) The partially observable Markov decision process (POMDP) ​​is used to model the test task execution process, describe the state transition and decision-making process, and thus identify the application state changes and corresponding operation behaviors;

[0012] (3) Model the interactive state of the application with the help of transition tuples, record the state transition sequence and operation behavior log during the execution of the test task, and lay a data foundation for defect detection and evaluation;

[0013] (4) Use real applications, artificially defective applications, and synthetic data to build large-scale datasets to evaluate and optimize the training of large multimodal language models;

[0014] (5) Utilize the evaluated and trained optimized multimodal model for automated GUI testing.

[0015] The method of the present invention realizes the standardized process of automated GUI testing by constructing a unified test benchmark environment (GTArena), organically combines test intent generation, test task execution and GUI defect detection, and provides a unified data set construction and optimization capability for multimodal models.

[0016] In step (1), the test environment GTArena can provide unified and repeatable experimental conditions for the evaluation and optimization of MLLM.

[0017] In the test intent generation stage, by analyzing the model output, we ensure that the generated intent covers the core functions and main operation flows of the application, thereby ensuring the comprehensiveness and representativeness of the test. In the test task execution stage, by automatically identifying the interface changes and responses during the operation, we can achieve full monitoring and data collection of the test process, providing strong support for subsequent defect detection. In the GUI defect detection stage, by comparing the test results with the standard annotation set, we can calculate the accuracy, recall rate and other indicators of the model in the defect detection task, and then objectively evaluate its defect detection performance.

[0018] In the GUI defect detection stage, the defect detection types include: display defects, including content display errors, element misalignment and style anomalies; interaction defects, including unresponsive operations, abnormal keyboard input and task results that do not meet expectations.

[0019] In step (2), the definition of POMDP includes:

[0020] (2-1) Construct the state space, observation space and operation set of the application, and define the state transition function and reward function;

[0021] 1) State space: used to represent the set of GUI states of an application at a given moment. Each state can be described by multimodal features, including visual information (such as screenshots, UI element position, size, color) and structured data (such as element hierarchy, accessibility attributes, text content). The construction of state space enables the test agent to identify and perceive the operation of the application under various complex interfaces.

[0022] 2) Observation space: In actual test scenarios, the agent's acquisition of internal information about the application is incomplete (partially observable), and can only be inferred through the current visible elements of the interface, control properties, and action history. The observation space consists of some information that the agent can directly perceive, including interface screenshots, features and properties of UI elements, previous operation records, etc. The agent needs to estimate the probability of potential defects based on this, and dynamically decide on subsequent actions.

[0023] 3) Operation set: It consists of interactive actions that can be performed on the GUI, such as click, scroll, type, enter, and stop. These actions are used to simulate real user interactions to explore application interface paths that may cause defects.

[0024] 4) State transition function: describes the probability distribution of the subsequent states that may be reached after performing a specific operation in a given state. Since the evolution of the application interface is affected by internal logic, page jumps and interactive responses, the state transition function needs to be combined with statistical characteristics or learning modeling methods to provide the agent with reasonable predictions and judgments of the next state under incomplete observation conditions.

[0025] 5) Reward function: used to give positive incentives when the test agent performs actions and completes specific goals (such as finding defects or exploring unknown interface areas). This function helps guide the agent to find potential problems faster and more comprehensively during the testing process by linking the agent's decision-making with defect discovery, interface coverage, and efficiency indicators.

[0026] (2-2) Design a reward mechanism to encourage active exploration of potential defects through reward functions, thereby maximizing the comprehensiveness and accuracy of defect detection:

[0027] 1) Defect discovery rewards: When an agent successfully triggers or identifies an interface display defect (such as misaligned or overlapping UI elements, abnormal text format) or an interaction defect (such as unresponsive buttons, disordered navigation logic) by performing a certain operation, a positive reward will be given. This will encourage the agent to choose similar operation sequences that can expose defects in subsequent tests.

[0028] 2) Exploration coverage reward: Rewards are given for pages, functions, or control areas that the agent has not yet explored, encouraging the agent to continuously expand the scope of exploration, improve the comprehensiveness of defect detection, and avoid repeated attempts on known paths.

[0029] 3) Balance between efficiency and negative incentives: Give low or even negative rewards to invalid or redundant operations to ensure that the agent maintains a certain test efficiency while pursuing defect coverage. Through a reasonable balance of rewards and punishments, the agent can achieve maximum defect exposure while minimizing repeated or invalid operations.

[0030] Through the above definitions and mechanisms, the test agent can continuously optimize the decision-making strategy in a partially observable environment, expand the test coverage and efficiently discover potential defects, thereby achieving comprehensiveness, accuracy and efficiency of automated GUI testing.

[0031] In step (3), the interactive state of the application is modeled with the help of the transition tuple, which specifically includes:

[0032] (3-1) Define the current state of the application, the corresponding operation, and the new state after the operation;

[0033] (3-2) Detect whether the operation causes interface changes, record the state transition process and mark potential defects.

[0034] In step (4), the specific process of constructing a large-scale dataset includes:

[0035] (4-1) Collect defect data from real applications and screen and organize public defect records;

[0036] (4-2) Use the MutAPK tool to inject artificial defects into the application to simulate common display and interaction problems;

[0037] (4-3) Generate synthetic defect scene data through image processing and interface operations to expand the scale and diversity of the data set.

[0038] In step (4), when evaluating the multimodal large language model, the evaluation indicators include the test intent coverage in the test intent generation phase, the test task execution success rate in the test task execution phase, and the GUI defect detection accuracy and recall rate in the GUI defect detection phase.

[0039] To support the continuous optimization and performance improvement of the model, we build a basic data set by collecting defect data from real applications, and use artificial injection and synthetic data generation tools to expand the scale and diversity of the data set. This provides rich and representative training and evaluation data, and promotes the performance improvement of the model in various scenarios.

[0040] This paper provides an effective way to evaluate and optimize multimodal models in automated GUI testing by building a unified test framework and standardized data sets. By systematically and repeatably evaluating the entire process, researchers and developers can clearly identify the shortcomings of existing models and directions for improvement, thereby continuously improving the practical application value and reliability of automated GUI testing.

[0041] An automated GUI testing system based on a multimodal model comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, the automated GUI testing method is implemented.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Based on predefined defect classification standards and detection processes, the present invention can analyze the changes in interface states in each transfer tuple to confirm the type and severity of the defect, and guide the agent to prioritize the key issues that have a greater impact on user experience through a reward function mechanism. Through the strict division and systematic execution of the three stages of test intent generation, test task execution, and GUI defect detection, not only is full-link coverage of automated GUI testing achieved, but also the traceability, scalability, and robustness of the testing process are ensured. At the same time, the execution and feedback records of the multimodal model in the GTArena environment provide a new paradigm for the research and practice of automated GUI testing, which helps to quickly locate key defects in subsequent tests. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention provides an overall flow chart of an automated GUI testing method based on a multimodal model.

[0045] Figure 2 The diagram is a schematic diagram of data set construction and expansion in an embodiment of the present invention.

[0046] Figure 3The first set of GUI defect simulation examples in synthetic data.

[0047] Figure 4 Examples of the second set of GUI defect simulations in the synthetic data.

[0048] Figure 5 Examples of the third set of GUI defect simulations in the synthetic data. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0050] like Figure 1 As shown in the figure, an automated GUI testing method based on a multimodal model is mainly composed of three parts: environment construction and operation process, data set construction and expansion, and model evaluation and performance analysis.

[0051] Part 1: Environment construction and operation process.

[0052] In this embodiment, an automated GUI testing process is built and executed based on the GTArena test environment to verify the effectiveness of multimodal models (such as GPT-4o and LLaVA) in simulating user interactions, identifying interface defects, and providing test feedback. Compared with the idealized "simple instructions are tests" concept, actual automated GUI testing is more complex and requires detailed decomposition and structured design of test objects and test processes. This embodiment follows a systematic workflow from test intent generation, test case execution, to defect detection and classification to ensure the rigor and repeatability of the testing process.

[0053] First, break down the entire automated testing workflow into three main phases:

[0054] (1) Test Intention Generation: In this phase, the task planning agent constructs test cases for the target application based on the preset test objectives and expected user interaction patterns. These test cases specify the interaction steps to be performed (such as clicking, scrolling, and entering text) and the expected observation points (interface element status, navigation logic, etc.), thereby providing guidance for the subsequent test task execution.

[0055] (2) Test Task Execution: In the GTArena environment, the multimodal model simulates real user operations based on the input test instructions and test predictions. This includes mobile terminal interactive behaviors such as clicking, sliding, and text input on the interface. During the execution of the operation, the model will record the operation log and interface screenshots in real time to form a complete state transition trajectory. These trajectories are composed of a series of transition tuples and are used for subsequent defect analysis. The execution process of the test task is characterized by the POMDP (Partially Observable Markov Decision Process) framework, and the agent's decision process and operation strategy are modeled through the state space (S), observation space (O), action set (A), state transition function (T) and reward function (R).

[0056] (3) GUI Defect Detection: After the operation is completed, the agent will analyze the recorded state transition sequence to identify and classify GUI defects. These defects include display defects (content errors, misplaced or missing interface elements, inconsistent color styles, etc.) and interaction defects (unresponsive operations, navigation logic errors, or task results that do not meet expectations, etc.).

[0057] Based on the predefined defect classification standards and detection processes, the interface state changes in each transfer tuple can be analyzed to confirm the type and severity of the defect, and the reward function mechanism can be used to guide the agent to prioritize the key issues that have a greater impact on the user experience. Through the strict division and systematic execution of the above three stages, the test process constructed in this embodiment not only achieves full-link coverage of automated GUI testing, but also ensures the traceability, scalability and robustness of the testing process. The execution and feedback records of the multimodal model in the GTArena environment provide a new paradigm for the research and practice of automated GUI testing, which helps to quickly locate key defects in subsequent tests.

[0058] Part 2, dataset construction and expansion.

[0059] In this embodiment, the purpose is to build a high-quality and diverse training and evaluation dataset to comprehensively improve the performance and applicability of the automated GUI test agent. The construction and expansion of the dataset is based on systematic mining of defect logs in real applications, deep integration of open source datasets and code resources, and significant expansion of data coverage and defect type dimensions through artificial injection and synthesis technology. Specifically, the present invention relies on the following three types of data sources to build and expand the dataset. The data sources and construction are as follows: Figure 2 As shown:

[0060] (1) Real application defect data: To ensure that the data set contains real and representative GUI defects, this embodiment systematically collects and filters upgrade logs and problem reports related to GUI defects from open source projects (such as application repositories in GitHub). In this process, the present invention focuses on interface problems that are explicitly recorded by developers or users, including various types such as misaligned interface elements, abnormal layouts, and interaction failures. By filtering and verifying version change records and problem tickets, a series of real and reproducible defect cases can be obtained. These defects generated in actual environments help evaluate the performance of the test agent in dealing with complex, changeable, and unpredictable real defects.

[0061] (2) Data with controllable injected defects: In view of the uncontrollability and imbalance of real application defects, this embodiment achieves orderly expansion of the data set by injecting specific types of GUI defects at the application source code level (for example, using the MutAPK tool). By deliberately introducing reproducible defects in different interface elements and process logic (such as unclickable buttons, abnormal navigation logic, incorrect text display format, etc.), the present invention can not only ensure the balance and diversity of the data set in terms of defect distribution and complexity, but also accurately evaluate the detection and diagnosis capabilities of the agent in a targeted manner. The introduction of this type of "artificially controllable defects" provides an experimental control basis for comparing and verifying different test strategies and model parameters.

[0062] (3) Synthetic defect dataset: For some mature commercial applications that have undergone multiple rounds of rigorous testing and have rarely exposed defects, we use image processing and visual transformation technology to simulate defects on stable interface screenshots. For example, by synthesizing visual or interactive anomalies such as overlapping elements, garbled text, and response delays in screenshots, we can construct a "simulated" defect dataset. The defect simulation effect is as follows: Figure 3-5 This synthesis strategy enables the present invention to quickly generate various types of defect scenarios without being limited by the actual development and maintenance cycle of the application, ensuring that the test agent still has robustness and generalization capabilities when facing highly complex, rare or even potential problems that have not yet appeared in actual scenarios.

[0063] Combining the above three types of data sources, we can build a hierarchical and diverse data set, which enables the automated GUI test agent to not only deal with naturally occurring real defects during training and evaluation, but also perform performance verification and comparison in a strictly controllable experimental environment, and improve its adaptability to potential new defects through synthetic data. The establishment of this data set provides sufficient experimental basis for subsequent model optimization, algorithm improvement, and test strategy iteration, and ultimately facilitates the actual deployment and iterative optimization of automated GUI testing in real application environments.

[0064] Part III, model evaluation and performance analysis.

[0065] In this embodiment, a variety of different multimodal large models (such as GPT-4o, Claude, LLaVA and Qwen2-VL) are deployed in the GTArena test environment, and their overall performance in automated GUI testing tasks is evaluated in a systematic and standardized process. This evaluation not only focuses on the performance of the model in test intent generation and test task execution, but also pays more attention to its accuracy and reliability in GUI defect detection tasks, thereby providing a reference for subsequent model optimization and improvement strategies.

[0066] (1) Evaluation indicators and experimental design: The evaluation follows the aforementioned automated GUI testing framework and workflow (see Example 1 and Example 2 for details), including three core links: test case generation, test task execution, and GUI defect detection. The specific evaluation indicators are as follows:

[0067] a. Test intent coverage:

[0068] The model needs to generate a set of test intents based on the target application and background information. By matching and comparing with the manually annotated standard test intent set, this embodiment calculates the proportion of the model correctly aligned with the standard test intent, thereby measuring its completeness and coverage at the test intent generation level.

[0069] b. Test task execution success rate and related indicators:

[0070] Under the premise that the test intent has been determined, the model needs to perform specific operations (such as clicking, inputting, scrolling, etc.) according to the corresponding test cases composed of state transition tuples. In order to evaluate the execution quality, this embodiment introduces the following indicators: TM (Type Match) is used to determine whether the operation type predicted by the model is consistent with the actual expected type; EM (Exact Match) further evaluates whether the operation target elements and parameters accurately meet the requirements based on the operation type matching; SR (Success Rate) indicates the proportion of the model that can complete all operation steps completely and accurately in several test tasks.

[0071] c. GUI defect detection accuracy and recall rate:

[0072] For the GUI defect detection link, this embodiment includes real defect data and normal data (no defects) in the test set. The model needs to determine whether there is a GUI defect after analyzing the state transition path and interface changes. The evaluation indicators include: the overall judgment accuracy of the model on all data (including defects and normal data), the correct judgment ratio of the model on defective data (reflecting its ability to identify actual defects), and the correct judgment ratio of the model on non-defective data (used to measure its ability to distinguish normal situations).

[0073] (2) Experimental objects and baseline models: To establish a baseline and comparison, this embodiment selects multimodal large models with different architectures and capabilities for comparison, including GPT-4o, Claude (commercial closed-source model) and LLaVA, Qwen2-VL (open-source model). By evaluating these models under a unified framework and data set, we can deeply analyze the differences and deficiencies between closed-source and open-source models in terms of test intent generation, test task execution, and defect detection capabilities.

[0074] (3) Experimental results and analysis: The research of this embodiment shows that testing multimodal large models under unified standards can effectively quantify their ability differences in automated GUI testing, and also provides an important reference for subsequent improvement of model performance, optimization of test strategies and design of more targeted training data sets. By introducing accurate and repeatable evaluation processes and data indicators, the present invention provides a scientific basis for the selection, training and iteration of automated GUI test agents. The experimental results and model performance comparison of the method of this embodiment are shown in Table 1 below. It can be seen that different multimodal large models have their own advantages and disadvantages in automated GUI testing. In terms of test intent coverage, GPT-4o and Claude performed outstandingly, showing good capabilities for expanding the test scope, exploring new paths and new interface areas, while LLaVA and Qwen2-VL are relatively limited in this regard. In terms of task success rate, Claude is slightly better than GPT-4o, and is slightly better in the reliability of executing instructions to complete the predetermined test tasks. The success rates of LLaVA and Qwen2-VL are relatively low, showing that the ability to parse complex GUI interfaces and plan interactive strategies still needs to be improved. In terms of defect detection accuracy, GPT-4o and Qwen2-VL have more advantages in identifying and locating GUI defects, showing higher recognition accuracy and anomaly capture capabilities, while Claude's performance in this regard is relatively weak, indicating that it still has obvious deficiencies in interface anomaly recognition and logical judgment.

[0075] Table 1

[0076]

[0077] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An automated GUI testing method based on a multimodal model, characterized in that: The following steps are involved: (1) Build a unified test benchmark environment GTArena, which decomposes the automated GUI testing process into three consecutive stages: test intent generation, test task execution, and GUI defect detection; Among them, in the test intent generation stage, the multimodal large language model is used to generate test operation intentions to simulate the interaction path of real users using the application; in the test task execution stage, the multimodal large language model performs corresponding GUI operations based on the generated test intent, and records and analyzes the state transition and operation behavior logs generated during the execution process; in the GUI defect detection stage, the multimodal large language model uses the recorded state transition and operation logs to detect and evaluate defects in the GUI interface; (2) The partially observable Markov decision process (POMDP) ​​is used to model the test task execution process, describe the state transition and decision-making process, and thus identify the application state changes and corresponding operation behaviors; (3) Model the interactive state of the application with the help of transition tuples, record the state transition sequence and operation behavior log during the execution of the test task, and lay a data foundation for defect detection and evaluation; (4) Use real applications, artificially defective applications, and synthetic data to build large-scale datasets to evaluate and optimize the training of large multimodal language models; (5) Utilize the evaluated and trained optimized multimodal model for automated GUI testing.

2. The automated GUI testing method based on a multimodal model according to claim 1, characterized in that: In step (1), in the GUI defect detection stage, the defect detection types include: Display defects include incorrect content display, element misalignment, and abnormal style; interaction defects include unresponsive operations, abnormal keyboard input, and task results that do not meet expectations.

3. The automated GUI testing method based on a multimodal model according to claim 1, characterized in that: In step (2), the definition of POMDP includes: (2-1) Construct the state space, observation space and operation set of the application, and define the state transition function and reward function; (2-2) Design a reward mechanism to encourage active exploration of potential defects through reward functions, thereby maximizing the comprehensiveness and accuracy of defect detection.

4. The multimodal model-based automated GUI testing method according to claim 3, characterized in that: The state space includes visual information and structured data; The observation space includes the current screenshot, visual control properties and operation history; the operation set includes click, scroll, input, enter and terminate operations.

5. The multimodal model-based automated GUI testing method according to claim 1, characterized in that: In step (3), the interactive state of the application is modeled with the help of the transition tuple, which specifically includes: (3-1) Define the current state of the application, the corresponding operation, and the new state after the operation; (3-2) Detect whether the operation causes interface changes, record the state transition process and mark potential defects.

6. The multimodal model-based automated GUI testing method according to claim 1, characterized in that: In step (4), the specific process of constructing a large-scale dataset includes: (4-1) Collect defect data from real applications and screen and organize public defect records; (4-2) Use the MutAPK tool to inject artificial defects into the application to simulate common display and interaction problems; (4-3) Generate synthetic defect scene data through image processing and interface operations to expand the scale and diversity of the data set.

7. The multimodal model-based automated GUI testing method according to claim 1, characterized in that: In step (4), when evaluating the multimodal large language model, the evaluation indicators include the test intent coverage in the test intent generation phase, the test task execution success rate in the test task execution phase, and the GUI defect detection accuracy and recall rate in the GUI defect detection phase.

8. An automated GUI testing system based on a multimodal model, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the automated GUI testing method described in any one of claims 1 to 7.

Images