An automated GUI testing method and system based on a multi-modal model

Abstract

The application discloses a kind of based on multi-modal model's automated GUI test method and system, comprising: (1) constructs unified test benchmark environment GTArena, GUI test procedure is decomposed into test intent generation, test task execution and GUI defect detection three continuous stages, and multi-modal model is used to execute the procedure of three stages;(2) the test task execution process is modeled using POMDP, state transition and decision process are described;(3) the interactive state of application is modeled with transition tuple, the state transition sequence and operation behavior log in the test task execution process are recorded;(4) a large-scale dataset is constructed using real application, artificially injected defect application and synthetic data, and the multi-modal large language model is evaluated and trained to optimize;(5) the multi-modal model evaluated and trained to optimize is used to carry out automated GUI test.Using the application, the overall efficiency and reliability of automated GUI testing can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of graphical user interface testing, and particularly relates to an automated GUI testing method and system based on a multi-modal model. BACKGROUND

[0002] GUI (Graphical User Interface) has become one of the most important components of interactive software, and is widely used in various application software and software tools. GUI brings great convenience to user operation, and also makes GUI software more complex and difficult to test.

[0003] With the rapid development of multi-modal large language models (MLLMs), models that integrate visual and language inputs have shown potential value in a wide range of fields. However, in the context of automated GUI testing and defect detection, the applicability, scalability, and performance of these models still have a lot of room for improvement.

[0004] Chinese patent document with publication number CN118519916A discloses a GUI test case generation method, computer program and terminal; Chinese patent document with publication number CN119088717A discloses an Android application GUI test case automatic generation method based on a large language model.

[0005] However, current researches mostly focus on single-link (such as navigation or simple operation) tasks, and fail to provide a unified framework to support end-to-end automated GUI testing processes. In addition, the lack of standardized evaluation environments and public testing benchmarks limits the performance improvement and practical application promotion of existing models.

[0006] To solve the above problems, a systematic and standardized testing environment-based evaluation and optimization framework is needed. This framework should cover the entire process from test intent generation, test task execution to GUI defect detection, and provide standardized datasets and evaluation indicators. By conducting rigorous and comprehensive experiments on multi-modal models in a unified environment, the applicability, stability, and continuous optimization potential of the models in real application scenarios can be objectively evaluated. SUMMARY

[0007] The present application provides an automated GUI testing method and system based on a multi-modal model, which can objectively evaluate the comprehensive performance of MLLM in real application environment, and optimize the weak links of the model, ultimately improving the overall efficiency and reliability of automated GUI testing.

[0008] An automated GUI testing method based on a multi-modal model, comprising the following steps:

[0009] (1) Construct a unified test benchmark environment GTArena, and decompose the automated GUI test process into three continuous stages of test intention generation, test task execution and GUI defect detection;

[0010] In the test intention generation stage, a multimodal large language model is used to generate test operation intentions to simulate the interaction path of a real user using an application; in the test task execution stage, the multimodal large language model executes the corresponding GUI operation based on the generated test intention, and records and analyzes the state transition and operation behavior log generated during the execution process; in the GUI defect detection stage, the multimodal large language model uses the recorded state transition and operation log to detect and evaluate the defects of the GUI interface;

[0011] (2) Use a partially observable Markov decision process (POMDP) to model the test task execution process, describe the state transition and decision process, and identify the application state change and corresponding operation behavior;

[0012] (3) Use transition tuples to model the interaction state of the application, record the state transition sequence and operation behavior log during the test task execution process, and lay a data foundation for defect detection and evaluation;

[0013] (4) Use real application programs, artificially injected defect application programs and synthetic data to construct a large-scale data set to evaluate and optimize the multimodal large language model;

[0014] (5) Use the evaluated and optimized multimodal model for automated GUI testing.

[0015] The method of the application realizes the standardized process of automated GUI testing by constructing a unified test benchmark environment (GTArena), organically combines test intention generation, test task execution and GUI defect detection, and provides a unified data set construction and optimization capability for the multimodal model.

[0016] In step (1), the test environment GTArena can provide uniform and repeatable experimental conditions for the evaluation and optimization of the MLLM.

[0017] In the test intention generation stage, the model output is analyzed to ensure that the generated intentions cover the core functions and main operation flows of the application, thereby ensuring the comprehensiveness and representativeness of the test. In the test task execution stage, the interface changes and response during the operation process are automatically identified to realize full-process monitoring and data collection of the test process, providing strong support for subsequent defect detection. In the GUI defect detection stage, by comparing the detection results with the standard annotation set, the accuracy, recall rate and other indicators of the model on the defect detection task can be calculated, and the defect detection performance of the model can be objectively evaluated.

[0018] In the GUI defect detection stage, the defect detection types include: display defects, including content display errors, element misplacement, and style abnormalities; interaction defects, including non-responsive operations, keyboard input abnormalities, and task results not meeting expectations.

[0019] In step (2), the definition of POMDP includes:

[0020] (2-1) Construct the state space, observation space, and action set of the application, and define the state transition function and reward function;

[0021] 1) State space: used to represent the set of GUI states of the application at a given time. Each state can be described by multi-modal features, including visual information (such as screenshots, UI element positions, sizes, colors) and structured data (such as element hierarchy, accessibility properties, text content). The construction of the state space enables the test agent to identify and perceive the running situation of the application under various complex interfaces.

[0022] 2) Observation space: In actual testing scenarios, the agent's internal information acquisition of the application is incomplete (partially observable), and can only be inferred through the current visible elements, control attributes, and action history. The observation space consists of the information that the agent can directly perceive, including interface screenshots, UI element features and attributes, and previous operation records. The agent needs to estimate the probability of the occurrence of potential defects based on this information, and then dynamically decide the subsequent actions.

[0023] 3) Action set: composed of interactive actions that can be performed on the GUI, such as Click, Scroll, Type, Enter, and Stop operations. These actions are used to simulate real user interactions to explore the application interface paths that may trigger defects.

[0024] 4) State transition function: describes the probability distribution of the possible successor states after performing a specific operation in a given state. Since the evolution of the application interface is influenced by internal logic, page jumps, and interaction responses, the state transition function needs to combine statistical characteristics or learning modeling methods to provide reasonable predictions and judgments of the next state for the agent under incomplete observation conditions.

[0025] 5) Reward function: used to give positive incentives when the test agent performs operations and completes specific goals (such as finding defects or exploring unknown interface areas). This function helps guide the agent to find potential problems faster and more comprehensively during the testing process by linking the agent's decisions to defect discovery, interface coverage, and efficiency indicators.

[0026] (2-2) Design reward mechanisms to encourage active exploration of potential defects through reward functions, maximizing the comprehensiveness and accuracy of defect detection:

[0027] 1) Defect discovery reward: When the agent successfully triggers or identifies interface display defects (such as UI element misplacement, overlapping, text format abnormalities) or interaction defects (such as button non-response, navigation logic disorder) by performing certain operations, give positive rewards. This encourages the agent to prefer similar operation sequences that can expose defects in subsequent testing.

[0028] 2) Exploration coverage reward: Reward the agent for exploring pages, functions, or control areas that have not been explored, encouraging the agent to continuously expand the exploration range, improving the comprehensiveness of defect detection, and avoiding repeated attempts on known paths.

[0029] 3) Efficiency balance and negative incentive: Give lower or even negative rewards for invalid or redundant operations to ensure that the agent maintains a certain level of testing efficiency while pursuing defect coverage. Through reasonable reward and punishment balance, the agent can maximize defect exposure based on minimizing repeated or invalid operations.

[0030] Through the above definitions and mechanisms, the testing agent can continuously optimize decision-making strategies in a partially observable environment, expand testing coverage, and efficiently discover potential defects, achieving comprehensiveness, accuracy, and efficiency in automated GUI testing.

[0031] In step (3), the interaction state of the application is modeled using transition tuples, which specifically includes:

[0032] (3-1) Define the current state of the application, the corresponding operation action, and the new state after the operation;

[0033] (3-2) Detect whether the operation triggers interface changes, record the state transition process and mark potential defects.

[0034] In step (4), the specific process of constructing a large-scale dataset includes:

[0035] (4-1) Collect defect data from real applications and filter and organize public defect records;

[0036] (4-2) Use MutAPK tools to inject artificial defects in the application to simulate common display and interaction problems;

[0037] (4-3) Generate synthetic defect scene data through image processing and interface operations to expand the size and diversity of the dataset.

[0038] In step (4), when evaluating the multi-modal large language model, the evaluation indicators include test intent coverage in the test intent generation stage, test task execution success rate in the test task execution stage, GUI defect detection accuracy and recall rate in the GUI defect detection stage.

[0039] To support continuous optimization and performance improvement of the model, a basic dataset is constructed by collecting defect data in real applications, and the scale and diversity of the dataset are expanded using artificial injection and synthetic data generation tools. Thus, rich and representative training and evaluation data are provided to promote the performance of the model in various scenarios.

[0040] The present application provides an effective way for the evaluation and optimization of multi-modal models in automated GUI testing by constructing a unified test framework and standardized dataset. Through systematic and repeatable evaluation of the entire process, researchers and developers can clearly identify the shortcomings and improvement direction of existing models, thereby continuously improving the practical application value and reliability of automated GUI testing.

[0041] An automated GUI testing system based on a multi-modal model, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the above-mentioned automated GUI testing method.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] Based on the predefined defect classification standard and detection process, the present application can analyze the interface state changes in each transition tuple to confirm the type and severity of the defect, and guide the agent to preferentially discover key issues that have a greater impact on user experience through a reward function mechanism. Through strict division and systematic execution of the three stages of test intent generation, test task execution and GUI defect detection, not only is the full-link coverage of automated GUI testing realized, but also the traceability, scalability and robustness of the testing process are ensured. At the same time, the execution and feedback record of the multi-modal model in the GTArena environment provides a new paradigm for the research and practice of automated GUI testing, which helps to quickly locate key defects in subsequent testing. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A flowchart of an embodiment of the present application, a multi-modal model-based automated GUI testing method.

[0045] Figure 2 An embodiment of the present application, a multi-modal model-based automated GUI testing method.

[0046] Figure 3For the first set of GUI defect simulation examples in synthetic data.

[0047] Figure 4 For the second set of GUI defect simulation examples in synthetic data. DETAILED DESCRIPTION

[0048] The application will be further described below in conjunction with the accompanying drawings and examples, it should be pointed out that the following examples are intended to facilitate the understanding of the application and do not have any limiting effect on it.

[0049] As Figure 1 shown, an automated GUI testing method based on a multi-modal model mainly consists of three parts: environment construction and operation process, dataset construction and expansion, model evaluation and performance analysis.

[0050] First part, environment construction and operation process.

[0051] In this embodiment, based on the GTArena test environment, the automated GUI testing process is constructed and executed to verify the utility of multi-modal models (such as GPT-4o and LLaVA) in simulating user interaction, identifying interface defects and providing test feedback. Compared with the idealized "simple instruction test" assumption, the actual automated GUI testing is more complex, which requires fine disassembly and structured design of the test object and test process. This embodiment follows a systematic workflow from test intention generation, test case execution, to defect detection and classification, ensuring the rigor and repeatability of the testing process.

[0052] First, the entire automated testing workflow is divided into three main stages:

[0053] (1) Test Intention Generation: In this stage, the task planning agent will construct test cases for the target application according to the preset test objectives and expected user interaction patterns. These use cases clearly define the interaction steps to be performed (such as clicking, scrolling, inputting text) and the expected observation points (interface element status, navigation logic, etc.), thereby providing guidance for subsequent test task execution.

[0054] (2) Test Task Execution: In the GTArena environment, the multi-modal model simulates real user operations according to the input test instructions and test predictions. This includes mobile terminal interaction behaviors such as clicking, sliding, and text input on the interface. The model records operation logs and interface screenshots in real time during the operation process, forming a complete state transition trajectory. These trajectories are composed of a series of transition tuples, which are used for subsequent defect analysis. The execution process of the test task is described by the POMDP (Partially Observable Markov Decision Process) framework, which models the decision-making process and operation strategy of the agent through state space (S), observation space (O), action set (A), state transition function (T), and reward function (R).

[0055] (3) GUI Defect Detection: After the operation is completed, the agent analyzes the recorded state transition sequence to identify and classify GUI defects. These defects include display defects (content errors, interface element misplacement or missing, color style inconsistency, etc.) and interaction defects (operation non-response, navigation logic errors, or task results not meeting expectations, etc.).

[0056] Based on the predefined defect classification criteria and detection process, the interface state changes in each transition tuple can be analyzed to confirm the type and severity of the defect, and the agent is guided to prioritize the discovery of key issues that have a greater impact on user experience through the reward function mechanism. Through the strict division and systematic execution of the above three stages, the test process constructed in this embodiment not only realizes the full-link coverage of automated GUI testing, but also ensures the traceability, scalability, and robustness of the test process. The execution and feedback record of the multi-modal model in the GTArena environment provides a new paradigm for the research and practice of automated GUI testing, which helps to quickly locate key defects in subsequent testing.

[0057] The second part is the construction and expansion of the dataset.

[0058] In this embodiment, the aim is to construct a high-quality and diverse training and evaluation dataset to comprehensively improve the performance and applicability of the automated GUI testing agent. The construction and expansion of the dataset is based on systematic mining of defect logs in real applications, deep integration of open-source datasets and code resources, and significant expansion of data coverage and defect type dimensions through artificial injection and synthesis techniques. Specifically, the invention relies on the following three types of data sources to construct and expand the dataset, as shown in Figure 2

[0059] ​(1) Real application defect data: To ensure that the dataset contains real and representative GUI defects, this embodiment collects and screens upgrade logs and problem reports related to GUI defects from open source projects (such as application repositories in GitHub). In this process, the invention focuses on interface problems explicitly recorded by developers or users, including interface element misplacement, layout abnormalities, interaction failures, and other types. By filtering and verifying version change records and problem tickets, a series of real and reproducible defect cases can be obtained. These defects generated in actual environment help to evaluate the performance of the test agent in handling complex and unpredictable real defects.

[0060] (2) Controllable injected defect data: Given the uncontrollability and imbalance of real application defects, this embodiment realizes the ordered expansion of the dataset by injecting specific types of GUI defects at the application source code level (for example, using MutAPK tools). By intentionally introducing reproducible defects (such as button unclickable, navigation logic abnormal, text display format error, etc.) at different interface elements and process logic, the invention not only ensures the balance and diversity of the dataset in terms of defect distribution and complexity, but also accurately evaluates the detection and diagnosis capabilities of the agent. The introduction of such "artificial controllable defects" provides a basis for experimental control for comparing and verifying different testing strategies and model parameters.

[0061] (3) Synthetic defect dataset: For some mature commercial applications that have been rigorously tested for multiple rounds and have few exposed defects, image processing and visual transformation techniques are used to simulate defects in stable interface screenshots. For example, by synthesizing element overlap, text disorder, response delay, and other visual or interactive abnormalities in screenshots, a "simulated" defect dataset is constructed, as shown in Figures 3-4 This synthetic strategy enables the invention to quickly generate multiple types of defect scenarios without being limited by the actual development and maintenance cycle of the application, ensuring that the test agent remains robust and generalizable when faced with highly complex, rare, or even potential problems that have not yet appeared in actual scenarios.

[0062] By combining the above three types of data sources, a hierarchical and multi-type dataset is constructed, enabling the automated GUI test agent to handle both naturally occurring real defects and performance verification and comparison in a strictly controlled experimental environment, and to improve its adaptability to potential new defects through synthetic data. The establishment of this dataset provides sufficient experimental basis for subsequent model optimization, algorithm improvement, and test strategy iteration, ultimately facilitating the practical deployment and iterative optimization of automated GUI testing in real application environments.

[0063] Third part, model evaluation and performance analysis.

[0064] In this embodiment, a variety of different multi-modal large models (such as GPT-4o, Claude, LLaVA, and Qwen2-VL) are deployed into the GTArena test environment to systematically and standardize the evaluation of their overall performance in automated GUI testing tasks. This evaluation not only focuses on the performance of the models in test intent generation and test task execution, but also emphasizes their accuracy and reliability in GUI defect detection tasks, thereby providing a reference basis for subsequent model optimization and improvement strategies.

[0065] (1) Evaluation indicators and experimental design: The evaluation follows the aforementioned automated GUI testing framework and workflow (see embodiments 1 and 2), including three core links of test case generation, test task execution, and GUI defect detection. The specific evaluation indicators are as follows:

[0066] a. Test intent coverage rate:

[0067] The model needs to generate a set of test intents based on the target application and background information. By matching and comparing with the standard test intent set labeled by humans, this embodiment calculates the proportion of correctly aligned standard test intents to measure the completeness and coverage of the model in the test intent generation layer.

[0068] b. Test task execution success rate and related indicators:

[0069] Under the premise of determining the test intent, the model needs to execute specific operations (such as clicking, inputting, scrolling, etc.) according to the corresponding test cases composed of state transition tuples. To evaluate the execution quality, this embodiment introduces the following indicators: TM (Type Match) to determine whether the model's predicted operation type matches the actual expected type; EM (Exact Match) to further evaluate whether the operation target element and parameters meet the requirements accurately based on the matching of operation types; and SR (Success Rate) to represent the proportion of models that can complete all operation steps completely and accurately in a number of test tasks.

[0070] c. GUI defect detection accuracy and recall rate:

[0071] For the GUI defect detection link, this embodiment includes real defect data and normal data (no defects) in the test set. The model needs to analyze the state transition path and interface changes to determine whether there is a GUI defect. The evaluation indicators include: the overall correct determination rate of the model on all data (including defect and normal data), the correct determination rate of the model on defect data (reflecting its ability to identify actual defects), and the correct determination rate of the model on non-defect data (to measure its ability to distinguish normal situations).

[0072] (2) Experimental subjects and baseline models: To establish a baseline and control, this embodiment selects multi-modal large models with different architectures and capabilities for comparison, including GPT-4o, Claude (a commercial closed-source model), and LLaVA, Qwen2-VL (open-source models). By evaluating these models under a unified framework and dataset, the differences and shortcomings of closed-source and open-source models in test intent generation, test task execution, and defect detection capabilities can be analyzed in depth.

[0073] (3) Experimental results and analysis: The research of this embodiment shows that testing multi-modal large models under unified standards can effectively quantify their differences in capabilities in automated GUI testing, and also provides important references for subsequent improvements in model performance, optimization of testing strategies, and design of more targeted training datasets. By introducing precise and repeatable evaluation processes and data indicators, the invention provides scientific basis for the selection, training, and iteration of automated GUI testing agents. The experimental results and model performance of the method of this embodiment are shown in Table 1 below. As can be seen, different multi-modal large models have their own advantages and disadvantages in automated GUI testing. In terms of test intent coverage, GPT-4o and Claude perform outstandingly, showing good ability to expand test scope, explore new paths, and new interface areas, while LLaVA and Qwen2-VL are relatively limited in this aspect. In terms of task success rate, Claude is slightly better than GPT-4o, and has a slight advantage in reliability in executing instructions to complete predetermined test tasks. The success rates of LLaVA and Qwen2-VL are lower, indicating that their ability to analyze and interact with complex GUI interfaces still needs to be improved. In terms of defect detection accuracy, GPT-4o and Qwen2-VL are more advantageous in identifying and locating GUI defects, showing higher identification accuracy and abnormal capture ability, while Claude performs relatively weakly in this aspect, indicating that it still has significant shortcomings in interface anomaly identification and logical judgment.

[0074] Table 1

[0075]

[0076] The above-described embodiments have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above-described embodiments are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements, and equivalent replacements made within the principle range of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for automated GUI testing based on a multi-modal model, the method comprising: The method comprises the following steps: (1) constructing a unified test benchmark environment GTArena, and decomposing an automated GUI test process into three continuous stages of test intention generation, test task execution, and GUI defect detection; In the test intention generation stage, a multimodal large language model is used to generate a test operation intention to simulate the interaction path of a real user using an application program; in the test task execution stage, the multimodal large language model executes the corresponding GUI operation based on the generated test intention, and records and analyzes the state transition and operation behavior log generated in the execution process; in the GUI defect detection stage, the multimodal large language model detects and evaluates the defects of the GUI interface by using the recorded state transition and operation log; (2) modeling the test task execution process by using a partially observable Markov decision process (POMDP), describing the state transition and decision process, so as to identify the state change of the application program and the corresponding operation behavior; (3) modeling the interaction state of the application program by using a transition tuple, recording the state transition sequence and operation behavior log in the test task execution process, and laying a data foundation for defect detection and evaluation; (4) constructing a large-scale data set by using real application programs, artificially injected defect application programs, and synthetic data to evaluate and optimize the multimodal large language model; (5) using the multimodal model evaluated and optimized to perform automated GUI testing.

2. The automated GUI testing method based on a multi-modal model according to claim 1, wherein, In step (1), in the GUI defect detection stage, the defect detection types include: display defects, including content display errors, element misplacement, and style abnormalities; interaction defects, including non-responsive operations, keyboard input abnormalities, and task results not meeting expectations.

3. The automated GUI testing method based on a multi-modal model according to claim 1, wherein, In step (2), the definition of POMDP includes: (2-1) constructing the state space, observation space, and operation set of the application program, and defining the state transition function and reward function; (2-2) designing a reward mechanism to encourage active exploration of potential defects through the reward function, thereby maximizing the comprehensiveness and accuracy of defect detection.

4. The automated GUI testing method based on a multi-modal model according to claim 3, wherein, The state space includes visual information and structured data; The observation space includes the current screenshot, visual control attributes, and operation history; the operation set includes clicking, scrolling, inputting, entering, and terminating operations.

5. The automated GUI testing method based on a multi-modal model according to claim 1, wherein, In step (3), the interaction state of the application program is modeled by using a transition tuple, specifically including: (3-1) defining the current state of the application program, the corresponding operation action, and the new state after the operation; (3-2) detecting whether the operation triggers an interface change, recording the state transition process and marking potential defects.

6. The automated GUI testing method based on a multi-modal model according to claim 1, wherein, In step (4), the specific process of constructing a large-scale data set includes: (4-1) collecting defect data in real application programs, and screening and organizing public defect records; (4-2) using the MutAPK tool to inject artificial defects in the application program to simulate common display and interaction problems; (4-3) generating synthetic defect scene data through image processing and interface operation to expand the size and diversity of the data set.

7. The automated GUI testing method based on a multi-modal model according to claim 1, wherein, In step (4), when the multi-modal large language model is evaluated, the evaluation indexes include test intent coverage in the test intent generation stage, test task execution success rate in the test task execution stage, GUI defect detection accuracy and recall rate in the GUI defect detection stage.

8. An automated GUI testing system based on a multi-modal model, characterized by, The device comprises a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the automatic GUI test method in any one of claims 1-7.

Images