User interface testing method and device

Abstract

The embodiment of the invention provides a user interface testing method and device, and the method comprises the steps: obtaining a to-be-tested task, and obtaining a first current user interface corresponding to the to-be-tested task; identifying a current interface element in the first current user interface image; obtaining a first current executable operation through a first strategy model according to the to-be-tested task, the first current user interface image and the current interface element; and executing the first current executable operation, obtaining the next executable operation through reasoning of the first strategy model according to the execution context of the first current executable operation, and completing the to-be-tested task until a test stop condition is met. When the first current executable operation is executed, an execution path corresponding to the to-be-tested task is obtained in real time, a structured sample is generated according to the execution path obtained in real time and the to-be-tested task, and the execution path comprises an executable operation sequence and state change corresponding to the to-be-tested task, a user interface sequence and an interface element sequence.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a user interface testing method. This application also relates to a user interface testing apparatus, a computing device, and a computer-readable storage medium. Background Technology

[0002] User Interface Testing (UIT) is a software testing methodology designed to verify the correctness, usability, and consistency of a user interface to ensure the quality of the user experience. In the game development field, with the rapid increase in mobile game genres and frequent game version iterations, the requirements for user interface testing techniques are becoming increasingly stringent.

[0003] Typically, image matching methods are used for user interface testing. Specifically, by comparing a screenshot of the user interface with a pre-stored image template, specific user interface elements can be located, and corresponding actions, such as clicking or swiping, can be performed. However, because image matching methods rely on image templates, they are prone to failure when game versions iterate or elements in the game's user interface change—for example, button positions shift, icons are replaced, or elements are obscured. This can prevent user interface testing from being performed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a user interface testing method. One or more embodiments of this specification also relate to a user interface testing apparatus, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a user interface testing method is provided, comprising: Obtain the task to be tested and obtain the first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language description text; Identify the current interface element in the first current user interface image; Based on the task to be tested, the first current user interface image, and the current interface elements, a first currently executable operation is obtained through reasoning using a first strategy model, wherein the first strategy model is a UI multimodal model; Execute the first currently executable operation, and obtain the next executable operation through reasoning based on the execution context of the first currently executable operation using the first strategy model, until the test stop condition is met, and complete the task to be tested; Specifically, when executing the first currently executable operation, the execution path corresponding to the task to be tested is obtained in real time, and a structured sample is generated based on the real-time obtained execution path and the task to be tested. The execution path includes the executable operation sequence and state changes, user interface sequence and interface element sequence corresponding to the task to be tested.

[0006] According to a second aspect of the embodiments of this specification, a user interface testing apparatus is provided, comprising: The interface awareness module is configured to: acquire a task to be tested and acquire a first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language description text; and identify current interface elements in the first current user interface image. The hierarchical reasoning module is configured to obtain a first currently executable operation by reasoning through a first strategy model based on the task to be tested, the first current user interface image, and the current interface elements, wherein the first strategy model is a UI multimodal model. The operation execution module is configured to execute the first currently executable operation, and obtain the next executable operation through reasoning using the first strategy model based on the execution context of the first currently executable operation, until the test stop condition is met, thereby completing the task to be tested; wherein, when executing the first currently executable operation, the execution path corresponding to the task to be tested is obtained in real time, and a structured sample is generated based on the real-time obtained execution path and the task to be tested, wherein the execution path includes the sequence of executable operations and state changes, the user interface sequence, and the sequence of interface elements corresponding to the task to be tested.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0009] The user interface testing method provided in this specification can acquire a task to be tested and acquire a first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language descriptive text; identify current interface elements in the first current user interface image; based on the task to be tested, the first current user interface image, and the current interface elements, obtain a first currently executable operation through reasoning using a first strategy model, wherein the first strategy model is a UI multimodal model; execute the first currently executable operation, and obtain the next executable operation based on the execution context of the first currently executable operation through the first strategy model, until the test stop condition is reached, thus completing the task to be tested; wherein, when executing the first currently executable operation, the execution path corresponding to the task to be tested is acquired in real time, the execution path including the sequence of executable operations and state changes, the user interface sequence, and the interface element sequence corresponding to the task to be tested, and a structured sample is generated based on the real-time acquired execution path and the task to be tested.

[0010] One embodiment of this specification enables rapid identification of interface changes without relying on image templates for image matching. It uses a first strategy model to perform semantic understanding of interface elements, obtaining executable operations. This reduces interface disturbances such as button position offsets and occlusions, improving the accuracy and stability of user interface testing and lowering testing and maintenance costs. Furthermore, it can construct structured samples based on real-time acquired data, increasing the richness of sample data for the UI multimodal model and thus improving the accuracy of the UI multimodal model trained on the sample data. Attached Figure Description

[0011] Figure 1 This is a flowchart of a user interface testing method provided in one embodiment of this specification; Figure 2 This is a schematic diagram of the text labels corresponding to controls in a user interface provided in one embodiment of this specification; Figure 3 This is a data flow diagram provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a user interface testing device provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0012] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0015] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0016] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0017] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] Multimodal Large Model: Refers to a large-scale pre-trained model capable of simultaneously processing multiple modal inputs such as images, text, and structured data, such as GPT-4o. In this specification, the multimodal large model is used to combine game interface screenshots with task semantic descriptions to achieve interface understanding and operation path reasoning.

[0020] UI-TARS: An open-source, multimodal GUI interaction architecture with visual perception, natural language understanding, and operation planning capabilities. In this specification, the UI multimodal model specifically refers to UI-TARS, which can serve as a deep inference engine for task decomposition and action sequence generation.

[0021] Reinforcement Learning (RL) is a machine learning method that optimizes policies based on environmental interaction feedback. In this specification, it is used in the self-healing module to achieve path reconstruction and process recovery in abnormal states by recording failure trajectories and optimizing policies.

[0022] PPO / GRPO: PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization) are common reinforcement learning policy optimization algorithms used to train models to select the optimal action sequence under different states.

[0023] Chain-of-Thought (CoT) reasoning mechanism: a reasoning method that introduces intermediate thinking steps into large models, enabling the model to perform multiple rounds of thinking and task decomposition. In this specification, it is used to support multi-step task planning in complex processes.

[0024] Task Semantics: refers to the natural language task objectives input by the user, such as "Complete the first chapter of the beginner's guide" or "Claim all rewards". The system needs to parse the semantics and map them into interface interaction actions.

[0025] RAG (Retrieval-Augmented Generation): A retrieval-enhanced generation mechanism that assists large models in generating more accurate task plans or response results by retrieving relevant documents / rules from external knowledge bases. In this invention, it is used to provide background knowledge such as process semantics and control definitions, enhancing the model's reasoning context.

[0026] SFT (Supervised Fine-Tuning): Supervised fine-tuning refers to the process of retraining a pre-trained large model using labeled data to improve its performance in specific task scenarios. In this specification, it is used to adapt to specific game flows and interface styles. State caching mechanism: By using interface fingerprints (such as pHash, SimHash) to determine whether the current state is consistent with the historical records, repeated perception and reasoning steps are skipped, thereby improving execution efficiency.

[0027] Reinforcement learning-based self-healing refers to a mechanism in which a system can automatically recover its process by trying new paths through reinforcement learning when it encounters execution anomalies (such as infinite loops or UI errors).

[0028] Game UI automation testing: refers to a testing method that uses programs to automatically perform interface operation tasks (such as clicking, swiping, and jumping) in a game environment to verify the integrity of the process and the correctness of the functions.

[0029] In the field of game development, with the rapid increase in mobile game categories and frequent game version iterations, the requirements for user interface testing technology are becoming increasingly stringent. Game user interfaces are highly dynamic and non-linear, with frequent changes, rich interface elements, and complex interface transition paths. Traditional script-driven automated testing technologies for game UIs are insufficient to meet these testing needs.

[0030] As mentioned earlier, image matching methods are prone to failure when game versions iterate or elements in the game's user interface change, such as button position shifts, icon replacements, or occlusions, making user interface testing impossible. Additionally, while there are currently testing methods based on script recording mechanisms, these lack support for multi-state transitions and flow branches, resulting in limited test coverage and failing to guarantee the accuracy and stability of user interface testing.

[0031] Based on this, this application provides a user interface testing method, which can acquire a task to be tested and acquire a first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language description text; identify current interface elements in the first current user interface image; obtain a first currently executable operation through reasoning using a first strategy model based on the task to be tested, the first current user interface image, and the current interface elements, wherein the first strategy model is a UI multimodal model; execute the first currently executable operation, and obtain the next executable operation through the first strategy model based on the execution context of the first currently executable operation, until the test stop condition is reached, thereby completing the task to be tested; wherein, when executing the first currently executable operation, the execution path corresponding to the task to be tested is acquired in real time, the execution path including the sequence of executable operations and state changes, the user interface sequence, and the interface element sequence corresponding to the task to be tested, and generate structured samples based on the real-time acquired execution path and the task to be tested.

[0032] It enables rapid identification of interface changes without relying on image templates for image matching. Through a first-strategy model, it performs semantic understanding of interface elements to obtain executable operations, reducing interface disturbances such as button misalignment and occlusion. This improves the accuracy and stability of user interface testing and lowers testing and maintenance costs. Furthermore, it can construct structured samples based on real-time acquired data, increasing the richness of sample data for the UI multimodal model and thus improving the accuracy of the UI multimodal model trained on the sample data.

[0033] Furthermore, current game UI automated testing technologies generally lack anomaly recognition and self-healing mechanisms. If anomalies such as recognition failure, redirection errors, or resource loading errors occur during execution, the test task cannot be completed, resulting in poor test stability and accuracy. As with the two methods mentioned above, the system cannot automatically repair or roll back when anomalies are detected during test task execution. However, one or more embodiments in this specification also provide anomaly event monitoring and a reinforcement learning-based self-healing mechanism. Even when anomalies are detected, the test task can still be completed accurately and smoothly, improving test stability and accuracy.

[0034] This specification provides a user interface testing method, and also relates to a user interface testing apparatus, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0035] See Figure 1 , Figure 1 A flowchart of a user interface testing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0036] Step 102: Obtain the task to be tested and obtain the first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language description text.

[0037] In one or more embodiments of this specification, the executing entity of the user interface testing method can be any computing device with computing capabilities, including but not limited to terminals (such as mobile phones, tablets), servers, etc., and this specification does not impose specific limitations on it.

[0038] In practical applications, the system can obtain the task to be tested in response to user input, or it can receive a test request carrying the task to be tested, thereby obtaining the task to be tested from the test request. Furthermore, the task to be tested in this specification can be a natural language description text corresponding to the task; that is, in practical applications, user input can be a natural language description text, such as: completing a beginner's tutorial task.

[0039] It should be noted that in practical applications, the natural language description text corresponding to the test task can originate from the graph structure corresponding to the test task, such as the graph structure corresponding to "entering the game store interface - purchasing a specific virtual character". It can also originate from the target image example corresponding to the test task, which can be a screenshot of the desired interface. In other words, the user input can be the graph structure or the target image example corresponding to the test task, which can then be converted into natural language description text to obtain the natural language description text corresponding to the test task for subsequent processing.

[0040] In one or more embodiments of this specification, it is also necessary to determine a first current user interface corresponding to the task to be tested. Here, the first current user interface refers to the user interface corresponding to the current time step.

[0041] It should be understood that the task to be tested requires multiple executable operations, i.e., multiple user interface transitions, to be completed. In this specification, the entire process of executing the test task is divided into several time steps. Each time step corresponds to at least one user interface and at least one executable operation. The executable operation corresponding to each time step is determined by the user interface and the task to be tested for each time step.

[0042] Step 104: Identify the current interface elements in the first current user interface image.

[0043] The first current user interface image is the image corresponding to the first current user interface. In one or more embodiments of this specification, the first current user interface image can be obtained by capturing the image corresponding to the first current user interface using any screenshot tool.

[0044] In one or more embodiments of this specification, a lightweight model including object detection functionality can be used to identify current interface elements in a first current user interface image. Interface elements may include controls on the interface, specifically including the type and position of the controls. These controls include, but are not limited to, interactive controls such as buttons, icons, input fields, labels, and sliders, as well as structural controls. In other words, the lightweight model can identify the position and type of controls in the first current user interface image.

[0045] In one or more embodiments of this specification, interface elements may also include text labels. Therefore, for each identified control, the text label corresponding to each control can be obtained through a text extraction function. In the gaming field, text labels can be set on controls. These text labels refer to text pre-rendered on the control, representing the control's function, purpose, or operational intent in the game. These text labels can be used for semantic understanding and task matching. For example, in completing a beginner's tutorial test task, a button control including the text "Next" can be displayed on a certain interface. See [link to documentation]. Figure 2 , Figure 2 This is a schematic diagram of the text labels corresponding to controls in a user interface provided in one embodiment of this specification. It can be seen that a button control displays the text "Next". The "Next" indicates the user's operation path. By clicking the "Next" button, you can jump to the next user interface in the novice guide process.

[0046] It should be noted that this specification does not limit the specific forms of object detection and text extraction. For example, lightweight models may include YOLO, Faster R-CNN, or other object detection models to locate the bounding boxes of each control in the first current user interface, i.e., determine the position of the control, and output the type of each control. Text extraction models may be optical character recognition models, etc., to extract the text displayed on each control in the first current user interface, i.e., text labels.

[0047] Based on the aforementioned lightweight model and text extraction model, the current interface elements in the first current user interface can be obtained. These current interface elements may include the position, type, and text label of the control. In practical applications, the position, type, and text label of the control can be obtained for subsequent multimodal understanding and executable operation decision-making of the first strategy model, thereby improving the first strategy model's focus on the control and improving the accuracy of the executable operation output by the first strategy model, so that the test task can be completed accurately and smoothly.

[0048] Step 106: Based on the task to be tested, the first current user interface image, and the current interface elements, obtain the first currently executable operation through reasoning using the first strategy model, wherein the first strategy model is a UI multimodal model.

[0049] The executable operations include, but are not limited to, user interface interaction operations such as clicking and swiping.

[0050] In practical applications, the task to be tested, the first current user interface image, and the current interface elements can be concatenated and input into the first strategy model to obtain the first currently executable operation.

[0051] In one or more embodiments of this specification, the first strategy model may be a multimodal large model, such as the UI-TARS model, the Flamingo model, and the LLaVA model.

[0052] Furthermore, in one or more embodiments of this specification, the first strategy model may include an image encoder such as ViT and Swin Transformer, a text encoder such as LLaMA and Qwen, a fusion network such as Cross-Attention, and a prediction network such as a fully connected network. The image encoder is used to encode input images, such as the target task image, user interface image, and control image corresponding to the test task, to obtain image features. The text encoder is used to encode input text, such as the natural language description text and text labels corresponding to the test task, to obtain text features. The fusion network is used to fuse image features and text features to obtain fused features. The prediction network can obtain executable operations based on the fused features. This specification does not limit the structure of the first strategy model, the specific input data, or the data flow process. The structure of the first strategy model can be set according to actual needs, and the input data and data flow process of each structure of the first strategy model can be determined according to actual needs.

[0053] In one or more embodiments of this specification, when obtaining a first currently executable operation based on the task to be tested, the first current user interface image, and the current interface elements through a first strategy model, historical information corresponding to the task to be tested can also be obtained. This historical information includes historical executable operations, historical user interfaces, and historical interface elements. Then, the historical information, the task to be tested, the first current user interface image, and the current interface elements are input into the first strategy model to obtain the first currently executable operation.

[0054] It should be noted that the historical information corresponding to the task to be tested refers to the information prior to the first currently executable operation, which may include historical executable operations prior to the first currently executable operation, historical user interfaces prior to the first current user interface, and historical interface elements in the historical user interface.

[0055] In one or more embodiments of this specification, the historical information corresponding to the task to be tested is also input into the first strategy model to provide auxiliary information for the first strategy model, thereby improving the accuracy of the first currently executable operation determined by the first strategy model.

[0056] Step 108: Execute the first currently executable operation, and obtain the next executable operation according to the execution context of the first currently executable operation through the first strategy model, until the test stop condition is reached, and complete the task to be tested.

[0057] It should be noted that the execution context includes historical information corresponding to the next executable operation. Therefore, the execution context of the first currently executable operation can be used to obtain auxiliary information for the next executable operation, which can improve the accuracy of the determined next executable operation.

[0058] The execution context of the first currently executable operation includes at least the first currently executable operation, the user interface and interface elements obtained after the first currently executable operation is executed.

[0059] The test stopping conditions can be determined based on actual needs, such as completing the task objective corresponding to the task to be tested, obtaining the current user interface as the target interface, or including the target interface element in the interface elements, or the number of time steps of the task to be tested reaching a preset number, or the number of times the executable operation is determined reaching a preset number, etc. This specification does not impose specific restrictions on these conditions.

[0060] Step 110: When performing the first currently executable operation, the execution path corresponding to the task to be tested is obtained in real time, and a structured sample is generated based on the real-time obtained execution sequence and the task to be tested. The execution path includes the executable operation sequence and state changes, user interface sequence and interface element sequence corresponding to the task to be tested.

[0061] It should be noted that for a given task to be tested, there can be multiple execution paths, and therefore multiple structured samples. Specifically, for each time step except the last time step, the execution path of the task to be tested corresponding to that time step can be determined, denoted as the intermediate execution path. This yields the structured sample corresponding to the task to be tested at that time step, which can be called the intermediate structured sample. Similarly, the execution path of the task to be tested corresponding to the last time step can be determined, denoted as the global execution path. This execution path is the complete execution path corresponding to the task to be tested, and thus the structured sample corresponding to the task to be tested at the last time step can be obtained, which can be called the global structured sample.

[0062] It should be understood that intermediate execution paths are part of the global execution path. For example, if the global execution path is ABCD, then intermediate execution paths may include AB and ABC.

[0063] It should be understood that, as described in step 102, the task to be tested needs to be completed through multiple executable operations, i.e. multiple user interface jumps. Therefore, the execution path corresponding to the task to be tested at each time step can be obtained. The execution path includes the sequence of executable operations and state changes, user interface sequence and interface element sequence corresponding to the task to be tested at each time step.

[0064] Here, "state change" refers to the change in state of the next frame of the user interface image obtained after executing the first currently executable operation, relative to the current user interface image. In one or more embodiments of this specification, this state change can be set based on actual needs, and this specification does not limit it. For example, based on the next frame of the user interface image, it can be determined whether the interface transition is correct, and thus the state change can be set to transition successful or transition failed. Alternatively, the state change can be an identifier corresponding to the user interface image, including the identifier of the current user interface image and the identifier of the next frame of the user interface image obtained after executing the first currently executable operation.

[0065] Based on the above user interface testing method, a test task can be obtained, and a first current user interface corresponding to the test task can be obtained, wherein the test task is natural language description text; current interface elements in the first current user interface image can be identified; based on the test task, the first current user interface image, and the current interface elements, a first currently executable operation can be obtained through reasoning using a first strategy model, wherein the first strategy model is a UI multimodal model; the first currently executable operation can be executed, and the next executable operation can be obtained through the first strategy model based on the execution context of the first currently executable operation, until the test stop condition is reached, thus completing the test task; wherein, when executing the first currently executable operation, the execution path corresponding to the test task can be obtained in real time, the execution path including the sequence of executable operations and state changes, the user interface sequence, and the interface element sequence corresponding to the test task, and structured samples can be generated based on the real-time obtained execution path and the test task.

[0066] It enables rapid identification of interface changes without relying on image templates for image matching. Through a first-strategy model, it performs semantic understanding of interface elements to obtain executable operations, reducing interface disturbances such as button misalignment and occlusion. This improves the accuracy and stability of user interface testing and lowers testing and maintenance costs. Furthermore, it can construct structured samples based on real-time acquired data, increasing the richness of sample data for the UI multimodal model and thus improving the accuracy of the UI multimodal model trained on the sample data.

[0067] Furthermore, in one or more embodiments of this specification, step 104 above, identifying the current interface element in the first current user interface, includes: Obtain the first identifier corresponding to the current user interface image and the second identifier corresponding to the previous user interface image; If the first identifier is the same as the second identifier, obtain the previous executable operation and continue to execute the previous executable operation; If the first identifier is different from the second identifier, identify the current interface element in the first current user interface.

[0068] In one or more embodiments of this specification, when the first identifier and the second identifier are the same, the previous executable operation can be obtained and executed, wherein the previous executable operation is the executable operation corresponding to the previous frame of the user interface of the first current user interface. After executing the previous executable operation in the first current user interface, the next frame of the user interface can be obtained, and the next executable operation can be obtained until the test stop condition is met, thus completing the task to be tested. When the first identifier and the second identifier are different, the current interface element in the first current user interface can be identified, and the subsequent reasoning process, i.e., steps 106-110, can be entered.

[0069] In practical applications, when a computing device executes a task under test, there may be cases where the user interface jump fails. When determining the first currently executable operation corresponding to the first current user interface, it can be determined whether the previous frame user interface image corresponding to the first current user interface image is the same as the first current user interface image. If not, that is, if the previous frame user interface image corresponding to the first current user interface image is not the same as the first current user interface image, it means that the user interface jump is successful. The current interface element in the first current user interface image can be identified, and the subsequent reasoning process, namely steps 106-110, can be performed.

[0070] If the previous frame of the user interface image corresponding to the first current user interface image is the same as the first current user interface image, it indicates that the user interface jump failed. If the subsequent reasoning process is carried out, it will lead to repeated identification and repeated determination of executable operations. In this case, the previous executable operation can be obtained. The previous executable operation is the executable operation corresponding to the previous frame of the user interface of the current first user interface. The previous executable operation can be executed on the first current user interface to obtain the next frame of the user interface. According to the above user interface testing method, based on the obtained next frame of the user interface, the next executable operation corresponding to the next frame of the user interface can be obtained until the test stop condition is reached and the test task is completed.

[0071] Specifically, when determining whether the previous frame user interface image corresponding to the first current user interface image is the same as the first current user interface image, it can be determined by an identifier. This identifier can be the hash value of the image; that is, the first identifier is the hash value corresponding to the first current user interface image, and the second identifier is the hash value corresponding to the previous frame user interface image of the first current user interface image. Of course, other methods can also be used, such as manual comparison, and this specification does not impose specific restrictions on this.

[0072] Based on the above method, an interface fingerprint mechanism is used—that is, a hash value that identifies a user interface image—to determine whether the previous frame user interface image corresponding to the first current user interface image is the same as the first current user interface image. This skips the element recognition and first strategy model inference steps, reduces repetition, and improves execution efficiency. In other words, by comparing the first current user interface image with the previous frame user interface image, the situation of repeatedly determining the first currently executable operation due to user interface jump failure can be avoided, improving testing efficiency and testing accuracy.

[0073] Furthermore, in one or more embodiments of this specification, before obtaining the first currently executable operation, the following steps are also included: The current interface element in the first current user interface image is identified using a lightweight model, wherein the current interface element includes the position, type, and text label of the control; If the difference between the current interface element identified by the lightweight model and the preset benchmark interface element is within a preset range, the lightweight model directly outputs the second currently executable operation based on the task to be tested, the first current user interface image, and the current interface element.

[0074] In practical applications, when a computing device executes a task to be tested, in addition to considering changes in the entire user interface image as described above, changes in interface elements can also be considered. In one or more embodiments of this specification, a lightweight model can be used to identify the current interface element in the first current user interface image. If the benchmark difference between the current interface element identified by the lightweight model and a preset benchmark interface element is within a preset range, in order to save computing resources and improve testing efficiency, a second currently executable operation can be directly obtained through a lightweight strategy model. That is, the task to be tested, the first current user interface image, and the current interface element are input into the lightweight model to obtain the second currently executable operation output by the lightweight model. Then, the subsequent steps can be performed, that is, according to steps 106-108 in the above-described user interface testing method, the second currently executable operation is executed, and the next executable operation is determined until the test stop condition is met, thus completing the task to be tested.

[0075] In one or more embodiments of this specification, the baseline differences may include differences in the type of controls corresponding to interface elements, differences in the position of controls, and differences in the text labels of controls.

[0076] In one or more embodiments of this specification, when determining the benchmark difference between the current interface element identified by the lightweight model and a preset benchmark interface element, the preset benchmark interface element can be obtained first. Then, the benchmark difference between the preset benchmark interface element and the current interface element identified by the lightweight model can be determined. When the benchmark difference is within a preset range, a second currently executable operation can be obtained through the lightweight strategy model based on the task to be tested, the first current user interface image, and the current interface element. The preset benchmark interface element can be set according to actual needs.

[0077] It should be noted that when determining the benchmark difference, the difference between the feature vectors corresponding to the preset benchmark interface elements and the current interface elements identified by the lightweight model can be determined. For example, a pre-trained feature encoding model or the lightweight model can be used to obtain the first feature corresponding to the preset benchmark interface elements, and the second feature corresponding to the current interface elements can be obtained based on the lightweight model. Thus, the difference between the first and second features can be determined to obtain the benchmark difference. This specification does not specifically limit the method for determining the difference between the first and second features; for example, the cosine similarity calculation method can be used to determine the difference between the first and second features. Of course, in practical applications, the aforementioned benchmark difference can also be determined manually.

[0078] In other words, during the execution of the test task, the user interface testing method provided in this manual can select different execution strategies based on different situations. As mentioned above, if the user interface image does not change, the recognition and model reasoning steps can be omitted to avoid repeated recognition and model reasoning. If the user interface changes but the interface elements in the user interface image do not change much, a lightweight model can be used for recognition and reasoning, thereby improving the efficiency and accuracy of executing the test task.

[0079] Furthermore, in one or more embodiments of this specification, obtaining a first currently executable operation through reasoning using a first strategy model based on the task to be tested, the first current user interface image, and the current interface elements includes: Based on the RAG structured search library, RAG-enhanced search is performed on the current structured sample to obtain the current search results; The current search result, the task to be tested, the first current user interface image, and the current interface elements are input into the first strategy model to infer the first currently executable operation. Accordingly, the method further includes: The RAG structured search library is updated based on the structured samples.

[0080] In other words, when the first strategy model obtains the first currently executable operation through reasoning, it can also retrieve target structured samples, i.e., search results, that are similar to the current structured samples corresponding to the task under test, based on the RAG structured retrieval library. For example, it can retrieve similar process or control information from previously recorded execution trajectories and structured samples, and concatenate these results into the input of the first strategy model as contextual hints. This helps the first strategy model to more accurately understand the task objective or the operation mode of the current interface, thereby improving the reasoning effect.

[0081] In practical applications, the RAG structured retrieval library can be constructed based on structured samples corresponding to historical test tasks. These structured samples include executable operation sequences, state changes, user interface sequences, and interface element sequences corresponding to historical test tasks. Therefore, when executing each test task, a target test task matching the current test task can be determined from the RAG structured retrieval library; specifically, a target structured sample matching the current structured sample corresponding to the current test task. In one or more embodiments of this specification, when determining each executable operation of the test task, a matching operation can be performed to obtain a target structured sample matching the current structured sample corresponding to the test task. The target structured sample, the test task, the first current user interface image, and the current interface elements can then be concatenated and input into a first strategy model to obtain a first current executable operation. Alternatively, the target structured sample, the current structured sample, the test task, the first current user interface image, and the current interface elements can be concatenated and input into the first strategy model to obtain a first current executable operation.

[0082] Accordingly, in practical applications, the RAG structured search library can be updated based on the structured samples corresponding to the current test task. The structured samples corresponding to the current test task can be added to the RAG structured search library.

[0083] It should be noted that for a given test task, there can be multiple structured samples. Specifically, for each time step except the last one, a structured sample corresponding to the test task at that time step can be determined. Such a structured sample can be called an intermediate structured sample. The structured sample obtained at the last time step is the complete structured sample corresponding to the entire test task and can be called the final structured sample. Therefore, when using the RAG enhanced retrieval structure as input to the first strategy model, a retrieval result can be determined once for each time step. Alternatively, a retrieval result can be determined only at the beginning of the test task, and no further retrieval results can be determined for the remaining time steps to improve testing efficiency.

[0084] In practical applications, both the intermediate and final structured samples corresponding to the task under test can be stored in the RAG structured retrieval library. Alternatively, intermediate structured samples can be selectively stored. Alternatively, after the task under test is completed, only the final structured sample corresponding to the task under test—that is, the entire sequence of executable operations, state changes, user interface sequences, and interface element sequences—can be stored in the RAG structured retrieval library.

[0085] Based on the above approach, a Retrieval Enhancement Generation (RAG) mechanism is established. This mechanism, a structured retrieval library built from structured samples corresponding to historical test tasks, provides auxiliary information for the execution of the current test task, thereby improving the accuracy of the first executable operation determined by the first strategy model. Furthermore, during the execution of the test task, the knowledge graph library can be updated in real time, enhancing the diversity and comprehensiveness of the data within it, further improving the accuracy of the determined first executable operation.

[0086] Furthermore, in one or more embodiments of this specification, after completing the test task, the method further includes: The training dataset is updated based on the structured samples, wherein the training dataset includes the executable operation sequence and state changes, user interface sequence, interface element sequence and test results corresponding to the historical test tasks; The first policy model is fine-tuned under supervision based on the training dataset.

[0087] In one or more embodiments of this specification, a training dataset may also be constructed, which includes structured samples corresponding to the task to be tested and test results, so that the first policy model can be fine-tuned under supervision based on the training data.

[0088] The test results can be an evaluation value for each structured sample. This evaluation value can be obtained by evaluating the structured samples corresponding to each test task using a large language model or manually. This evaluation value is the test result. The test result can also be the execution status of the test task, such as success or failure, which can be obtained through a large language model or manually.

[0089] In practical applications, the first policy model is fine-tuned under supervision based on the training dataset until the model training stopping condition is met, resulting in a fine-tuned first policy model. The next test task can then be executed based on this fine-tuned model. The model training stopping condition includes, but is not limited to, the loss being less than a preset loss threshold, the amount of training data used reaching a preset number of training data points, and the number of iterations reaching a preset number of iterations.

[0090] It should be noted that when supervising the fine-tuning of the first strategy model, the first strategy model can be supervised and fine-tuned under the condition that the preset conditions are met. The preset conditions can be that the number of times the test task is executed reaches a specified number, the current time is a specified time, etc., which is a preset period. This manual does not make specific restrictions here.

[0091] It should be noted that using supervised fine-tuning to train the first policy model is a relatively mature technique, and will not be elaborated upon here. For example, at least one sample test task can be identified, which can be any test task in the training dataset. For each time step in the sample test task, the executable operation sequence, state changes, user interface image sequence, interface element sequence, and test result corresponding to that time step are obtained. The executable operation at that time step can be used as the labeled executable operation, and the test result at that time step can be used as the labeled test result. The sample test task, the executable operation sequence, state changes, user interface image sequence, and interface element sequence corresponding to that time step are input into the first policy model to obtain the predicted executable operation. The first policy model is fine-tuned with the goal of minimizing the difference between the labeled executable operation and the predicted executable operation. Furthermore, the predicted test results of the structured samples composed of the predicted executable operations can also be determined, and the first policy model is fine-tuned based on the labeled test results and the predicted test results.

[0092] Furthermore, in practical applications, the prediction user interface at the next time step after the execution of the prediction executable operation can be obtained, and the prediction interface elements in the prediction user interface at the next time step can be obtained. The user interface and interface elements corresponding to the next time step in the sample test task are respectively used as the label user interface and label interface elements. Based on the prediction interface elements, prediction user interface, label interface elements and label user interface, the first strategy model is fine-tuned.

[0093] In practical applications, high-quality training data from the training dataset can be used to supervise the fine-tuning of the first policy model. This can be achieved by using structured samples where all state changes are successful and the test results are better than the preset test results as high-quality training data.

[0094] Based on the above method, during the execution of user interface testing tasks, structured samples are collected in real time and used as training datasets to supervise and fine-tune the first policy model, which can improve the performance of the first policy model and increase its sensitivity to testing tasks.

[0095] As mentioned earlier, current game UI automated testing technologies generally lack anomaly identification and self-healing mechanisms. Once anomalies such as identification failure, jump anomalies, or resource loading errors occur during execution, the test task cannot be completed, resulting in poor test stability and accuracy. As mentioned earlier, the two current user interface testing methods cannot automatically repair or roll back when anomalies are detected during the execution of test tasks.

[0096] Therefore, this specification provides a reinforcement learning self-healing mechanism, which can automatically restore the process by trying new paths through reinforcement learning methods when encountering execution abnormalities (such as infinite loops or interface abnormalities).

[0097] Specifically, regarding the aforementioned user interface testing methods, abnormal events during the user interface testing process can be monitored while completing the task to be tested. When an abnormal event is detected, the second current user interface is obtained, along with the sequence of executable operations, the user interface sequence, and the sequence of interface elements preceding the second current user interface. The second current user interface image, the task to be tested, the sequence of executable operations before the second current user interface, the user interface sequence, and the sequence of interface elements are input into the second policy model to obtain the third current executable operation. The second policy model is a policy model that generates executable operations through a reinforcement learning algorithm. Execute the third currently executable operation; and write the successful path of the task to be tested into the state cache and the training dataset.

[0098] The second policy model can be a policy model constructed using a reinforcement learning algorithm that can generate executable operations. This reinforcement learning algorithm can be PPO or GRPO.

[0099] It should be noted that the specific methods for generating executable operations using a second policy model constructed through reinforcement learning algorithms such as PPO and GRPO are existing technologies and will not be elaborated upon in this specification.

[0100] In one or more embodiments of this specification, monitoring anomalous events during user interface testing includes: If no next frame of the user interface is obtained after executing the first currently executable operation, an abnormal event is determined to have been detected; or, If no current interface element in the first current user interface image is identified, an abnormal event is determined to have been detected; or, If there is no response when operating the first current user interface, an abnormal event is detected.

[0101] Of course, in practical applications, abnormal events may also include failure to execute an executable operation, the number of times the same executable operation is executed exceeds a preset threshold, and the time taken to execute the task under test exceeds a preset time threshold.

[0102] Based on the above, abnormal events can be monitored, and based on the reinforcement learning self-healing mechanism, the testing task can still be completed accurately and smoothly even when abnormal events are detected, thereby improving the stability and accuracy of the test.

[0103] Based on the user interface testing method described above, this specification provides a data flow diagram to more clearly reflect the execution of the test task and the training of the first strategy model. See also... Figure 3 , Figure 3 This is a data flow diagram provided in one embodiment of this specification. The user interface testing method provided in this specification uses multimodal information, including the task under test, user interface images, and interface elements, to enable a first strategy model to perform semantic reasoning of the task under test and predict executable operations. In practical applications, the first strategy model can use a chain-of-thought (CoT) approach to decompose the task under test, thereby achieving contextual understanding of complex tasks and generation of multi-step execution paths, improving the stability and accuracy of user interface testing.

[0104] Furthermore, a reinforcement learning-driven self-healing execution mechanism is introduced, which can automatically identify, roll back, and retry when abnormal events are detected, improving the fault tolerance and recovery capabilities of user interface testing and ensuring a closed-loop testing process. Additionally, structured training data is constructed, and combined with RAG and SFT mechanisms, the input information when the first policy model predicts executable operations is increased, improving the accuracy of the generated executable operations. Incremental training of the first policy model allows it to continuously learn and adaptively evolve in real-world testing tasks, thereby improving its performance.

[0105] Corresponding to the above method embodiments, this specification also provides embodiments of a user interface testing device. Figure 4 A schematic diagram of a user interface testing apparatus according to one embodiment of this specification is shown. Figure 4 As shown, the device includes: The interface perception module 402 is configured to: acquire a task to be tested and acquire a first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language description text; and identify current interface elements in the first current user interface image. The hierarchical reasoning module 404 is configured to obtain a first currently executable operation by reasoning through a first strategy model based on the task to be tested, the first current user interface image, and the current interface elements, wherein the first strategy model is a UI multimodal model. The operation execution module 406 is configured to execute the first currently executable operation, and obtain the next executable operation through reasoning using the first strategy model based on the execution context of the first currently executable operation, until the test stop condition is met, thereby completing the task to be tested; wherein, when executing the first currently executable operation, the execution path corresponding to the task to be tested is obtained in real time, and a structured sample is generated based on the real-time obtained execution path and the task to be tested, wherein the execution path includes the sequence of executable operations and state changes, the user interface sequence, and the sequence of interface elements corresponding to the task to be tested.

[0106] The operation execution module 406 includes an agent, which is used to execute the intermediate inference chain and multi-step operation plan generated by the first policy model based on the chain-of-thought mechanism; wherein, the agent entrusts the executor to execute the sequence of test actions step by step.

[0107] Optionally, the interface perception module 402 is further configured to: obtain a first identifier corresponding to the first current user interface image and a second identifier corresponding to the previous frame user interface image; if the first identifier and the second identifier are the same, obtain the previous executable operation and continue to execute the previous executable operation; if the first identifier and the second identifier are different, identify the current interface element in the first current user interface.

[0108] Optionally, the interface perception module 402 is further configured to identify current interface elements in the first current user interface image through a lightweight model, wherein the current interface elements include the position, type, and text label of controls; if the benchmark difference between the current interface element identified by the lightweight model and the preset benchmark interface element is within a preset range, the lightweight model outputs a second currently executable operation based on the task to be tested, the first current user interface image, and the current interface element.

[0109] Optionally, the hierarchical reasoning module 404 is further configured to perform RAG-enhanced retrieval on the current structured sample based on the RAG structured retrieval library to obtain the current retrieval result; and input the current retrieval result, the task to be tested, the first current user interface image, and the current interface elements into the first strategy model for reasoning to obtain the first currently executable operation. Accordingly, the user interface testing device also includes a data update module 408; The data update module 408 is configured to update the RAG structured search library based on the structured sample.

[0110] Optionally, the user interface testing device further includes a model fine-tuning module 410; The model fine-tuning module 410 is configured to update the training dataset based on the structured samples, wherein the training dataset includes executable operation sequences and state changes, user interface sequences, interface element sequences, and test results corresponding to historical test tasks; and to supervise the fine-tuning of the first policy model based on the training dataset.

[0111] Then, the first policy model, which has been trained and updated, is effectively injected into the Agent of the operation execution module 406.

[0112] Optionally, the user interface testing device further includes a self-healing testing module 412; The test self-healing module 412 is configured to monitor abnormal events during the user interface testing process: when an abnormal event is detected, it acquires a second current user interface and acquires the executable operation sequence, user interface sequence, and interface element sequence preceding the second current user interface; it inputs the second current user interface image, the task to be tested, the executable operation sequence preceding the second current user interface, the user interface sequence, and the interface element sequence into a second policy model to obtain a third current executable operation, wherein the second policy model is a policy model that generates executable operations through a reinforcement learning algorithm; it executes the third current executable operation; and it writes the successful path of the task to be tested into the state cache and the training dataset.

[0113] Optionally, the test self-healing module 412 is further configured to determine that an abnormal event has been detected if no next frame of user interface is obtained after the first currently executable operation is performed; or, determine that an abnormal event has been detected if the current interface element in the first current user interface image is not identified; or, determine that an abnormal event has been detected if there is no response when operating the first current user interface.

[0114] It should be noted that the specific implementation process of the recognition method and reasoning method involved in the user interface testing device provided in this embodiment is consistent with that described in the above-mentioned user interface testing method, and will not be repeated here.

[0115] Based on the user interface testing device provided in this embodiment, a test task can be acquired, and a first current user interface corresponding to the test task can be acquired, wherein the test task is natural language description text; current interface elements in the first current user interface image can be identified; based on the test task, the first current user interface image, and the current interface elements, a first currently executable operation can be obtained through reasoning using a first strategy model, wherein the first strategy model is a UI multimodal model; the first currently executable operation can be executed, and the next executable operation can be obtained through the first strategy model based on the execution context of the first currently executable operation, until the test stop condition is reached, thus completing the test task; wherein, when executing the first currently executable operation, the execution path corresponding to the test task can be acquired in real time, the execution path including the sequence of executable operations and state changes, the user interface sequence, and the interface element sequence corresponding to the test task, and structured samples can be generated based on the real-time acquired execution path and the test task.

[0116] The solution implemented in this specification allows for rapid identification of interface changes without relying on image templates for image matching. By using a first strategy model to perform semantic understanding of interface elements, executable operations can be obtained. This reduces interface disturbances such as button position offsets and occlusions, improving the accuracy and stability of user interface testing and lowering testing and maintenance costs. Furthermore, structured samples can be constructed based on real-time acquired data, increasing the richness of sample data for the UI multimodal model and thereby improving the accuracy of the UI multimodal model trained on the sample data.

[0117] The above is an illustrative scheme of a user interface testing device according to this embodiment. It should be noted that the technical solution of this user interface testing device and the technical solution of the user interface testing method described above belong to the same concept. For details not described in detail in the technical solution of the user interface testing device, please refer to the description of the technical solution of the user interface testing method described above.

[0118] Figure 5 A structural block diagram of a computing device 500 provided according to one embodiment of this specification is shown.

[0119] The computing device 500 includes: Memory 510 and processor 520; The memory 510 is used to store computer programs / instructions, and the processor 520 is used to execute the computer programs / instructions, which, when executed by the processor 520, implement the steps of the user interface testing method.

[0120] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0121] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0122] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0123] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the user interface testing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the user interface testing method described above.

[0124] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the user interface testing method described above.

[0125] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are described simply because they are substantially similar to the user interface testing method embodiments; relevant parts can be referred to in the description of the user interface testing method embodiments.

[0126] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0127] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0128] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0129] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A user interface testing method, characterized in that, include: Obtain the task to be tested and obtain the first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language description text; Identify the current interface element in the first current user interface image; Based on the task to be tested, the first current user interface image, and the current interface elements, a first currently executable operation is obtained through reasoning using a first strategy model, wherein the first strategy model is a UI multimodal model; Execute the first currently executable operation, and obtain the next executable operation through reasoning based on the execution context of the first currently executable operation using the first strategy model, until the test stop condition is met, and complete the task to be tested; Specifically, when executing the first currently executable operation, the execution path corresponding to the task to be tested is obtained in real time, and a structured sample is generated based on the real-time obtained execution path and the task to be tested. The execution path includes the executable operation sequence and state changes, user interface sequence and interface element sequence corresponding to the task to be tested.

2. The method as described in claim 1, characterized in that, Identify the current interface elements in the first current user interface, including: Obtain the first identifier corresponding to the current user interface image and the second identifier corresponding to the previous user interface image; If the first identifier is the same as the second identifier, obtain the previous executable operation and continue to execute the previous executable operation; If the first identifier is different from the second identifier, identify the current interface element in the first current user interface.

3. The method as described in claim 1, characterized in that, Before obtaining the first currently executable operation, it also includes: The current interface elements in the first current user interface image are identified using a lightweight model, wherein the current interface elements include the position, type, and text label of the controls; If the difference between the current interface element identified by the lightweight model and the preset benchmark interface element is within a preset range, the lightweight model outputs a second currently executable operation based on the task to be tested, the first current user interface image, and the current interface element.

4. The method as described in claim 1, characterized in that, Based on the task to be tested, the first current user interface image, and the current interface elements, a first currently executable operation is obtained through reasoning using a first strategy model, including: Based on the RAG structured search library, RAG-enhanced search is performed on the current structured sample to obtain the current search results; Input the current search results, the task to be tested, the first current user interface image, and the current interface elements into the first strategy model to obtain the first currently executable operation; Accordingly, the method further includes: The RAG structured search library is updated based on the structured samples.

5. The method as described in claim 1, characterized in that, After completing the task to be tested, the following is also included: The training dataset is updated based on the structured samples, wherein the training dataset includes the executable operation sequence and state changes, user interface sequence, interface element sequence and test results corresponding to the historical test tasks; The first policy model is fine-tuned under supervision based on the training dataset.

6. The method as described in claim 1, characterized in that, Also includes: Monitor abnormal events during user interface testing; When an abnormal event is detected, the second current user interface is obtained, along with the sequence of executable operations, the user interface sequence, and the sequence of interface elements preceding the second current user interface. The second current user interface image, the task to be tested, the sequence of executable operations before the second current user interface, the user interface sequence, and the sequence of interface elements are input into the second policy model to obtain the third current executable operation. The second policy model is a policy model that generates executable operations through a reinforcement learning algorithm. Execute the third currently executable operation; and write the successful path of the task to be tested into the state cache and the training dataset.

7. The method as described in claim 6, characterized in that, Monitor abnormal events during user interface testing, including: If no next frame of the user interface is obtained after executing the first currently executable operation, an abnormal event is determined to have been detected; or, If no current interface element in the first current user interface image is identified, an abnormal event is determined to have been detected; or, If there is no response when operating the first current user interface, an abnormal event is detected.

8. A user interface testing device, characterized in that, include: The interface awareness module is configured to: acquire a task to be tested and acquire a first current user interface corresponding to the task to be tested, wherein the task to be tested is natural language description text; and identify current interface elements in the first current user interface image. The hierarchical reasoning module is configured to obtain a first currently executable operation by reasoning through a first strategy model based on the task to be tested, the first current user interface image, and the current interface elements, wherein the first strategy model is a UI multimodal model. The operation execution module is configured to execute the first currently executable operation, and obtain the next executable operation through the first strategy model based on the execution context of the first currently executable operation, until the test stop condition is reached, thereby completing the task to be tested; wherein, when executing the first currently executable operation, the execution sequence corresponding to the task to be tested is obtained in real time, and a structured sample is generated based on the execution sequence and the task to be tested, wherein the execution sequence includes the executable operation sequence and state changes, user interface sequence and interface element sequence corresponding to the task to be tested.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images