A user interface test navigation system, method, and product

Abstract

This application provides a user interface testing navigation system, method, and product, relating to the field of user interface testing technology. The system includes: a memory module for maintaining a masked operation space; masked operations represent UI operations that did not achieve the expected results after execution in the corresponding UI state; a domain knowledge loader for deleting masked operations from the action space based on the masked operation space and the current UI state to obtain an optimized action space; an execution engine for generating operation prompts based on the optimized action space, enabling the first large language model (LLM) to determine the UI test actions for this round based on the operation prompts; obtaining the UI state after executing the UI test actions on the test application (AUT); and the domain knowledge loader for verifying whether the UI test achieved the expected results based on the UI state after the test. If the expected results were not achieved, the UI state is restored to the current UI state, and the UI test is re-performed.

Description

Technical Field

[0001] This application relates to the field of user interface testing technology, and in particular to a user interface testing navigation system, method and product. Background Technology

[0002] User interface (UI) testing is essential for ensuring the high quality of the application under test (AUT). To reduce the high manual labor cost of creating such UI tests, related technologies propose methods that automatically explore the application under test based on test objective descriptions to obtain UI action sequences that can be automatically generated for the test objectives (i.e., feature-based automated UI testing). Considering that feature-based automated UI testing is similar to traditional artificial intelligence planning problems, related technologies primarily employ Large Language Models (LLMs), renowned for their effectiveness in AI planning. The test objectives and related prompt text are input into the LLM, which generates a UI action sequence. This sequence is then automatically executed on the AUT (the application under test), achieving automated UI testing of the AUT.

[0003] However, UI testing tasks (i.e., feature-based automated UI testing) are sequential planning problems, requiring LLM to perform multi-step inference. The inference performance of LLM decreases with increasing task complexity, potentially leading to invalid UI operations in the inferred UI operation sequence. This results in the AUT failing to achieve the expected testing results when executing the UI operation sequence. Furthermore, LLM performs poorly in following multiple or fine-grained instructions, and it's difficult to directly improve the accuracy of LLM in planning UI operations by modifying the prompt text. Therefore, there is an urgent need to propose a user interface testing navigation system, method, and product to improve the effectiveness of LLM-based automated UI testing. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a user interface testing navigation system, method and product to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this application provides a user interface testing navigation system, the system comprising:

[0006] A memory module is used to maintain the masked operation space and historical test information; the masked operation space includes one or more masked operations, which represent UI operations that did not achieve the expected effect after being executed in the corresponding UI state;

[0007] A domain knowledge loader is used to optimize the action space based on the shielded operation space and the current UI state, delete the shielded operations in the action space, and obtain the optimized action space.

[0008] An execution engine is used to generate operation prompts based on the optimized action space, enabling the first large language model LLM to determine the UI test action for this round based on the operation prompts; and to obtain the UI state after the test is performed on the test application AUT.

[0009] The domain knowledge loader is also used to verify whether the current round of UI testing has achieved the expected results based on the UI state after the test. If the expected results have not been achieved, the UI state is restored to the current UI state and the UI test is repeated.

[0010] In one possible implementation, the domain knowledge loader includes: an optimizer;

[0011] The optimizer is used to parse the list of available UI operations of the current AUT interface according to the current UI state and generate the action space, wherein the action space represents the set of all UI operations that can be performed on the current AUT interface.

[0012] The optimizer optimizes the action space based on the masked operation space obtained from the memory module, and deletes the masked operations in the action space to obtain the optimized action space.

[0013] In one possible implementation, the execution engine includes: a parser, an LLM driver, and a UI driver;

[0014] The parser generates operation prompts based on the optimized action space and the historical test information obtained from the memory module; the operation prompts include: the optimized action space, the test target, and the historical UI operation sequence;

[0015] Using the LLM driver, the operation prompt is sent to the first LLM, and the response result obtained by the first LLM based on the operation prompt is received; the response result is the identifier of the UI operation that needs to be performed in the current UI state in this round of testing, as determined by the first LLM.

[0016] The parser analyzes the response result to obtain the UI test action for this round; the UI test action for this round includes one or more of the following: clicking, swiping, long pressing, and inputting text;

[0017] Using the UI driver, perform the current round of UI testing actions on the AUT to obtain the UI state after the test.

[0018] In one possible implementation, the domain knowledge loader includes: a verifier; the step of verifying whether the current round of UI testing achieved the expected results based on the post-test UI state includes:

[0019] The verifier sends the response result of the first LLM, the current UI state, and the UI state after the test as a triple to the second LLM, so that the second LLM can determine whether the UI state after the test is consistent with the planned test state and receive the verification result of the second LLM.

[0020] The verifier, based on heuristic rules, analyzes the UI state after the test to determine whether the UI test achieved the expected results.

[0021] In one possible implementation, the step of analyzing the post-test UI state based on heuristic rules to determine whether the current round of UI testing achieved the expected results includes:

[0022] Based on the UI state after the test, determine whether the same UI state has appeared in previous test rounds;

[0023] Determine whether the UI state after the test is an unresponsive UI state;

[0024] Based on the UI state after the test, determine whether a test state loop has occurred.

[0025] In one possible implementation, the domain knowledge loader includes: an error handling program; the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operations represent the UI operations executed to achieve the expected results in the corresponding round of UI testing, and the effective UI states represent the UI states obtained to achieve the expected results in the corresponding round of UI testing; the effective UI operations and the effective UI states correspond one-to-one.

[0026] If the expected effect is not achieved, the error handling procedure is used to perform UI state rollback, detect the position of the rolled-back UI state in the historical UI state sequence, and re-execute the corresponding effective UI operation according to the historical UI operation sequence to restore the current UI state.

[0027] In one possible implementation, if the expected effect is not achieved, the memory module treats the current round of UI testing action as a blocking operation, takes the current UI state as the UI state corresponding to the blocking operation, and updates the blocking operation space.

[0028] If the expected results are achieved, the memory module updates the historical test information, adds the current round of UI test actions as the effective UI operations to the historical UI operation sequence, and adds the post-test UI state as the effective UI state to the historical UI state sequence.

[0029] A second aspect of this application also provides a user interface testing navigation method, applied to the user interface testing navigation system described in the first aspect of this application, the method comprising:

[0030] Based on the masked operation space and the current UI state, the action space is optimized by deleting masked operations in the action space to obtain the optimized action space; the masked operation space includes one or more masked operations, which represent UI operations that do not achieve the expected effect after execution in the corresponding UI state;

[0031] Based on the optimized action space, operation prompts are generated, enabling the first large language model LLM to determine the UI test action for this round based on the operation prompts;

[0032] Obtain the UI state after the current round of UI test actions are performed on the test application AUT;

[0033] Based on the UI state after the test, verify whether the UI test has achieved the expected results. If the expected results have not been achieved, restore the UI state to the current UI state and retest the UI.

[0034] A third aspect of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the user interface testing navigation method described in the second aspect of this application.

[0035] The fifth aspect of this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the user interface testing navigation method described in the second aspect of this application.

[0036] The sixth aspect of this application also provides a computer program product that, when run on an electronic device, causes a processor to execute the steps in the user interface test navigation method described in the second aspect of this application.

[0037] This application provides a user interface testing navigation system, comprising: a memory module for maintaining a masked operation space and historical test information; the masked operation space includes one or more masked operations, each representing a UI operation that did not achieve the expected effect after execution in the corresponding UI state; a domain knowledge loader for optimizing the action space based on the masked operation space and the current UI state, deleting masked operations in the action space to obtain an optimized action space; an execution engine for generating operation prompts based on the optimized action space, enabling a first large language model (LLM) to determine the current round of UI test actions based on the operation prompts; obtaining the post-test UI state obtained by executing the current round of UI test actions on the test application (AUT); the domain knowledge loader is further configured to verify whether the current round of UI testing achieved the expected effect based on the post-test UI state, and if the expected effect was not achieved, restore the UI state to the current UI state and re-perform the UI test.

[0038] The specific beneficial effects are as follows:

[0039] UI testing requires a series of UI operations. This application's embodiments optimize the action space (the set of selectable UI operations for the current UI state) using a domain knowledge loader, removing invalid UI operations (i.e., masked operations, meaning UI operations that did not achieve the expected results after execution in the corresponding UI state) from the action space. Furthermore, through the domain knowledge loader, after each UI operation is completed, the UI state after testing is used to promptly verify whether the current round of UI testing achieved the expected results. If the expected results were not achieved, the UI state is restored to the state before the invalid operation was executed (i.e., the current UI state). This allows the LLM to re-determine the UI operations to be executed within the optimized action space, avoiding repeated selection of incorrect UI operations. This enables the LLM to re-plan UI operation sequences in a timely manner based on current information when errors occur in the currently planned UI operations, generating reliable and accurate UI operation sequences for the test target, thus improving the effectiveness of automated UI testing based on LLM. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the structure of a user interface testing navigation system provided in an embodiment of this application;

[0042] Figure 2 This is a flowchart illustrating a UI operation based on LLM planning, provided in an embodiment of this application.

[0043] Figure 3 This is a flowchart illustrating the steps of a user interface testing navigation method provided in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0046] User interface (UI) testing is essential for ensuring the high quality of the application under test (AUT). To reduce the high manual labor cost of creating such UI tests, related technologies have proposed methods to automatically explore the core functionality of the application under test based on the test objective description, resulting in the automatic generation of UI operation sequences for the test objective (i.e., feature-based automated UI testing). Specifically, given an AUT (the application under test) and a test objective, a UI operation is selected sequentially on the UI screen, triggering the UI operation on the AUT. Then, the next UI operation is selected on a new UI screen, and so on, until the test objective is achieved. Therefore, feature-based automated UI testing is essentially a sequential planning problem.

[0047] Given that feature-based UI testing tasks resemble traditional AI planning problems, related technologies primarily employ Large Language Models (LLMs), renowned for their effectiveness in AI planning. LLMs generate a sequence of executable UI actions on the AUT (Automatic Under Test) based on the input testing objective. While LLMs are highly effective for sequential planning problems, empirical studies show that LLM-based UI testing methods have lower effectiveness. LLMs are not adept at developing planning strategies specific to this type of exploration task (i.e., feature-based UI testing). Specifically, on one hand, LLMs struggle to follow specific UI testing instructions. LLMs lack domain knowledge in UI exploration, and existing methods incorporate UI-specific instructions into prompts to provide assistance. However, LLMs struggle to execute according to domain-specific task descriptions, leading to low effectiveness. For example, survey data shows that although drobotGPT uses explicit instruction prompts to avoid selecting already chosen actions, 36% of planned UI actions are simply repetitions of historical UI actions. On the other hand, LLMs struggle to replan based on new information when errors occur in the currently planned UI actions. For example, on the AUT's homepage, the LLM selects the "Click the search button" UI action to continue setting up a shortcut for night mode (the test objective is: to enable night mode on the AUT). When it finds that the search tab cannot achieve its testing purpose, the LLM cannot replan the UI action with new information and repeatedly clicks the search tab. Automated UI testing involves highly dynamic, real-time exploration of the AUT, as the same functionality can be implemented differently in different applications. As exploration progresses, newly discovered information may invalidate previously planned UI actions. In tasks like UI testing, multi-step reasoning is crucial, and the performance of the LLM may decrease with increasing task complexity. Furthermore, experience shows that the LLM performs poorly in following multiple or fine-grained instructions. In scenarios involving long text, the trend of declining model accuracy in following instructions is even more pronounced.

[0048] In view of the above problems, this application proposes a user interface testing navigation system, method, and product to improve the effectiveness of automated UI testing based on LLM. The user interface testing navigation system provided by this application will be described in detail below with reference to the accompanying drawings and through some embodiments and application scenarios.

[0049] The first aspect of this application provides a user interface testing navigation system, referring to... Figure 1 , Figure 1 This is a schematic diagram of the structure of a user interface testing navigation system provided in an embodiment of this application, such as... Figure 1 As shown, the system includes:

[0050] A memory module is used to maintain a masked operation space and historical test information; the masked operation space includes one or more masked operations, which represent UI operations that did not achieve the expected results after being executed in the corresponding UI state.

[0051] A domain knowledge loader is used to optimize the action space based on the shielded operation space and the current UI state, and to delete the shielded operations in the action space to obtain the optimized action space.

[0052] An execution engine is used to generate operation prompts based on the optimized action space, enabling the first large language model (LLM) to determine the UI test actions for this round based on the operation prompts; and to obtain the UI state after the test is performed on the test application (AUT).

[0053] The domain knowledge loader is also used to verify whether the current round of UI testing has achieved the expected results based on the UI state after the test. If the expected results have not been achieved, the UI state is restored to the current UI state and the UI test is repeated.

[0054] To automate UI testing, the test objectives and related prompts are often directly input into an LLM (Application Management Model), which generates a sequence of UI actions. These actions are then automatically executed on the AUT (Application Under Test) to automate UI testing. However, UI testing is a sequential planning problem, requiring the LLM to perform multi-step reasoning. The inference performance of an LLM may decrease with increasing task complexity, leading to invalid UI actions in the resulting sequence. Consequently, executing this sequence on the AUT may not achieve the desired testing results. Considering that LLM performs poorly in following multiple or fine-grained instructions, and it is difficult to improve the accuracy of LLM in planning UI operations by directly modifying the prompt text, this application provides a user interface testing navigation system. This system is a runtime system used to verify whether each UI operation achieves the expected effect when LLM plans the UI operations to be executed for the test target. If the expected effect is not achieved, the action space is optimized and invalid UI operations (i.e., masked operations) are deleted, thereby avoiding LLM repeatedly selecting the same invalid UI operations. Furthermore, by restoring the UI state to the state before executing the invalid UI operation, LLM can replan the next UI operation based on the previously planned correct UI operation sequence. Thus, there is no need to make too many changes to the prompt text input to LLM, which improves the accuracy of the planned UI operation sequence and ensures the inference efficiency of LLM.

[0055] Furthermore, it should be noted that the user interface testing navigation system proposed in this application can be applied not only to UI testing but also to user interface navigation application scenarios (e.g., as a user interface navigation assistant). In user interface navigation application scenarios, the functions performed by each module of the user interface testing navigation system are basically the same. Specifically, for a given user interface navigation target (equivalent to the test target in a UI testing scenario), the user interface testing navigation system uses a domain knowledge loader to optimize the action space by masking the operation space, resulting in an optimized action space. Then, the execution engine generates operation prompts based on the optimized action space, enabling the first large language model (LLM) to determine the UI operation for this round based on the operation prompts, thereby executing the UI operation for this round on the target application and obtaining the UI state after the operation (equivalent to the UI state after testing in a UI testing scenario). The domain knowledge loader then verifies whether the execution of this round of UI operation has achieved the expected effect based on the UI state after the operation. If the expected effect has not been achieved, the UI state is restored to the current UI state, and the user interface navigation for this round is repeated.

[0056] Reference Figure 2 , Figure 2 A flowchart illustrating a UI operation based on LLM planning is shown. The specific information processing flow of this user interface test navigation system is as follows:

[0057] Step 1: User Interface Testing. The navigation system obtains the current UI state from the AUT, and then uses the domain knowledge loader to optimize the action space based on the masked operation space and the current UI state, removing masked operations from the action space to obtain the optimized action space.

[0058] Specifically, such as Figure 2 As shown, the user interface test navigation system obtains the current UI state from the application under test (AUT), which includes the current user interface image. The user interface test navigation system includes a memory module that maintains a masked operation space (e.g., ...). Figure 2 As shown in B), a masking operation represents a UI operation that does not achieve the expected effect after execution in the corresponding UI state. A masking operation and its corresponding UI state form a tuple. This masking operation space consists of one or more tuples. Each time a UI operation is completed on the AUT, the memory module can update the masking operation space. This application embodiment uses a domain knowledge loader to integrate domain knowledge, converting it into algorithms for an action space optimizer, validator, and error handling program. The optimizer primarily optimizes the action space O based on domain knowledge, the masking operation space, and the current UI state, removing masking operations from the action space to obtain the optimized action space O'.

[0059] In one possible implementation, the domain knowledge loader includes: an optimizer;

[0060] The optimizer is used to parse the list of available UI operations of the current AUT interface according to the current UI state, and generate the action space, which represents the set of all UI operations that can be executed on the current AUT interface.

[0061] The optimizer optimizes the action space based on the masked operation space obtained from the memory module, and deletes the masked operations in the action space to obtain the optimized action space.

[0062] Specifically, in each iteration of feature-based automated UI testing, the LLM selects a UI operation from the action space (represented as a set of UI operations). The action space includes the current UI state, i.e., all possible UI operations under the current AUT's user interface. In this embodiment, the optimizer, based on relevant domain knowledge of UI testing, parses and obtains a list of available UI operations for the current AUT interface according to the current UI state. Thus, the action space is generated based on this list of available UI operations. Then, the optimizer retrieves the current masked operation space B from the memory module, optimizes the action space O, removes masked operations from the action space, obtains the optimized action space, and converts the specific instructions of the UI test into the LLM's planning task (i.e., selecting a UI operation from the optimized action space). For example, if the current UI state is state A, the action space includes UI operation 1 (long press) and UI operation 2 (click), and the masked operation space B includes the tuple "UI operation 1 (long press), state A", then when optimizing the action space, UI operation 1 (long press) needs to be removed from the action space. By taking the steps described above, offloading such computational tasks (removing invalid UI operations from the action space) from the LLM to an external system (user interface testing navigation system) can reduce the complexity of the task and the length of the context, thereby avoiding the theoretical limitations caused by computational complexity and the empirical limitations caused by multiple instruction traces and long context processing.

[0063] Step 2: Based on the optimized action space O' and the historical test information obtained from the memory module, generate operation prompts for LLM.

[0064] In one possible implementation, the execution engine includes: a parser, an LLM driver, and a UI driver;

[0065] The parser generates operation prompts based on the optimized action space and the historical test information obtained from the memory module; the operation prompts include: the optimized action space, the test target, and the historical UI operation sequence.

[0066] Specifically, the parser in the execution engine generates operation prompt text for the LLM. This prompt includes: a description of the instruction (corresponding to the test objective, such as turning on night mode), a prompt for the optimized action space, and UI operations already executed in previous rounds of UI testing (corresponding to the historical UI operation sequence). Within the operation prompt, the optimized action space can include identifiers for each UI operation, each uniquely identified using text attributes obtained from the UI hierarchy's XML file. In the description of the optimized action space within the operation prompt, each UI operation's description includes: event type, resource ID, text representation, and accessibility information (retrieved by the UI Auto-Manager).

[0067] The historical UI operation sequence can be obtained from the memory module. Besides maintaining the masked operation space, the memory module also stores historical test information, including: historical test information obtained from each round of UI testing (i.e., performing one UI operation on the AUT). The historical UI operation sequence is obtained from the historical test information and placed into the operation prompt text. Specifically, the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; effective UI operations represent the UI operations performed to achieve the expected results in the corresponding round of UI testing (i.e., the UI test actions verified and passed in step 5); effective UI states represent the UI states obtained to achieve the expected results in the corresponding round of UI testing (i.e., the UI states after testing); each effective UI operation corresponds one-to-one with the effective UI states.

[0068] In addition, the memory module also stores a historical dataset, each piece of historical data in the historical dataset including: activity name, UI hierarchy, UI operation planned by the first LLM for the UI hierarchy, and the UI state changed after the UI operation is executed on the AUT.

[0069] Considering that LLM performs poorly in following multiple or fine-grained instructions, and it is difficult to improve the accuracy of LLM in planning UI operations by directly modifying the prompt text, the embodiments of this application optimize the action space, thereby using the optimized action space to generate operation prompts for inputting LLM, and deleting disallowed operations from the prompt text, thereby avoiding excessive modification of the text content of the operation prompts, which helps LLM understand and follow the instructions.

[0070] Step 3: Input the operation prompt into the first LLM, and the LLM will respond according to the operation prompt to obtain the response result.

[0071] In this embodiment, after the parser generates the operation prompt, the LLM driver is used to send the operation prompt to the first LLM, and the response result obtained by the first LLM based on the operation prompt is received; the response result is the identifier of the UI operation that needs to be performed in the current UI state in this round of testing, as determined by the first LLM.

[0072] Specifically, such as Figure 1 As shown, the execution engine also includes an LLM driver. The LLM driver is responsible for the interaction between the user interface test navigation system and the LLM, including: sending operation prompts to the LLM (which can be the first LLM in this embodiment), and receiving the response result output by the first LLM based on the operation prompts. The response result can be the identifier corresponding to the UI operation that the first LLM should perform in the current UI state in this round of UI testing.

[0073] Step 4: By parsing the response results, obtain the UI operation that should be executed in this round of UI testing (i.e., the UI test action in this round), execute the UI operation on the AUT, so that the AUT changes from the current UI state to the UI state after the test.

[0074] In this embodiment, the parser parses the response result to obtain the UI test action for this round; the UI test action for this round includes one or more of the following: clicking, swiping, long pressing, and inputting text; using the UI driver, the UI test action for this round is executed on the AUT to obtain the UI state after the test.

[0075] Specifically, the parser analyzes the response result and, based on the UI operation identifier within the response, derives an executable UI operation (the UI test action for this round). The parser coordinates with the memory module to manage the instruction flow and ensures their correct application within the optimized action space. For example... Figure 1 As shown, the execution engine also includes a UI driver, which is responsible for the interaction between the user interface test navigation system and the AUT, including: performing UI operations on the AUT, and dumping the UI hierarchy file representing the AUT screen, UI operation-related data, such as the UI state after execution.

[0076] The UI operations that can be performed on the AUT are mainly divided into four categories: click, swipe, long press, and text input. Clicking means clicking the center point of a given UI element. Long press means pressing and holding the center point of a given UI element for a specified duration. This duration can be set according to actual application needs, such as 1 second. Swipe means performing a swipe operation based on a given swipe direction and distance. Text input means generating an appropriate string input and then inputting it into a given UI element.

[0077] Step 5: Based on the UI status after the test, verify whether this round of UI testing has achieved the expected results.

[0078] In this embodiment, after the planned UI operation is executed on the AUT, the verifier in the domain knowledge loader verifies whether the AUT has achieved the expected effect based on the relevant domain knowledge of UI testing. For example, if the expected effect is that the AUT turns on night mode, then whether the UI state after the test is in night mode can be used to determine whether this round of UI testing has achieved the expected effect.

[0079] In one possible implementation, the domain knowledge loader includes: a verifier; the step of verifying whether the current round of UI testing achieved the expected results based on the post-test UI state includes:

[0080] The verifier sends the response result of the first LLM, the current UI state, and the UI state after the test as a triple to the second LLM, so that the second LLM can determine whether the UI state after the test is consistent with the planned test state and receive the verification result of the second LLM.

[0081] The verifier, based on heuristic rules, analyzes the UI state after the test to determine whether the UI test achieved the expected results.

[0082] This application primarily employs two verification methods: LLM feedback and heuristic rules. Specifically, after executing the current round of UI testing, the LLM is consulted based on the initial plan (including operation prompts and LLM response results) to determine whether the result matches the expected result. The response result of the first LLM, the current UI state, and the UI state after testing (both can be screenshots) are sent as a triple to the second LLM (in this embodiment, the second LLM can be the same model as the first LLM or a different model). In addition, task-related description information can also be sent to the second LLM so that the second LLM can determine whether the UI state after testing is consistent with the planned test state. If the response from the LLM (second LLM) is "No", it indicates that the UI operation is invalid. In this case, the current round of UI testing is treated as a masked operation, the current UI state is used as the UI state corresponding to the masked operation, and the masked operation space is updated. In this embodiment, the first LLM or the second LLM can refer to any large language model capable of implementing UI operation planning, or a cluster of multiple large language models. On the other hand, the outcome of an action is evaluated in response to known exploration traps in the heuristic rules. That is, if a UI action leads to any one or more of the following three situations: unresponsiveness, repeated exploration, and exploration loop, then the UI action is considered invalid.

[0083] In one possible implementation, the step of analyzing the post-test UI state based on heuristic rules to determine whether the current round of UI testing achieved the expected results includes:

[0084] Based on the UI state after the test, it is determined whether the same UI state has already appeared in previous test rounds. Specifically, the heuristic rule is represented as a detection program for duplicate states. This detection program is executed to check the UI state after the test. If the detection determines that the UI state after the test is the same as the UI state that has already appeared in previous test rounds, it indicates that a duplicate exploration has occurred. This round of UI testing has not achieved the expected results, and the action of this round of UI testing is determined as a blocking operation, which is used by the memory module to update the blocking operation space.

[0085] Determine whether the UI state after the test is an unresponsive UI state. Specifically, the heuristic rule is represented as a detection program for unresponsiveness. This detection program is executed to detect the UI state after the test. If the detected UI state after the test is an unresponsive UI state, it means that this round of UI testing has not achieved the expected effect. The UI test action in this round is determined as a blocking operation, which is used by the memory module to update the blocking operation space.

[0086] Based on the UI state after the test, it is determined whether a test state loop has occurred. Specifically, the heuristic rule is represented as a detection program for test state loops. This detection program is executed to detect the UI state after the test. If the detection determines that a test state loop has occurred (the UI state repeatedly appears after the test), it means that this round of UI testing has not achieved the expected results. The UI test action in this round is then determined as a blocking operation, which is used by the memory module to update the blocking operation space.

[0087] Step 6: If the expected results are achieved, update the UI state after the test to the current UI state and repeat steps 1-5 to conduct the next round of UI testing; if the expected results are not achieved, restore the UI state to the current UI state (i.e., the state before the invalid UI operation was performed) and re-perform the UI test (i.e., re-execute steps 1-5).

[0088] In one possible implementation, the domain knowledge loader includes: an error handling program; the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operations represent the UI operations executed to achieve the expected results in the corresponding round of UI testing (i.e., the UI test actions that passed verification in step 5), and the effective UI states represent the UI states obtained to achieve the expected results in the corresponding round of UI testing (i.e., the UI states after testing); the effective UI operations and the effective UI states correspond one-to-one.

[0089] If the expected effect is not achieved, the error handling procedure is used to perform UI state rollback, detect the position of the rolled-back UI state in the historical UI state sequence, and re-execute the corresponding effective UI operation according to the historical UI operation sequence to restore the current UI state.

[0090] Specifically, if verification fails (e.g., the verification result of the second LLM is NO), a "back" action is triggered on the AUT to restore the state before the UI operation was performed. For example, during UI testing, after performing the nth UI operation, the UI state after the nth test is obtained. If verification confirms that the nth UI operation was invalid (i.e., the UI state after the nth test cannot achieve the expected effect), the error handling procedure is activated to perform UI state rollback (e.g., triggering a "back" action on the AUT). The rolled-back UI state needs to be determined based on the actual application situation. The rolled-back UI state may be the UI state after the kth UI operation (k is less than n-1), or it may be a direct rollback to the UI state after the (n-1)th UI operation. When the rolled-back UI state is the UI state after the (n-1)th UI operation, it indicates that the current UI state has been restored, and steps 1-5 can be re-executed. If the reverted UI state is the UI state after the k-th UI operation (k is less than n-1), then it is also necessary to check the position of the reverted UI state in the historical UI state sequence (located at the k-th position), and according to the historical UI operation sequence, replay the (k+1)-(n-1)-th UI operations to restore the current UI state. Once the UI state is restored, the next iteration is started, that is, steps 1-5 are re-executed, so that LLM re-plans the UI operations that should be executed in this round of testing based on the optimized action space (excluding previously invalid UI operations), thereby avoiding selecting invalid UI operations again.

[0091] In one possible implementation, if the expected effect is not achieved, the memory module treats the current round of UI testing action as a blocking operation, takes the current UI state as the UI state corresponding to the blocking operation, and updates the blocking operation space.

[0092] If the expected results are achieved, the memory module updates the historical test information, adds the current round of UI test actions as the effective UI operations to the historical UI operation sequence, and adds the post-test UI state as the effective UI state to the historical UI state sequence.

[0093] This application's embodiments, on one hand, optimize the action space (the set of selectable UI operations for the current UI state) using a domain knowledge loader, removing invalid UI operations (i.e., masked operations, meaning UI operations that did not achieve the expected results after execution in the corresponding UI state) from the action space. On the other hand, through the domain knowledge loader, after each UI operation is completed, the UI state after the test is evaluated to verify whether the current round of UI testing achieved the expected results. If the expected results were not achieved, the UI state is restored to the state before the invalid operation was executed (i.e., the current UI state). This allows the LLM to redetermine the UI operations to be executed in the optimized action space, avoiding the LLM repeatedly selecting incorrect UI operations. This enables the LLM to re-plan UI operation sequences in a timely manner based on current information when errors occur in the currently planned UI operations, generating reliable and accurate UI operation sequences for the test target, thus improving the effectiveness of automated UI testing based on LLM.

[0094] A second aspect of this application also provides a user interface testing navigation method, applied to the user interface testing navigation system described in the first aspect of this application, with reference to... Figure 3 , Figure 3 A flowchart illustrating the steps of a user interface testing navigation method is shown, such as... Figure 3 As shown, the method includes:

[0095] Step S101: Optimize the action space based on the masked operation space and the current UI state by deleting masked operations in the action space to obtain the optimized action space; the masked operation space includes one or more masked operations, which represent UI operations that do not achieve the expected effect after execution in the corresponding UI state;

[0096] Step S102: Based on the optimized action space, generate operation prompts so that the first large language model LLM can determine the UI test action for this round based on the operation prompts;

[0097] Step S103: Obtain the UI state after the test by performing the UI test action in this round on the test application AUT;

[0098] Step S104: Based on the UI state after the test, verify whether the UI test in this round has achieved the expected effect. If the expected effect has not been achieved, restore the UI state to the current UI state and retest the UI.

[0099] In one possible implementation, optimizing the action space based on the masked operation space and the current UI state, and deleting masked operations from the action space to obtain an optimized action space, includes:

[0100] Based on the current UI state, the list of available UI operations for the current AUT interface is parsed to generate the action space, which represents the set of all UI operations that can be performed on the current AUT interface.

[0101] Based on the masking operation space obtained from the memory module, the action space is optimized by deleting the masking operations in the action space to obtain the optimized action space.

[0102] In one possible implementation, based on the optimized action space, operation prompts are generated, enabling the first large language model (LLM) to determine the UI test actions for this round based on the operation prompts, including:

[0103] Based on the optimized action space and the obtained historical test information, an operation prompt is generated; the operation prompt includes: the optimized action space, the test target, and the historical UI operation sequence;

[0104] The operation prompt is sent to the first LLM, and the response result obtained by the first LLM based on the operation prompt is received; the response result is the identifier of the UI operation that needs to be performed in this round of testing under the current UI state, as determined by the first LLM.

[0105] The response result is parsed to obtain the UI test action for this round; the UI test action for this round includes one or more of the following: click, swipe, long press, and text input.

[0106] Perform the UI test actions described in this round on the AUT to obtain the UI state after the test.

[0107] In one possible implementation, verifying whether the current round of UI testing achieved the expected results based on the UI state after the test includes:

[0108] The response result of the first LLM, the current UI state, and the UI state after the test are sent as a triple to the second LLM, so that the second LLM can determine whether the UI state after the test is consistent with the planned test state and receive the verification result of the second LLM.

[0109] Based on heuristic rules, the UI state after the test is analyzed to determine whether the UI test achieved the expected results.

[0110] In one possible implementation, the step of analyzing the post-test UI state based on heuristic rules to determine whether the current round of UI testing achieved the expected results includes:

[0111] Based on the UI state after the test, determine whether the same UI state has appeared in previous test rounds;

[0112] Determine whether the UI state after the test is an unresponsive UI state;

[0113] Based on the UI state after the test, determine whether a test state loop has occurred.

[0114] In one possible implementation, the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operations represent the UI operations executed to achieve the expected results in the corresponding round of UI testing, and the effective UI states represent the UI states obtained to achieve the expected results in the corresponding round of UI testing; the effective UI operations and the effective UI states correspond one-to-one.

[0115] The method further includes:

[0116] If the expected effect is not achieved, a UI state rollback is performed to detect the position of the rolled-back UI state in the historical UI state sequence, and according to the historical UI operation sequence, the corresponding effective UI operation is re-executed to restore the current UI state.

[0117] In one possible implementation, the method further includes:

[0118] If the expected results are not achieved, the UI test action in this round will be treated as a blocking operation, the current UI state will be taken as the UI state corresponding to the blocking operation, and the blocking operation space will be updated.

[0119] If the expected results are achieved, the historical test information is updated, the UI test action of this round is added to the historical UI operation sequence as the effective UI operation, and the UI state after the test is added to the historical UI state sequence as the effective UI state.

[0120] This application also provides an electronic device, which is described in reference to... Figure 4 , Figure 4 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 4 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the user interface testing navigation method disclosed in the embodiments of this application.

[0121] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the user interface test navigation method disclosed in this application.

[0122] This application also provides a computer program product that, when run on an electronic device, causes a processor to execute the steps of the user interface testing navigation method disclosed in this application.

[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0124] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0128] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0129] The foregoing has provided a detailed description of the user interface testing navigation system, method, and product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A user interface test navigation system, characterized by, The system includes: A memory module is used to maintain the masked operation space and historical test information; the masked operation space includes one or more masked operations, which represent UI operations that did not achieve the expected effect after being executed in the corresponding UI state; A domain knowledge loader is used to optimize the action space based on the shielded operation space and the current UI state, delete the shielded operations in the action space, and obtain the optimized action space. The execution engine is used to generate operation prompts based on the optimized action space, enabling the first large language model LLM to determine the UI test action for this round based on the operation prompts; and to obtain the UI state after the test is performed on the test application AUT. The domain knowledge loader is also used to verify whether the UI test has achieved the expected effect based on the UI state after the test. If the expected effect has not been achieved, the UI state is restored to the current UI state, and the UI test is performed again. The current UI state is the state before the masking operation in the action space is executed. The domain knowledge loader includes: a verifier; The validator, based on heuristic rules, analyzes the post-test UI state to determine whether the UI test achieved the expected results, including: Based on the UI state after the test, determine whether the same UI state has appeared in previous test rounds; Determine whether the UI state after the test is an unresponsive UI state; Based on the UI state after the test, determine whether a test state loop occurs.

2. The user interface testing navigation system according to claim 1, characterized in that, The domain knowledge loader includes: an optimizer; The optimizer is used to parse the list of available UI operations of the current AUT interface according to the current UI state and generate the action space, wherein the action space represents the set of all UI operations that can be performed on the current AUT interface. The optimizer optimizes the action space based on the masked operation space obtained from the memory module, and deletes the masked operations in the action space to obtain the optimized action space.

3. The user interface testing navigation system according to claim 2, characterized in that, The execution engine includes: a parser, an LLM driver, and a UI driver; The parser generates operation prompts based on the optimized action space and the historical test information obtained from the memory module; the operation prompts include: the optimized action space, the test target, and the historical UI operation sequence; Using the LLM driver, the operation prompt is sent to the first LLM, and the response result obtained by the first LLM based on the operation prompt is received; the response result is the identifier of the UI operation that needs to be performed in the current UI state in this round of testing, as determined by the first LLM. The parser analyzes the response result to obtain the UI test action for this round; the UI test action for this round includes one or more of the following: clicking, swiping, long pressing, and inputting text; Using the UI driver, perform the current round of UI testing actions on the AUT to obtain the UI state after the test.

4. The user interface testing navigation system according to claim 1, characterized in that, The step of verifying whether the UI test achieved the expected results based on the UI state after the test includes: The verifier sends the response result of the first LLM, the current UI state, and the UI state after the test as a triple to the second LLM, so that the second LLM can determine whether the UI state after the test is consistent with the planned test state and receive the verification result of the second LLM.

5. The user interface testing navigation system according to claim 1, characterized in that, The domain knowledge loader includes an error handling program; the historical test information includes a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operations represent the UI operations executed to achieve the expected results in the corresponding round of UI testing, and the effective UI states represent the UI states obtained to achieve the expected results in the corresponding round of UI testing; there is a one-to-one correspondence between the effective UI operations and the effective UI states; If the expected effect is not achieved, the error handling procedure is used to perform UI state rollback, detect the position of the rolled-back UI state in the historical UI state sequence, and re-execute the corresponding effective UI operation according to the historical UI operation sequence to restore the current UI state.

6. The user interface testing navigation system according to claim 5, characterized in that, If the expected results are not achieved, the memory module will treat the current round of UI test actions as a blocking operation, take the current UI state as the UI state corresponding to the blocking operation, and update the blocking operation space. If the expected results are achieved, the memory module updates the historical test information, adds the current round of UI test actions as the effective UI operations to the historical UI operation sequence, and adds the post-test UI state as the effective UI state to the historical UI state sequence.

7. A user interface testing navigation method, characterized in that, The method, applied to the user interface testing navigation system according to any one of claims 1-6, comprises: Based on the masked operation space and the current UI state, the action space is optimized by deleting masked operations in the action space to obtain the optimized action space; the masked operation space includes one or more masked operations, which represent UI operations that do not achieve the expected effect after execution in the corresponding UI state; Based on the optimized action space, operation prompts are generated, enabling the first large language model LLM to determine the UI test action for this round based on the operation prompts; Obtain the UI state after the current round of UI test actions are performed on the test application AUT; Based on the UI state after the test, verify whether the UI test has achieved the expected results. If the expected results have not been achieved, restore the UI state to the current UI state and retest the UI. The current UI state is the state before the masking operation in the action space was executed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the user interface test navigation method of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the user interface test navigation method of claim 7.

Images