User interface test navigation system, method and product

Abstract

The invention provides a user interface test navigation system, method and product, and relates to the technical field of user interface test.The system comprises a memory module used for maintaining a shielding operation space; the shielding operation represents the UI operation which does not achieve the expected effect after being executed in the corresponding UI state; the domain knowledge loader is used for deleting the shielding operation in the action space according to the shielding operation space and the current UI state to obtain an optimized action space; the execution engine is used for generating an operation prompt according to the optimized action space, so that the first large language model LLM determines the current UI test action according to the operation prompt; obtaining a UI state after the test, which is obtained by executing the current round of UI test action on the test application program AUT; and the domain knowledge loader verifies whether the current UI test achieves an expected effect or not according to the UI state after the test, and under the condition that the expected effect is not achieved, the UI state is recovered to the current UI state, and the UI test is performed again.

Description

Technical Field

[0001] The present application relates to the technical field of user interface testing, and in particular to a user interface testing navigation system, method and product. Background Art

[0002] User interface (UI) testing is essential to ensure the high quality of the App under Test (AUT). In order to reduce the high manual labor cost of creating such UI tests, the related art proposes a method for automatically exploring the test application based on the test goal description to obtain a UI operation sequence that can automatically generate the test goal (i.e., feature-based UI automatic testing). Considering that the feature-based UI automatic testing task is similar to the traditional artificial intelligence planning problem, in the related art, the large language model (LLM) known for its effectiveness in artificial intelligence planning is mainly used to perform it. The test goal and the related prompt text are input into the LLM, and the UI operation sequence is generated by the LLM, so that the UI automatic testing of the AUT (application to be tested) is realized by automatically executing the UI operation sequence on the AUT.

[0003] However, tasks such as UI testing (i.e., feature-based UI automatic testing) are sequential planning problems that require LLM to perform multi-step reasoning. However, the reasoning performance of LLM decreases as the complexity of the task increases, resulting in invalid UI operations in the inferred UI operation sequence, which in turn makes it impossible to achieve the expected test results when executing the UI operation sequence on the AUT. In addition, LLM is less effective in following multiple or fine-grained instructions, and it is difficult to improve the accuracy of LLM in planning UI operations directly by modifying the prompt text. Therefore, it is urgent to propose a user interface test navigation system, method, and product to improve the effectiveness of automatic UI testing based on LLM. Summary of the invention

[0004] In view of the above problems, embodiments of the present application provide a user interface test navigation system, method and product to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a user interface test navigation system is provided, the system comprising:

[0006] A memory module, used to maintain a shielded operation space and historical test information; the shielded operation space includes one or more shielded operations, and the shielded operation represents a UI operation that does not achieve an expected effect after being executed in a corresponding UI state;

[0007] A domain knowledge loader, used to optimize the action space according to the shielded operation space and the current UI state, delete the shielded operation in the action space, and obtain an optimized action space;

[0008] The execution engine is used to generate an operation prompt according to the optimized action space, so that the first large language model LLM determines the current round of UI test actions according to the operation prompt; and obtains the post-test UI state obtained by executing the current round of UI test actions on the test application AUT;

[0009] The domain knowledge loader is also used to verify whether the current round of UI testing achieves the expected effect based on the post-test UI state. If the expected effect is not achieved, the UI state is restored to the current UI state and the UI test is performed again.

[0010] In a possible implementation, the domain knowledge loader includes: an optimizer;

[0011] The optimizer is used to parse and obtain a list of available UI operations of the current AUT interface according to the current UI state, and generate the action space, wherein the action space represents a set of all UI operations that can be performed on the current AUT interface;

[0012] The optimizer optimizes the action space according to the masked operation space obtained from the memory module, deletes the masked operations in the action space, and obtains the optimized action space.

[0013] In a possible implementation, the execution engine includes: a parser, an LLM driver, and a UI driver;

[0014] The parser generates an operation prompt according to the optimized action space and the historical test information obtained from the memory module; the operation prompt includes: the optimized action space, the test target, and the historical UI operation sequence;

[0015] Using the LLM driver, sending the operation prompt to the first LLM, and receiving a response result obtained by the first LLM according to the operation prompt response; the response result is an identifier of the UI operation to be performed in the current UI state in this round of testing determined by the first LLM;

[0016] The parser parses the response result to obtain the UI test action of this round; the UI test action of this round includes: one or more of clicking, sliding, long pressing, and inputting text;

[0017] The UI driver is used to execute the current round of UI test actions on the AUT to obtain the post-test UI state.

[0018] In a possible implementation, the domain knowledge loader includes: a verifier; the verifying whether the current round of UI testing achieves the expected effect according to the post-test UI state includes:

[0019] The verifier sends the response result of the first LLM, the current UI state and the post-test UI state as a triplet to the second LLM, so that the second LLM determines whether the post-test UI state is consistent with the planned test state, and receives the verification result of the second LLM;

[0020] The verifier analyzes the post-test UI status based on heuristic rules to determine whether this round of UI testing achieves the expected effect.

[0021] In a possible implementation, analyzing the post-test UI state based on heuristic rules to determine whether the current round of UI testing achieves the expected effect includes:

[0022] Determine, based on the post-test UI state, whether the same UI state has appeared in a previous test round;

[0023] Determine whether the post-test UI state is an unresponsive UI state;

[0024] According to the post-test UI state, it is determined whether a test state loop occurs.

[0025] In a possible implementation, the domain knowledge loader includes: an error handling program; the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operation represents the UI operation performed to achieve the expected effect of the corresponding round of UI test, and the effective UI state represents the UI state obtained when the corresponding round of UI test achieves the expected effect; the effective UI operation corresponds to the effective UI state one by one;

[0026] If the expected effect is not achieved, the error handler is used to perform UI state backtracking, detect the position of the UI state after rollback in the historical UI state sequence, and re-execute the corresponding effective UI operation according to the historical UI operation sequence to restore to the current UI state.

[0027] In a possible implementation manner, when the expected effect is not achieved, the memory module uses the current round of UI test action as a shielding operation, uses the current UI state as the UI state corresponding to the shielding operation, and updates the shielding operation space;

[0028] When the expected effect is achieved, the memory module updates the historical test information, adds the current round of UI test actions as the effective UI operations to the historical UI operation sequence, and adds the post-test UI state as the effective UI state to the historical UI state sequence.

[0029] The second aspect of the embodiment of the present application further provides a user interface test navigation method, which is applied to the user interface test navigation system described in the first aspect of the embodiment of the present application, and the method includes:

[0030] According to the shielded operation space and the current UI state, the action space is optimized, and the shielded operations in the action space are deleted to obtain an optimized action space; the shielded operation space includes one or more shielded operations, and the shielded operation represents a UI operation that does not achieve an expected effect after being executed in the corresponding UI state;

[0031] Generate an operation prompt according to the optimized action space, so that the first large language model LLM determines the UI test action of this round according to the operation prompt;

[0032] Obtaining a post-test UI state obtained by executing the current round of UI test actions on the test application AUT;

[0033] According to the post-test UI state, verify whether this round of UI testing achieves the expected effect. If the expected effect is not achieved, restore the UI state to the current UI state and re-perform the UI test.

[0034] The third aspect of the embodiment of the present application also provides an electronic device, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps in the user interface test navigation method described in the second aspect of the embodiment of the present application.

[0035] The fifth aspect of the embodiment of the present application further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the user interface test navigation method described in the second aspect of the embodiment of the present application are implemented.

[0036] A sixth aspect of the embodiments of the present application further provides a computer program product, which, when executed on an electronic device, enables a processor to implement the steps in the user interface test navigation method as described in the second aspect of the embodiments of the present application.

[0037] An embodiment of the present application provides a user interface test navigation system, the system comprising: a memory module, for maintaining a shielded operation space and historical test information; the shielded operation space comprises one or more shielded operations, the shielded operation representing a UI operation that does not achieve the expected effect after being executed in a corresponding UI state; a domain knowledge loader, for optimizing the action space according to the shielded operation space and the current UI state, deleting the shielded operations in the action space, and obtaining an optimized action space; an execution engine, for generating an operation prompt according to the optimized action space, so that a first large language model LLM determines the UI test action of this round according to the operation prompt; obtaining a post-test UI state obtained by executing the UI test action of this round on a test application AUT; the domain knowledge loader is also used to verify whether the expected effect of this round of UI testing is achieved according to the post-test UI state, and if the expected effect is not achieved, restore the UI state to the current UI state and re-perform the UI test.

[0038] The specific beneficial effects are:

[0039] During the UI testing process, a series of UI operations need to be performed. On the one hand, the embodiment of the present application uses a domain knowledge loader to optimize the action space (a set of selectable UI operations for the current UI state) and deletes invalid UI operations (i.e., shielded operations, which indicate UI operations that do not achieve the expected effect after being executed in the corresponding UI state) during the test from the action space. On the other hand, through the domain knowledge loader, after each UI operation is completed, it is promptly verified whether the current round of UI testing has achieved the expected effect based on the UI state after the test. If the expected effect is not achieved, the UI state is restored to the state before the invalid operation is performed (i.e., the current UI state), so that LLM can re-determine the UI operation that should be performed in the optimized action space, avoiding LLM from repeatedly selecting the wrong UI operation, and realizing that in the process of LLM planning the UI operation sequence, in the event of an error in the currently planned UI operation, timely re-planning is performed based on the current information to generate a reliable and accurate UI operation sequence for the test target, thereby improving the effectiveness of automatic UI testing based on LLM. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 It is a structural diagram of a user interface test navigation system provided in an embodiment of the present application;

[0042] Figure 2 It is a flowchart of a UI operation based on LLM planning provided by an embodiment of the present application;

[0043] Figure 3 It is a flowchart of the steps of a user interface test navigation method provided by an embodiment of the present application;

[0044] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0046] User interface (UI) testing is essential to ensure the high quality of the App under Test (AUT). In order to reduce the high manual labor cost of creating such UI tests, the related art proposes a method for automatically exploring the core functions of the test application based on the test target description, so as to obtain a UI operation sequence that can automatically generate the test target (i.e., feature-based automatic UI testing). The method can be specifically expressed as follows: given an AUT (application to be tested) and a test target, select a UI operation on the UI screen in turn, trigger the UI operation on the AUT, and select the next UI operation on the new UI screen, and perform a series of UI operations until the test target is achieved. Therefore, feature-based automatic UI testing is essentially a sequential planning problem.

[0047] Considering that the automatic feature-based UI testing task is similar to the traditional artificial intelligence planning problem, in the related art, the large language model (LLM) known for its effectiveness in artificial intelligence planning is mainly used to perform it, so that the LLM generates a sequence of UI operations that can be executed on the AUT according to the input test goal. Although LLM has good effectiveness for sequential planning problems, according to empirical investigations, the effectiveness of LLM-based UI testing methods is low, and LLM is not good at formulating planning strategies specific to such exploration tasks (i.e., feature-based UI testing). Specifically, on the one hand, it is difficult for LLM to follow specific UI test instructions. LLM lacks domain knowledge in UI exploration, and existing methods incorporate instructions specific to UI testing into prompts to provide assistance. However, LLM is difficult to execute according to task instructions in specific fields, resulting in low effectiveness. Exemplarily, survey data show that although drobotGPT uses clear instruction prompts to avoid selecting actions that have already been selected, 36% of the planned UI operations are just repeated historical UI operations. On the other hand, in the case of an error in the currently planned UI operation, it is difficult for LLM to re-plan based on new information. Exemplarily, on the homepage of the AUT, the LLM selects the UI action of "clicking the search button" and proceeds to set up a shortcut for the night mode (the test goal is: turn on the night mode on the AUT). When it is discovered that the search tab cannot achieve its test purpose, the LLM is unable to re-plan the UI action using the new information and repeatedly clicks the search tab. The task of automated UI testing involves highly dynamic, real-time exploration of the AUT, because the same functionality can be implemented in different ways in different applications. As the exploration progresses, newly discovered information may invalidate previously planned UI actions. In tasks like UI testing, multi-step reasoning is critical, and the performance of the LLM may decline as the complexity of the task increases. In addition, experience has shown that the LLM is less effective in following multiple or fine-grained instructions. In scenarios involving long texts, the tendency for the model's ability to accurately follow instructions to decline is more pronounced.

[0048] In view of the above problems, the embodiments of the present application propose a user interface test navigation system, method and product to improve the effectiveness of automatic UI testing based on LLM. In conjunction with the accompanying drawings, a user interface test navigation system provided by the embodiments of the present application is described in detail through some embodiments and their application scenarios.

[0049] The first aspect of the present application embodiment provides a user interface test navigation system, referring to Figure 1 , Figure 1 A structural diagram of a user interface test navigation system provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the system comprises:

[0050] The memory module is used to maintain a shielded operation space and historical test information; the shielded operation space includes one or more shielded operations, and the shielded operation represents a UI operation that does not achieve an expected effect after being executed in a corresponding UI state.

[0051] The domain knowledge loader is used to optimize the action space according to the shielded operation space and the current UI state, delete the shielded operations in the action space, and obtain the optimized action space.

[0052] The execution engine is used to generate operation prompts according to the optimized action space, so that the first large language model LLM determines the UI test action of this round according to the operation prompts; and obtains the post-test UI state obtained by executing the UI test action of this round on the test application AUT.

[0053] The domain knowledge loader is also used to verify whether the current round of UI testing achieves the expected effect based on the post-test UI state. If the expected effect is not achieved, the UI state is restored to the current UI state and the UI test is performed again.

[0054] In order to achieve automatic UI testing, the test objectives and related prompt texts are often directly input into LLM, which generates a UI operation sequence, and then automatically executes the UI operation sequence on the AUT (application to be tested) to achieve automatic UI testing of the AUT. However, tasks such as UI testing belong to sequential planning problems, which require LLM to perform multi-step reasoning. The reasoning performance of LLM may decrease as the complexity of the task increases, resulting in invalid UI operations in the inferred UI operation sequence, which makes it impossible to achieve the expected test effect when executing the UI operation sequence on the AUT. Considering that LLM is less effective in following multiple or fine-grained instructions, it is difficult to improve the accuracy of LLM in planning UI operations directly by modifying the prompt text. Therefore, an embodiment of the present application provides a user interface test navigation system, which belongs to a runtime system and is used to verify whether each step of the UI operation achieves the expected effect in a timely manner when LLM plans the UI operations to be performed for the test target. If the expected effect is not achieved, the action space is optimized and the invalid UI operations (i.e., shielding operations) therein are deleted, thereby avoiding LLM from repeatedly selecting the same invalid UI operation; and, by restoring the UI state to the state before executing the invalid UI operation, LLM can re-plan the next step of UI operation based on the previously planned correct UI operation sequence. Therefore, there is no need to make too many changes to the prompt text input to LLM, which not only improves the accuracy of the planned UI operation sequence, but also ensures the reasoning efficiency of LLM.

[0055] In addition, it is necessary to know that the user interface test navigation system proposed in the embodiment of the present application can not only be applied to UI test, but also can be applied to the application scenario of user interface navigation (for example, as a user interface navigation assistant). In the application scenario of user interface navigation, the functions performed by each module of the user interface test navigation system are basically the same. Specifically, for a given user interface navigation target (equivalent to the test target in the UI test scenario), the user interface test navigation system optimizes the action space by shielding the operation space through the domain knowledge loader to obtain the optimized action space. Then by the execution engine, according to the optimized action space, an operation prompt is generated, so that the first large language model LLM determines this round of UI operation according to the operation prompt, thereby performing this round of UI operation on the target application, and obtaining the UI state after operation (equivalent to the UI state after the test in the UI test scenario). Then by the domain knowledge loader according to the UI state after operation, it is verified whether the execution of this round of UI operation achieves the expected effect. If the expected effect is not achieved, the UI state is restored to the current UI state, and the user interface navigation of this round is re-performed.

[0056] Reference Figure 2 , Figure 2 A schematic diagram of a process flow based on LLM planning UI operation is shown. The information processing flow of the user interface test navigation system is as follows:

[0057] Step 1: The user interface test navigation system obtains the current UI state from the AUT, and then uses the domain knowledge loader to optimize the action space according to the masked operation space and the current UI state, deletes the masked operations in the action space, and obtains the optimized action space.

[0058] Specifically, Figure 2 As shown, the user interface test navigation system obtains the current UI state from the application to be tested AUT, and the current UI state includes the current user interface image. The user interface test navigation system includes a memory module, and the memory module maintains a shielding operation space (such as Figure 2 B) shown in , wherein the shielding operation represents a UI operation that does not achieve the expected effect after being executed in the corresponding UI state, the shielding operation and the corresponding UI state form a tuple, and the shielding operation space consists of one or more tuples. Each time a UI operation is completed on the AUT, the memory module can update the shielding operation space. The embodiment of the present application uses a domain knowledge loader to integrate domain knowledge and convert the domain knowledge into algorithms of an action space optimizer, a validator, and an error handler, wherein the optimizer mainly optimizes the action space O based on the domain knowledge, according to the shielding operation space, and the current UI state, deletes the shielding operation in the action space, and obtains the optimized action space O'.

[0059] In a possible implementation, the domain knowledge loader includes: an optimizer;

[0060] The optimizer is used to parse and obtain a list of available UI operations of the current AUT interface according to the current UI state, and generate the action space, wherein the action space represents a set of all UI operations that can be executed in the current AUT interface.

[0061] The optimizer optimizes the action space according to the masked operation space obtained from the memory module, deletes the masked operations in the action space, and obtains the optimized action space.

[0062] Specifically, in each iteration of feature-based automatic UI testing, LLM selects a UI operation from the action space (represented as a set of UI operations). The action space includes the current UI state, that is, all possible UI operations under the user interface of the current AUT. In this embodiment, the optimizer parses and obtains a list of available UI operations of the current AUT interface based on the relevant domain knowledge of UI testing and the current UI state. Thus, the action space is generated according to the list of available UI operations. Then, the optimizer obtains the current shielding operation space B from the memory module, optimizes the action space O, deletes the shielding operation in the action space, obtains the optimized action space, and converts the specific instructions of the UI test into the planning task of the LLM (that is, selects a UI operation from the optimized action space). Exemplarily, the current UI state is state A, the action space includes UI operation 1 (long press) and UI operation 2 (click), and the shielding operation space B includes the tuple "UI operation 1 (long press), state A", then when optimizing the action space, UI operation 1 (long press) needs to be deleted from the action space. Through the above steps, offloading such computational tasks (removing invalid UI operations from the action space) from the LLM to an external system (user interface test navigation system) can reduce the complexity of the task and the length of the context, thereby circumventing the theoretical limitations caused by computational complexity and the empirical limitations caused by multiple instruction tracking and long context processing.

[0063] Step 2: Generate operation prompts for the LLM based on the optimized action space O' and the historical test information obtained from the memory module.

[0064] In a possible implementation, the execution engine includes: a parser, an LLM driver, and a UI driver;

[0065] The parser generates an operation prompt according to the optimized action space and the historical test information obtained from the memory module; the operation prompt includes: the optimized action space, the test target, and the historical UI operation sequence.

[0066] Specifically, an operation prompt text is generated for the LLM through the parser in the execution engine, and the operation prompt includes: the described instructions (corresponding to the test target, such as turning on the night mode), the prompt of the optimized action space, and the UI operations that have been executed in the previous round of UI testing (corresponding to the historical UI operation sequence). Among them, in the operation prompt, the optimized action space can include identifiers of each UI operation, and each UI operation is uniquely identified using text attributes obtained from the XML file of the UI hierarchy. In the description of the optimized action space in the operation prompt, the description of each UI operation includes: event type, resource id, text representation and accessibility information (retrieved by the UI automatic manager).

[0067] The historical UI operation sequence can be obtained from the memory module. In addition to maintaining the shielded operation space, the memory module also stores historical test information. The historical test information includes: historical test information obtained from each round of UI testing (i.e., executing a UI operation on the AUT). The historical UI operation sequence is obtained from the historical test information and placed in the operation prompt text. Specifically, the historical test information includes: a historical UI operation sequence composed of effective UI operations in the order of execution, and a historical UI state sequence composed of effective UI states in the order of test execution; effective UI operations represent UI operations performed to achieve the expected effect of the corresponding round of UI testing (i.e., the UI test actions of this round that have been verified in step 5), and effective UI states represent UI states obtained when the corresponding round of UI testing achieves the expected effect (i.e., the post-test UI state); effective UI operations correspond one-to-one to the effective UI states.

[0068] In addition, the memory module also stores a historical data set, each piece of historical data in the historical data set includes: an activity name, a UI hierarchy, a UI operation planned by the first LLM for the UI hierarchy, and a UI state changed after executing the UI operation on the AUT.

[0069] Considering that LLM is less effective in following multiple or fine-grained instructions, it is difficult to improve LLM's accuracy in planning UI operations directly by modifying the prompt text. The embodiment of the present application optimizes the action space and uses the optimized action space to generate operation prompts for input to LLM, thereby deleting disallowed operations from the prompt text, thereby avoiding excessive modifications to the text content of the operation prompt, which helps LLM understand and follow instructions.

[0070] Step 3: Input the operation prompt into the first LLM, and the LLM responds according to the operation prompt to obtain a response result.

[0071] In this embodiment, after the parser generates an operation prompt, the LLM driver is used to send the operation prompt to the first LLM, and a response result obtained by the first LLM in response to the operation prompt is received; the response result is an identifier of the UI operation that needs to be performed in this round of testing under the current UI state, determined by the first LLM.

[0072] Specifically, Figure 1 As shown, the execution engine also includes an LLM driver, which is responsible for the interaction between the user interface test navigation system and the LLM, including: sending an operation prompt to the LLM (which may be the first LLM in this embodiment), and receiving a response result output by the first LLM based on the reasoning of the operation prompt. The response result may be an identifier corresponding to the UI operation that should be performed in the current UI state in this round of UI testing planned by the first LLM.

[0073] Step 4: By parsing the response results, the UI operation that should be performed in this round of UI testing (i.e., the UI test action in this round) is obtained, and the UI operation is executed on the AUT, so that the AUT changes from the current UI state to the post-test UI state.

[0074] In this embodiment, the parser parses the response result to obtain the current round of UI test actions; the current round of UI test actions includes: one or more of clicking, sliding, long pressing, and inputting text; using the UI driver, the current round of UI test actions is executed on the AUT to obtain the post-test UI state.

[0075] Specifically, the parser parses the response result and obtains an executable UI operation (this round of UI test action) according to the UI operation identifier in the response result. The parser coordinates with the memory module to manage the instruction flow and ensure that they can be correctly applied in the optimized action space. Figure 1 As shown, the execution engine also includes a UI driver, and the UI driver is responsible for the interaction between the user interface test navigation system and the AUT, including: executing UI operations on the AUT, and dumping UI hierarchy files representing the AUT screen, UI operation related data, such as the UI state after execution.

[0076] Among them, the UI operations that can be performed on the AUT are mainly divided into four categories: click, slide, long press, and input text. Click means clicking the center point of a given UI element. Long press means pressing the center point of a given UI element for a first duration. The first duration can be set according to actual application requirements, such as 1 second. Slide means performing a slide operation according to a given slide direction and distance. Input text means generating an appropriate string input and then inputting it into a given UI element.

[0077] Step 5: Based on the UI status after the test, verify whether this round of UI testing achieves the expected results.

[0078] In this embodiment, after executing the planned UI operation on the AUT, the verifier in the domain knowledge loader verifies whether the AUT after execution achieves the expected effect based on the relevant domain knowledge of the UI test. For example, if the expected effect is to turn on the night mode of the AUT, then whether the current round of UI testing achieves the expected effect can be determined based on whether the UI state after the test is the night mode.

[0079] In a possible implementation, the domain knowledge loader includes: a verifier; the verifying whether the current round of UI testing achieves the expected effect according to the post-test UI state includes:

[0080] The verifier sends the response result of the first LLM, the current UI state and the post-test UI state as a triplet to the second LLM, so that the second LLM determines whether the post-test UI state is consistent with the planned test state, and receives the verification result of the second LLM;

[0081] The verifier analyzes the post-test UI status based on heuristic rules to determine whether this round of UI testing achieves the expected effect.

[0082] The embodiment of the present application mainly adopts two methods of LLM feedback and heuristic rules for verification. Specifically, on the one hand, after executing this round of UI test actions, LLM is consulted according to the initial plan (including operation prompts and LLM response results), and LLM is used to determine whether the result matches the expected result. The response result of the first LLM, the current UI state and the UI state after the test (both can be in the form of screenshots) are sent as a triple to the second LLM (in this embodiment, the second LLM can be the same model as the first LLM, or it can be a different model). In addition, the task-related description information can also be sent to the second LLM, so that the second LLM determines whether the UI state after the test is consistent with the planned test state. If the response from the LLM (the second LLM) is "No", indicating that the UI operation is invalid, then this round of UI test actions is used as a shielding operation, and the current UI state is used as the UI state corresponding to the shielding operation, and the shielding operation space is updated. In this embodiment, the first LLM or the second LLM can refer to any large language model that can realize UI operation planning, or a cluster of multiple large language models. On the other hand, the results of actions are evaluated against known exploration traps in the heuristic rules, that is, if a UI operation leads to any one or more of the following three situations: unresponsiveness, repeated exploration, and exploration loop, the UI operation is considered invalid.

[0083] In a possible implementation, analyzing the post-test UI state based on heuristic rules to determine whether the current round of UI testing achieves the expected effect includes:

[0084] According to the post-test UI state, determine whether the same UI state has appeared in the previous test round. Specifically, the heuristic rule is expressed as a detection program for state duplication, and the detection program is executed to detect the post-test UI state. If the detection determines that the post-test UI state is the same UI state that has appeared in the previous test round, it means that repeated exploration has occurred, and this round of UI testing has not achieved the expected effect. The UI test action of this round is determined as a shielding operation, which is used for the memory module to update the shielding operation space.

[0085] Determine whether the post-test UI state is an unresponsive UI state. Specifically, the heuristic rule is represented by a detection program for unresponsiveness, and the detection program is executed to detect the post-test UI state. If the post-test UI state is detected to be an unresponsive UI state, it means that the current round of UI testing has not achieved the expected effect, and the current round of UI testing action is determined as a shielding operation, which is used for the memory module to update the shielding operation space.

[0086] According to the post-test UI state, determine whether a test state loop occurs. Specifically, the heuristic rule is represented by a detection program for a test state loop, and the detection program is executed to detect the post-test UI state. If the detection determines that a test state loop occurs (the post-test UI state appears repeatedly), it means that the current round of UI testing has not achieved the expected effect, and the current round of UI testing actions is determined as a shielding operation, which is used for the memory module to update the shielding operation space.

[0087] Step 6: If the expected effect is achieved, update the post-test UI state to the current UI state, and repeat steps 1-5 to perform the next round of UI testing; if the expected effect is not achieved, restore the UI state to the current UI state (i.e., the state before performing the invalid UI operation), and re-perform UI testing (i.e., re-execute steps 1-5).

[0088] In a possible implementation, the domain knowledge loader includes: an error handler; the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operations represent the UI operations performed to achieve the expected results of the corresponding round of UI testing (i.e., the UI test actions of this round that have been verified in step 5), and the effective UI states represent the UI states obtained when the corresponding round of UI testing achieves the expected results (i.e., the post-test UI states); the effective UI operations correspond one-to-one to the effective UI states.

[0089] If the expected effect is not achieved, the error handler is used to perform UI state backtracking, detect the position of the UI state after rollback in the historical UI state sequence, and re-execute the corresponding effective UI operation according to the historical UI operation sequence to restore to the current UI state.

[0090] Specifically, when the verification fails (for example, the verification result of the second LLM is NO), a "return" action is triggered on the AUT to restore to the state before the UI operation is performed. Exemplarily, during the UI test process, after executing the nth UI operation, the UI state after the nth test is obtained. After verification, it is determined that the executed nth UI operation is an invalid UI operation (that is, the UI state after the nth test cannot achieve the expected effect), then the error handler is activated to perform UI state backtracking (for example, triggering a "return" action on the AUT). The UI state after rollback needs to be determined according to the actual application situation. The UI state after rollback may be the UI state after the kth UI operation (k is less than n-1), or it may be directly rolled back to the UI state after the n-1th UI operation. When the UI state after rollback is the UI state after the n-1th UI operation, it means that the current UI state has been restored, and steps 1-5 can be re-executed. When the UI state after rollback is the UI state after the kth UI operation (k is less than n-1), it is also necessary to detect the position of the UI state after rollback in the historical UI state sequence (located at the kth position), and replay the k+1th to n-1th UI operations according to the historical UI operation sequence to restore to the current UI state. Once the UI state is restored, start the next iteration, that is, re-execute steps 1-5, so that LLM re-plans the UI operations that should be performed in this round of testing based on the optimized action space (excluding the previously invalid UI operations), thereby avoiding selecting invalid UI operations again.

[0091] In a possible implementation manner, when the expected effect is not achieved, the memory module uses the current round of UI test action as a shielding operation, uses the current UI state as the UI state corresponding to the shielding operation, and updates the shielding operation space;

[0092] When the expected effect is achieved, the memory module updates the historical test information, adds the current round of UI test actions as the effective UI operations to the historical UI operation sequence, and adds the post-test UI state as the effective UI state to the historical UI state sequence.

[0093] On the one hand, the embodiment of the present application optimizes the action space (the set of selectable UI operations for the current UI state) by using the domain knowledge loader, and deletes invalid UI operations (i.e., masked operations, which indicate UI operations that do not achieve the expected effect after being executed in the corresponding UI state) during the test from the action space. On the other hand, through the domain knowledge loader, after each execution of a UI operation, it is promptly verified whether the current round of UI testing has achieved the expected effect based on the UI state after the test. If the expected effect is not achieved, the UI state is restored to the state before the invalid operation is executed (i.e., the current UI state), so that the LLM can re-determine the UI operation that should be executed in the optimized action space, avoiding the LLM from repeatedly selecting the wrong UI operation, and realizing that in the process of LLM planning the UI operation sequence, in the event of an error in the currently planned UI operation, timely re-planning is performed based on the current information to generate a reliable and accurate UI operation sequence for the test target, thereby improving the effectiveness of automatic UI testing based on LLM.

[0094] The second aspect of the embodiment of the present application further provides a user interface test navigation method, which is applied to the user interface test navigation system described in the first aspect of the embodiment of the present application, referring to Figure 3 , Figure 3 A flowchart of the steps of a user interface test navigation method is shown. Figure 3 As shown, the method includes:

[0095] Step S101, optimizing the action space according to the shielded operation space and the current UI state, deleting the shielded operations in the action space, and obtaining an optimized action space; the shielded operation space includes one or more shielded operations, and the shielded operation represents a UI operation that does not achieve an expected effect after being executed in the corresponding UI state;

[0096] Step S102, generating an operation prompt according to the optimized action space, so that the first large language model LLM determines the UI test action of this round according to the operation prompt;

[0097] Step S103, obtaining a post-test UI state obtained by executing the current round of UI test actions on the test application AUT;

[0098] Step S104, verifying whether the current round of UI testing achieves the expected effect based on the post-test UI state, and if the expected effect is not achieved, restoring the UI state to the current UI state and re-performing the UI test.

[0099] In a possible implementation, optimizing the action space according to the shielded operation space and the current UI state, deleting the shielded operation in the action space, and obtaining the optimized action space includes:

[0100] According to the current UI state, a list of available UI operations of the current AUT interface is obtained by parsing, and the action space is generated, where the action space represents a set of all UI operations that can be performed on the current AUT interface;

[0101] The action space is optimized according to the masking operation space obtained from the memory module, and the masking operations in the action space are deleted to obtain the optimized action space.

[0102] In a possible implementation, generating an operation prompt according to the optimized action space, and enabling the first large language model LLM to determine the current round of UI test actions according to the operation prompt, includes:

[0103] Generate an operation prompt according to the optimized action space and the acquired historical test information; the operation prompt includes: the optimized action space, the test target, and the historical UI operation sequence;

[0104] Sending the operation prompt to the first LLM, and receiving a response result obtained by the first LLM in response to the operation prompt; the response result is an identifier of the UI operation to be performed in the current UI state in this round of testing determined by the first LLM;

[0105] Parse the response result to obtain the UI test action of this round; the UI test action of this round includes: one or more of clicking, sliding, long pressing, and inputting text;

[0106] The current round of UI test actions is executed on the AUT to obtain the post-test UI state.

[0107] In a possible implementation, verifying whether the current round of UI testing achieves the expected effect according to the post-test UI state includes:

[0108] Send the response result of the first LLM, the current UI state and the post-test UI state as a triplet to the second LLM, so that the second LLM determines whether the post-test UI state is consistent with the planned test state, and receives the verification result of the second LLM;

[0109] Based on the heuristic rules, the post-test UI status is analyzed to determine whether this round of UI testing achieves the expected effect.

[0110] In a possible implementation, analyzing the post-test UI state based on heuristic rules to determine whether the current round of UI testing achieves the expected effect includes:

[0111] Determine, based on the post-test UI state, whether the same UI state has appeared in a previous test round;

[0112] Determine whether the post-test UI state is an unresponsive UI state;

[0113] According to the post-test UI state, it is determined whether a test state loop occurs.

[0114] In a possible implementation, the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operation indicates the UI operation performed to achieve the expected effect of the corresponding round of UI test, and the effective UI state indicates the UI state obtained when the corresponding round of UI test achieves the expected effect; the effective UI operation corresponds to the effective UI state one by one;

[0115] The method further comprises:

[0116] If the expected effect is not achieved, the UI state is rolled back to detect the position of the rolled-back UI state in the historical UI state sequence, and the corresponding effective UI operation is re-executed according to the historical UI operation sequence to restore to the current UI state.

[0117] In a possible implementation, the method further includes:

[0118] If the expected effect is not achieved, the UI test action of this round is used as a shielding operation, the current UI state is used as the UI state corresponding to the shielding operation, and the shielding operation space is updated;

[0119] When the expected effect is achieved, the historical test information is updated, the current round of UI test actions are added as the effective UI operations to the historical UI operation sequence, and the post-test UI state is added as the effective UI state to the historical UI state sequence.

[0120] The present application also provides an electronic device, referring to Figure 4 , Figure 4 Schematic diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the user interface test navigation method disclosed in the embodiment of the present application.

[0121] The embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps in the user interface test navigation method disclosed in the embodiment of the present application are implemented.

[0122] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor implements the steps of the user interface test navigation method disclosed in the embodiment of the present application.

[0123] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0124] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0128] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0129] The above is a detailed introduction to a user interface test navigation system, method and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A user interface test navigation system, characterized in that: The system comprises: A memory module, used to maintain a shielded operation space and historical test information; the shielded operation space includes one or more shielded operations, and the shielded operation represents a UI operation that does not achieve an expected effect after being executed in a corresponding UI state; A domain knowledge loader, used to optimize the action space according to the shielded operation space and the current UI state, delete the shielded operation in the action space, and obtain an optimized action space; The execution engine is used to generate an operation prompt according to the optimized action space, so that the first large language model LLM determines the current round of UI test actions according to the operation prompt; and obtains the post-test UI state obtained by executing the current round of UI test actions on the test application AUT; The domain knowledge loader is also used to verify whether the current round of UI testing achieves the expected effect based on the post-test UI state. If the expected effect is not achieved, the UI state is restored to the current UI state and the UI test is performed again.

2. The user interface test navigation system according to claim 1, characterized in that: The domain knowledge loader includes: an optimizer; The optimizer is used to parse and obtain a list of available UI operations of the current AUT interface according to the current UI state, and generate the action space, wherein the action space represents a set of all UI operations that can be performed on the current AUT interface; The optimizer optimizes the action space according to the masked operation space obtained from the memory module, deletes the masked operations in the action space, and obtains the optimized action space.

3. The user interface test navigation system according to claim 2, characterized in that: The execution engine includes: a parser, an LLM driver and a UI driver; The parser generates an operation prompt according to the optimized action space and the historical test information obtained from the memory module; the operation prompt includes: the optimized action space, the test target, and the historical UI operation sequence; Using the LLM driver, sending the operation prompt to the first LLM, and receiving a response result obtained by the first LLM according to the operation prompt response; the response result is an identifier of the UI operation to be performed in the current UI state in this round of testing determined by the first LLM; The parser parses the response result to obtain the UI test action of this round; the UI test action of this round includes: one or more of clicking, sliding, long pressing, and inputting text; The UI driver is used to execute the current round of UI test actions on the AUT to obtain the post-test UI state.

4. The user interface test navigation system according to claim 1, characterized in that: The domain knowledge loader includes: a verifier; the verification of whether the current round of UI testing achieves the expected effect according to the post-test UI state includes: The verifier sends the response result of the first LLM, the current UI state and the post-test UI state as a triplet to the second LLM, so that the second LLM determines whether the post-test UI state is consistent with the planned test state, and receives the verification result of the second LLM; The verifier analyzes the post-test UI status based on heuristic rules to determine whether this round of UI testing achieves the expected effect.

5. The user interface test navigation system according to claim 4, characterized in that: The analyzing the post-test UI state based on the heuristic rule to determine whether the current round of UI testing achieves the expected effect includes: Determine, based on the post-test UI state, whether the same UI state has appeared in a previous test round; Determine whether the post-test UI state is an unresponsive UI state; According to the post-test UI state, it is determined whether a test state loop occurs.

6. The user interface test navigation system according to claim 1, characterized in that: The domain knowledge loader includes: an error handling program; the historical test information includes: a historical UI operation sequence composed of effective UI operations in execution order, and a historical UI state sequence composed of effective UI states in test execution order; the effective UI operation represents the UI operation performed to achieve the expected effect of the corresponding round of UI test, and the effective UI state represents the UI state obtained when the corresponding round of UI test achieves the expected effect; the effective UI operation corresponds to the effective UI state one by one; If the expected effect is not achieved, the error handler is used to perform UI state backtracking, detect the position of the UI state after rollback in the historical UI state sequence, and re-execute the corresponding effective UI operation according to the historical UI operation sequence to restore to the current UI state.

7. The user interface test navigation system according to claim 6, characterized in that: In the case that the expected effect is not achieved, the memory module uses the current round of UI test action as a shielding operation, uses the current UI state as the UI state corresponding to the shielding operation, and updates the shielding operation space; When the expected effect is achieved, the memory module updates the historical test information, adds the current round of UI test actions as the effective UI operations to the historical UI operation sequence, and adds the post-test UI state as the effective UI state to the historical UI state sequence.

8. A user interface test navigation method, characterized in that: The user interface test navigation system applied to any one of claims 1 to 7, the method comprising: According to the shielded operation space and the current UI state, the action space is optimized, and the shielded operations in the action space are deleted to obtain an optimized action space; the shielded operation space includes one or more shielded operations, and the shielded operation represents a UI operation that does not achieve an expected effect after being executed in the corresponding UI state; Generate an operation prompt according to the optimized action space, so that the first large language model LLM determines the UI test action of this round according to the operation prompt; Obtaining a post-test UI state obtained by executing the current round of UI test actions on the test application AUT; According to the post-test UI state, verify whether this round of UI testing achieves the expected effect. If the expected effect is not achieved, restore the UI state to the current UI state and re-perform the UI test.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the user interface test navigation method of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the user interface test navigation method described in claim 8 is implemented.

Images