Android application GUI test case automatic generation method based on large language model

Abstract

The invention discloses an Android application GUI test case automatic generation method based on a large language model, in the test process, an automatic test tool automatically runs Android Apk, current interface layout information is obtained, and then the layout information is provided for a large model for auxiliary analysis. Test requirements of testers and current layout information of the application are understood, the reasoning ability of the large language model is optimized through a cue word engineering method, a test path is automatically generated on the current test application to meet the test requirements, and finally a test assertion is generated to verify the correctness and integrity of the test. According to the method, the efficiency of automatically generating the test case by the Android is improved, and the method for helping developers to guarantee the reliability of the mobile application is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a method for automatically generating test cases for an Android application GUI based on a large language model. Background Art

[0002] Although a variety of automated test case generation methods for Android applications have been developed, the effectiveness of these methods in practical applications still needs to be improved. For example, random automated testing tools such as Monkey simulate user interaction events through a pseudo-random number generator, often generating a large number of invalid operations, resulting in low test efficiency, and often repeatedly testing the easily accessible interface while ignoring other interfaces. Although this method is simple and intuitive, its randomness leads to low test coverage and difficulty in discovering deep-seated software defects.

[0003] Model-based automated testing tools such as Stoat build application behavior models through static or dynamic analysis, but these models often cannot accurately describe the actual behavior of the application and have certain deviations. Although static analysis can quickly identify code structure and possible behavior patterns, it cannot execute code, so it is difficult to identify runtime problems or behaviors that depend on specific states. Although dynamic analysis can capture more runtime data and behaviors when executing applications, its effectiveness is limited by the quality and coverage of test cases, and it usually requires longer analysis time and more resource consumption.

[0004] Although reinforcement learning-based testing methods such as Q-Testing can learn the test value of controls through a continuous trial-and-error process to optimize the generation of test events, they often have difficulty generating long sequence tests with specific semantic information. The effectiveness of such methods depends largely on the design of the reward function. If the reward function does not accurately reflect the test goal, the action sequence generated by the test may not effectively trigger deep-seated errors.

[0005] Large language model testing methods such as GPTDroid can use the powerful capabilities of language models to generate test sequences with semantic information and simulate complex user operation behaviors, but this method has limited adaptability and flexibility when facing specific and complex test scenarios. Although this method can generate relatively reasonable test inputs based on the context, it may not fully meet the needs when in-depth testing of specific application logic is required.

[0006] Methods that generate test actions based on natural language descriptions, such as seq2act, locate target controls based on the similarity between controls and descriptions and generate corresponding test actions. The core advantage of this method is that it can convert natural language instructions into specific test actions, making the writing of test cases more intuitive and convenient. However, the effectiveness of such methods greatly depends on the accuracy and specificity of natural language descriptions, as well as the system's ability to understand the descriptions. If the description is unclear or contains ambiguity, the generated test actions may not accurately perform the intended test tasks. In addition, existing testing tools often fail to verify whether the application's response to the generated test inputs meets the tester's expectations. For example, assertion generation tools such as Magneto rely on templates to generate test assertions, but are not sufficient to adapt to the complexity and variability of test cases.

[0007] In summary, although various automated testing methods have their own advantages, they still face various limitations and challenges in practical applications. Summary of the invention

[0008] Purpose of the invention: The purpose of the present invention is to provide a method for automatically generating Android application GUI test cases based on a large language model, so as to solve the problem that current mobile application test scripts rely on manpower and lack effective automated testing methods, and to help program developers perform graphical user testing of business functions through automated means to ensure program reliability.

[0009] Technical solution: The method for automatically generating Android application GUI test cases based on a large language model described in the present invention comprises the following steps: (1) Automatically run the APK on an Android virtual machine or physical machine; obtain the test task assigned by the tester for this test; (2) Get the current control layout of the Android application; (3) Infer executable user interaction events; (4) Based on the Zero-Chain-Of-Thought traversal mode, the action that is most likely to complete the test task is selected for execution; (5) Repeat steps (3) to (4) until the large language model determines that the task is complete or the maximum test step length is reached; obtain a test case that satisfies the test task and a test assertion that verifies the correctness of the execution of the test case; (6) The large language model selects actions that can verify whether the current test task is completed correctly based on the test assertion; (7) Repeat steps (3), (4) and (6) until the large language model determines whether the current test task is completed correctly; (8) Infer the macro steps to complete the test task; (9) Based on the Tree-Of-Thought traversal mode, select the action that is most likely to complete the test task and execute it; (10) Repeat (3), (4) and (9) until the large language model determines that the task is completed or the maximum test step length is reached; (11) Repeat (7) until the large language model determines whether the test assertion has been executed; obtain the test case that meets the test task requirements and the test assertion that verifies its execution correctness.

[0010] Furthermore, in step (1), the tester gives a natural language description of the test task for the test intent; wherein the natural language description does not need to involve specific control information and step sequence.

[0011] Furthermore, in step (2), the UIAutomator tool is used to obtain the current control layout of the Android application, and the inclusion relationship of various controls is described in the form of a tree structure. At the same time, the current attribute value of each control is recorded, including: position, visibility, clickability, text information, and control type. After obtaining the tree structure, some nodes in the tree are pruned to only retain the control information of operable attributes and control information with text content.

[0012] Furthermore, in step (3), the interactive controls in the current control layout of the Android application are numbered and organized in XML format.

[0013] Furthermore, in step (5), the large language model will judge the completion status of the current test task. If it is inferred that the current test task has been completed, it will determine that the task is completed. The large language model will end the current use case generation when it determines that the task is completed or the maximum test step is reached. Among them, the maximum test step is the maximum test step manually set before the test starts.

[0014] Furthermore, in step (5), the test case that satisfies the test task and the test assertion that verifies the correctness of the execution of the test case are obtained as follows: the large language model combines the current test task and the application type to generate a test assertion, wherein the test assertion is used to determine whether the test task is executed correctly.

[0015] Furthermore, step (6) is specifically as follows: the large language model selects an action that meets the test assertion action from the interactive controls in the current control layout of the Android application and executes it.

[0016] Furthermore, step (9) is specifically as follows: the test actions of the test cases generated by the test tool are stored in short-term memory in a tree form, the large language model filters out the wrong choices that have been executed in the currently executable user interaction events, and selects the actions that are most likely to complete the test task in the interaction events that have not been executed for execution; at the same time, the large language model combines historical actions and the current interaction events that have not been executed, and will choose to backtrack when it is inferred that the execution of the current task does not meet expectations.

[0017] Furthermore, in step (11), after completion, the large language model will interact with the Android Apk again based on the test assertion to execute the test assertion and determine whether the program behavior is consistent with the test assertion.

[0018] Furthermore, in step (11), two files are obtained after the test is completed. One file records the complete test case, including the operation type and target control of each step, and the other file records the test assertions for the test case, which can be used to test the correctness of the current test case execution.

[0019] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: by utilizing the reasoning ability and semantic understanding ability of a large language model, the tester's test intent is understood, and interactive events that can meet the test intent are selected to execute to improve the efficiency of Android automatic test case generation, providing a method to help developers ensure the reliability of mobile applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a method for automatically generating Android application GUI test cases based on a large language model, comprising the following steps: S1: Automatically run APK on an Android virtual machine or physical machine; the details are as follows: First, you need to obtain the APK file of the Android application to be tested. Before officially starting the automatic test, you need to set the maximum number of macro steps for this test, the maximum test step length, and the package information of the application to be tested. After completing the preparations, you can start the automatic test process. Here, you can specify whether the Android application runs on an Android virtual machine or a physical machine through parameter configuration.

[0023] S2: Obtain the test task of the tester for this test; specifically: the tester describes the test task of this round of testing in natural language, without involving the operation type and target control of each detailed step.

[0024] S3: Get the current control layout of the Android application; as follows: By using the UIAutomator tool, you can capture the control layout on the current screen from the Android virtual machine or physical device in real time. This layout information is stored in XML format, where each XML element represents a control. These controls may contain multiple sub-controls, forming a nested structure, which as a whole constitutes a control tree. Each control element contains multiple attributes, such as bounds, checkable, package, resource ID, etc. These attributes describe the control's location, operability, application package, control ID and other status information.

[0025] S4: Infer executable user interaction events; specifically as follows: the test tool first deeply traverses the control layout tree, infers interaction events for all nodes whose operable attributes are true, and after the inference is completed, summarizes them on the basis of retaining their semantic information to reduce the token cost in the process of querying the large language model. The executable event inference comprehensively considers the properties of the control itself, the properties of its nested child nodes and other information. If multiple operable controls form a nested structure, their nested structure will be retained here. After traversing the control tree, the interactive control nodes will be numbered and reorganized into XML format so that the large language model can quickly understand the current test application interface status information.

[0026] S5: Based on the traversal mode of Zero-Chain-Of-Thought, the action that is most likely to complete the test task is selected for execution; specifically, the test tool abstracts the entire decision-making process into a Q&A process with the large language model, and provides the following information to the large language model during the inquiry process: the current test task, the executable user interaction event, and historical decision information. Based on this information, the large language model selects an executable user interaction event from the executable user interaction events provided and explains the reason. The test tool matches the expected answer in a regular manner from the query results of the large language model. If it is found that the large language model cannot generate the structure of the expected format due to problems such as hallucinations, it will repeatedly query the large language model until the result that meets the expected format is returned.

[0027] S6: Repeat S3 to S5 until the large language model determines that the task is completed or the maximum test step is reached; the maximum test step is the maximum test step set manually before the test starts; in addition, the large language model will judge the completion status of the current test task. If it is inferred that the current test task has been completed, it will judge the task to be completed in advance; when the large language model determines that the task is completed or the maximum test step is reached, the current use case generation will be terminated and S7 will be executed.

[0028] S7: Get the test case that meets the test task and the test assertion that verifies the correctness of the execution of the case; specifically: the test tool regards the large language model as a test expert with testing experience, first lets the large language model infer the application type based on the package information of the application to be tested, and then combines the current test task to obtain a test assertion that can verify whether the current task is correctly executed. The form of test assertions includes traditional keyword matching methods and innovative interactive test assertions. The test tool will verify whether the current test task is correctly executed through keyword matching, historical action analysis, re-execution, result verification and other methods.

[0029] S8: The large language model selects an action based on the test assertion to verify whether the current test task is completed correctly; specifically, the test tool selects the next interaction event based on the test assertion and the currently executable user interaction event to verify whether the current task is executed correctly. If the test tool believes that the current test assertion has not been executed, it will select the next interaction event. Otherwise, it will determine whether the current program behavior meets the test assertion expectations and whether the current test task is completed correctly.

[0030] S9: Repeat S3, S4 and S8 until the large language model determines whether the current test task is completed correctly; if the test tool is still in the verification process, it will repeat S3, S4 and S8 until the test tool determines whether the current program behavior meets the test assertion expectations. If the verification program behavior meets expectations, it will execute S14, otherwise, execute S10.

[0031] S10: Infer the macro steps to complete the test task; specifically: the test tool decomposes the test task into macro test steps based on the reasoning ability of the large language model. The maximum number of macro steps is the maximum macro step length manually set before the test starts. For each decomposed step, the large language model will analyze the general execution idea.

[0032] S11: Based on the Tree-of-Thought traversal mode, the action that is most likely to complete the test task is selected for execution; specifically, the test process adopts the Tree-of-Thought-based prompt engineering method. This technology assists large models in reasoning and decision-making through tree-shaped thinking links. The core is to use the tree structure to organize and process decision data. Specifically, the test tool first converts the test case generation problem at this stage into a tree-shaped test path exploration problem. The current application interface to be tested is abstracted as a node of the tree, and all executable user interaction events are regarded as edges starting from the node. Such abstraction allows the decision-making process to be carried out on this decision tree, and each interaction is equivalent to selecting a possible path. Furthermore, the test tool uses the short-term memory of historical decisions maintained by the tree structure to optimize the decision-making process. The system will check the previous decisions at the current node, identify and filter out those decision paths that have been proven to be wrong at historical moments. In this way, the tool can select the action that is most likely to help complete the test task from the remaining, non-excluded nodes for execution. In addition, if the large language model analysis determines that the jump caused by the previous decision did not meet the initial expectations, or if the depth of the historical decision tree has reached the maximum limit set manually before the test starts at the current macro step, the system will trigger a backtracking mechanism. This mechanism allows the decision-making process to return to the previous node and make a new decision after re-evaluating the situation.

[0033] S12: Repeat S3, S4 and S11 until the large language model determines that the task is completed or the maximum test step is reached; if the large language model infers that the current test task is not completed, it will continue to select the next test action; if the inference believes that the current test task has been completed, the current use case generation is terminated.

[0034] S13: Repeat S9 until the large language model determines whether the test assertion is executed; specifically, the large language model selects an action that can verify whether the current test task is correctly completed based on the test assertion and the currently executable user interaction event to execute.

[0035] S14: Obtain test cases that meet the requirements of the test task and test assertions that verify the correctness of their execution; specifically: during the traversal process of the test tool based on the reasoning of the large language model, collect operation paths and jump relationships to constitute the test cases output by this test; the test assertions obtained by the large language model reasoning can be used to determine the correctness of the execution of the test case by the current program.

Claims

1. A method for automatically generating Android application GUI test cases based on a large language model, characterized in that: The following steps are involved: (1) Automatically run APK on an Android virtual machine or physical machine; Get the test task of the tester for this test; (2) Get the current control layout of the Android application; (3) Infer executable user interaction events; (4) Based on the Zero-Chain-Of-Thought traversal mode, the action that is most likely to complete the test task is selected for execution; (5) Repeat steps (3) to (4) until the large language model determines that the task is complete or the maximum test step length is reached; obtain a test case that satisfies the test task and a test assertion that verifies the correctness of the execution of the test case; (6) The large language model selects actions that can verify whether the current test task is completed correctly based on the test assertion; (7) Repeat steps (3), (4) and (6) until the large language model determines whether the current test task is completed correctly; (8) Infer the macro steps to complete the test task; (9) Based on the Tree-Of-Thought traversal mode, select the action that is most likely to complete the test task and execute it; (10) Repeat (3), (4) and (9) until the large language model determines that the task is completed or the maximum test step length is reached; (11) Repeat (7) until the large language model determines whether the test assertion has been executed; obtain the test case that meets the test task requirements and the test assertion that verifies its execution correctness.

2. According to the method for automatically generating Android application GUI test cases based on a large language model according to claim 1, it is characterized in that: In step (1), the tester gives a natural language description of the test task for the test intent; wherein the natural language description does not need to involve specific control information and step sequence.

3. According to the method for automatically generating Android application GUI test cases based on a large language model as described in claim 1, it is characterized in that: In step (2), the UIAutomator tool is used to obtain the current control layout of the Android application, and the inclusion relationship of various controls is described in the form of a tree structure. At the same time, the current attribute value of each control is recorded, including: position, visibility, clickability, text information, and control type. After obtaining the tree structure, some nodes in the tree are pruned to only retain the control information of operable attributes and control information with text content.

4. According to the method for automatically generating Android application GUI test cases based on a large language model as described in claim 1, it is characterized in that: In step (3), the interactive controls in the current control layout of the Android application are numbered and organized in XML format.

5. According to the method for automatically generating Android application GUI test cases based on a large language model as described in claim 1, it is characterized in that: In step (5), the large language model will judge the completion status of the current test task. If it is inferred that the current test task has been completed, it will determine that the task is completed. The large language model will end the current use case generation when it determines that the task is completed or the maximum test step is reached. Among them, the maximum test step is the maximum test step manually set before the test starts.

6. According to the method for automatically generating Android application GUI test cases based on a large language model as described in claim 1, it is characterized in that: In step (5), the test case that satisfies the test task and the test assertion that verifies the correctness of the execution of the test case are obtained as follows: the large language model combines the current test task and application type to generate a test assertion, where the test assertion is used to determine whether the test task is executed correctly.

7. According to the method for automatically generating Android application GUI test cases based on a large language model as described in claim 1, it is characterized in that: Step (6) is as follows: the large language model selects an action that meets the test assertion action from the interactive controls in the current control layout of the Android application and executes it.

8. According to the method for automatically generating Android application GUI test cases based on a large language model as described in claim 1, it is characterized in that: Step (9) is as follows: the test actions of the test cases generated by the test tool are stored in short-term memory in a tree form; the large language model filters out the wrong choices that have been executed in the currently executable user interaction events, and selects the actions that are most likely to complete the test task in the interaction events that have not been executed for execution; at the same time, the large language model combines historical actions and the current interaction events that have not been executed, and will choose to backtrack when it is inferred that the execution of the current task does not meet expectations.

9. The method for automatically generating Android application GUI test cases based on a large language model according to claim 1, characterized in that: In step (11), after completion, the large language model will execute the test assertion based on the test assertion and determine whether the program behavior is consistent with the test assertion by interacting with the Android Apk again.

10. The method for automatically generating Android application GUI test cases based on a large language model according to claim 1, characterized in that: In step (11), two files are obtained after the test is completed. One file records the complete test case, including the operation type and target control of each step, and the other file records the test assertions for the test case, which can be used to test the correctness of the current test case execution.

Images