LLM semantic combination enhancement method oriented to GUI automatic exploration test

Abstract

The invention discloses an LLM semantic combination enhancement method oriented to a GUI automatic exploration test. The method comprises the following steps: (1) constructing a semantic enhanced UTG based on an LLM; (2) refining the semantic enhanced UTG, merging state transitions with equivalent functions, identifying and eliminating redundant paths in the graph, and then decomposing the graph into a series of test units; (3) using LLM as a state combiner, and performing target-oriented combination on the test units obtained by decomposition to generate a long test sequence; (4) performing path completion on the long test sequence to form a complete execution path, and performing automatic execution and monitoring on the complete execution path; and (5) evaluating the coverage rate of the deep business logic through logs and state data collected in the execution stage, and evaluating the triggering capability of application defects by analyzing the deviation between an actual execution result and an expected behavior. By utilizing the method and the system, the automation, reproducible generation and verification of the deep function test of the application can be realized.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and software engineering, and in particular relates to an LLM semantic composition enhancement method for GUI automated exploration testing. Background Technology

[0002] With the increasing complexity of mobile and web applications, automated GUI exploration testing has become a crucial step in ensuring software quality. Existing automated GUI exploration testing techniques, such as random exploration-based, model-based, or reinforcement learning-based methods, can already achieve a certain degree of automated traversal of application interfaces and verification of basic functions.

[0003] For example, Chinese patent document CN120429235A discloses a method and system for automated testing of mobile application GUI based on hierarchical reinforcement learning; Chinese patent document CN109634831A discloses a method and system for automatic traversal testing of Android application GUI.

[0004] However, most of these technologies are limited to the syntactic level (such as control recognition, clicks, and input), and their exploration behavior mainly relies on the state transitions of the static structure of the GUI, making it difficult to deeply understand the complex business logic behind page elements. This results in test paths that are usually shallow and cannot effectively reach those deep functions that require multiple steps with strong logical connections between them. This limitation creates a difficult-to-bridge "semantic gap" between traditional automated exploration testing and complex business scenarios of real users. Summary of the Invention

[0005] To address the shortcomings of existing automated exploration testing technologies, which are limited to the syntactic level and struggle to cover deep business logic, this invention provides an LLM semantic composition enhancement method for GUI automated exploration testing. By introducing the powerful natural language understanding and logical reasoning capabilities of LLM, the low-level UI trajectories generated by automated tools are semantically enhanced and combined into complex paths that simulate real business scenarios, thereby achieving automated, reproducible generation and verification of deep functional testing of applications.

[0006] An LLM semantic composition enhancement method for GUI-based automated exploratory testing includes the following steps: (1) Construct a semantically enhanced user transition graph (UTG) based on LLM; the nodes of the UTG are defined as the static page states of the GUI, and the edges are represented as executable user operations connecting different states; (2) The semantically enhanced UTG is refined, functionally equivalent state transitions are merged, redundant paths in the graph are identified and eliminated, and then the UTG after eliminating redundancy is decomposed into a series of minimal, state-independent, composable test units. (3) Using LLM as a state combiner, the decomposed test units are combined in a goal-oriented manner to generate long test sequences that simulate real user journeys and complex multi-step business workflows. (4) Complete the path of long test sequences to form a complete execution path, and automatically execute and monitor the complete execution path; (5) The coverage of deep business logic is evaluated by collecting logs and status data during the execution phase, and the ability to trigger application defects is evaluated by analyzing the deviation between the actual execution results and the expected behavior.

[0007] The core idea of ​​this invention is to transform the test generation paradigm from traditional syntactic state exploration to LLM-driven semantic enhancement and composition. First, the reasoning capabilities of LLM are used to analyze and expand the User Transition Graph (UTG) generated by traditional tools. Then, the enhanced path is decomposed into reusable test units. Finally, LLM is used as a "state combiner" to recombine these units into long test sequences with complex business logic.

[0008] Step (1) integrates the semantic understanding capabilities of LLM with traditional modeling-based automated exploration tools to construct a high-density, highly semantic UTG. During this hybrid exploration process, LLM performs semantic analysis and priority ranking on the GUI components of newly discovered pages, and dynamically generates high-semantic-value test inputs and resource creation test cases to guide the exploration process in overcoming bottlenecks and reaching deeper states. This process is iterative, continuously expanding and semantically enhancing the UTG to provide a foundational model for generating high-quality test sequences.

[0009] In step (1), a semantically enhanced user transformation graph (UTG) is constructed based on LLM. The specific process is as follows: (1-1) A hybrid exploration loop is adopted, in which the automated exploration tool is responsible for performing specific UI operations, while the LLM dynamically guides the exploration direction; (1-2) At each step of the exploration loop, LLM performs semantic analysis and priority sorting on the GUI components of the newly discovered page, and dynamically generates test inputs and resource creation test cases with high semantic value. (1-3) The automated exploration tool performs specific operations based on the priority ordering provided by the LLM, test inputs with high semantic value, and resource creation test cases; during the execution process, the UTG is dynamically expanded and semantically enhanced in a structured manner. (1-4) Iterate the above steps until the preset coverage or time threshold is met, and finally obtain the semantically enhanced user conversion graph UTG.

[0010] In steps (1-3), all newly discovered GUI nodes and edges are recorded in real time during the execution process.

[0011] The specific process of step (2) is as follows: (2-1) Apply the automatic test case extraction algorithm to perform graph structure analysis on the semantically enhanced UTG, and identify and extract acyclic paths with path simplicity and state independence as constraints. (2-2) The extracted acyclic path is deduplicated and its state is normalized, and it is defined as the smallest atomic unit of state transition, i.e., the test unit. (2-3) Define the pre-state required for execution and the post-state after execution in a structured manner for each test unit to ensure its reusability and composability.

[0012] In step (2-1), each acyclic path corresponds to a short test case with independent execution logic.

[0013] The specific process of step (3) is as follows: (3-1) LLM uses semantically enhanced UTG as the semantic and structural basis, and performs state matching and connection based on the pre-state and post-state of each test unit; (3-2) Dynamically select and sort test units, and combine them into a test chain with high exploration value and continuous execution, forming a long test sequence.

[0014] The specific process of step (4) is as follows: (4-1) Perform strict connectivity verification on the subsequent and preceding states of adjacent test units in a long test sequence; (4-2) If the states do not match, the necessary transition operations are automatically identified and injected based on the topology information, control semantics and historical transfer trajectory of the semantically enhanced UTG, so as to generate a logically coherent and state-synchronized complete execution path. (4-3) Call the automated execution engine to execute each test unit and the transition operations between them step by step according to the complete execution path; among them, the transition operations include cross-page jump, context synchronization, intermediate input or confirmation actions, and page condition waiting.

[0015] (4-4) Collect GUI state snapshots, control response events, operation logs and exception stack information in real time to build a detailed execution trajectory.

[0016] The specific process of step (5) is as follows: (5-1) Verify the business consistency of the endpoint state based on the execution trajectory, that is, compare the final actual state of the test sequence with the expected endpoint state based on business semantic reasoning; (5-2) Based on the collected logs and status data, conduct an in-depth evaluation of status coverage, transition coverage, and especially the coverage of deep business logic; (5-3) Assess the ability to trigger application defects by analyzing the deviation between the actual execution results and the expected behavior (e.g., crashes, abnormal interface response, logical navigation errors, or inconsistent states).

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention transforms the test generation paradigm from syntactic exploration to semantic enhancement and composition, leveraging the reasoning and composition capabilities of LLM to significantly improve the automated exploration testing capabilities and defect detection rate for deep application functions. It provides an efficient and scalable new path to solve the semantic gap and deep coverage challenges in automated exploration testing. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an LLM semantic composition enhancement method for GUI-based automated exploratory testing, according to an embodiment of the present invention. Figure 2 This is a system framework diagram corresponding to the method in the embodiments of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0022] like Figure 1 and Figure 2 As shown, an LLM semantic composition enhancement method for GUI-based automated exploratory testing includes the following steps: Step S1, semantically enhanced UTG construction.

[0023] This step proposes a UTG construction method based on the collaboration of automated exploration and Large Language Model (LLM) inference to achieve high semantic coverage and functional exploration of the target application (AUT, Application Under Test). This method achieves systematic transformation and iterative optimization from low-level interaction data to high-level semantic test cases through structured modeling of the interaction trajectories obtained from automated exploration, combined with the semantic reasoning and generation capabilities of LLM.

[0024] (1) Hybrid Exploration Phase: This method does not adopt a sequential model of exploration followed by analysis, but instead initiates a hybrid exploration loop that integrates LLM and automated exploration tools. In this loop, the automated exploration tool is responsible for executing specific UI operations, while LLM acts as the "command brain," dynamically guiding the exploration direction. During the exploration process, the system records a series of GUI operation sequences, such as: (Page A → Click button b1 → Page B → Input text t1 → …). The above low-level exploration trajectory is then structured into a GUI page transition graph. In UTG, each node represents a unique static GUI page state, usually identified by page structure, hierarchical relationships, and component hashes; each edge corresponds to an executable interactive operation from one page state to another, such as clicking, swiping, inputting, or navigation back.

[0025] (2) LLM Dynamic Guidance and Semantic Analysis: At each step of the exploration loop, when the exploration tool discovers a new page or reaches a key state node, the LLM intervenes immediately and performs the following tasks: a. Semantic analysis and priority ranking of GUI components: LLM performs real-time semantic analysis on the GUI components (buttons, input boxes, list items, etc.) of the current page, assesses their business importance, and dynamically sets priorities to guide the exploration algorithm to prioritize traversing paths with high business value.

[0026] b. High-semantic test input generation: For specific input fields that are identified (such as search, login, form), LLM dynamically generates high-semantic-value test inputs based on the context (e.g., valid account format, specific search keywords, dates that meet constraints).

[0027] c. Resource creation use case generation: LLM proactively generates specific, strongly business-related, multi-step operation sequences related to resource creation (e.g., "create a new schedule", "post a message", "add to cart") by reasoning context.

[0028] (3) Guided Exploration and Dynamic UTG Construction: The automated exploration tool executes specific operations based on the priorities, high semantic inputs, and resource creation use cases provided by the LLM. This effectively guides the exploration process to overcome bottlenecks that traditional exploration methods struggle to overcome (such as pages requiring specific input or functions requiring prior resource creation), thereby efficiently reaching the deep state of the application. During execution, all newly discovered GUI states (nodes) and successful state transitions (edges) are recorded in real time and the UTG is continuously and dynamically expanded and semantically enhanced in a structured manner. This "exploration-analysis-execution-expansion" cycle iterates until the preset coverage or time threshold is met.

[0029] Step S1 tightly integrates LLM with automated exploration, injecting LLM's semantic reasoning capabilities into the core of the exploration loop, thus transforming the traditional syntactic traversal into semantically guided exploration. This method significantly improves the accessibility to deep business states, and the final constructed UTG possesses high coverage breadth and high semantic depth, providing a high-quality foundational model for subsequent test unit decomposition and long sequence combination.

[0030] Step S2, test unit refinement and decomposition.

[0031] This step proposes an LLM-based test path decomposition method to automate test generation and execution for complex business scenarios. By decomposing test paths in the UTG into reusable test units and using LLM for dynamic combination and scheduling, this method can achieve multi-level, multi-scenario test logic composite construction, thereby improving the system's test coverage and scalability.

[0032] First, an automated test case extraction algorithm is applied to perform graph structure analysis on the semantically enhanced UTG. This algorithm, constrained by path simplicity and state independence, identifies and extracts all acyclic paths, with each path corresponding to a short test case with independent execution logic. During the extraction process, the system automatically deduplicates paths and normalizes states, ensuring logical independence and non-interference in execution between different short test cases. To guarantee consistency and reusability during composition, we define strict pre- and post-states for each short test case. The pre-state describes the page or logical environment the system must be in before the test case is executed, such as "User is logged in" or "Located on the product details page"; the post-state defines the target state the system should reach after the test case is executed, such as "Item quantity in shopping cart +1" or "Return to homepage and display login success message". Through this structured definition, short test cases can logically be regarded as the smallest atomic unit of state transition, possessing good reusability and composability.

[0033] Step S3: Semantic combination of long test sequences.

[0034] After completing the decomposition and definition of short test cases, LLM independently selects and sorts short test cases based on the UTG topology and state dependencies, without requiring manual or system-predefined test objectives. LLM uses the enhanced UTG as its semantic and structural foundation, autonomously reasoning and combining test path sequences with high exploratory value. During the combination process, LLM performs state matching and connection based on the preceding and following states of each short test case, dynamically forming a continuously executable test chain.

[0035] Step S4: Transition path generation and long sequence execution.

[0036] In this step, a method for automatically completing and executing long test sequences is proposed, addressing the connectivity, executability, and business logic consistency of the generated sequences. This method enhances the existing topology of the UTG by first completing the necessary transition operations between short test cases to form reachable paths. Then, an automated execution engine runs the complete sequence and collects logs of the execution process, thereby achieving closed-loop detection and evaluation of the end-to-end business process.

[0037] (1) Path reachability completion stage: Based on the semantically enhanced UTG topology information and the pre- and post-states of short test cases, the system automatically identifies and generates necessary transition operations between short test cases. The transition operations include, but are not limited to, cross-page jumps, context / session synchronization, necessary intermediate inputs or confirmation actions, and page condition waiting. By analyzing the connection relationships of nodes in the UTG, control semantics, and historical transfer trajectories, the system derives feasible intermediate operation sequences and verifies their connectivity and reachability within the graph, ensuring that the short test case sequence obtained by LLM combination constitutes a logically continuous and reachable long test path during actual execution.

[0038] (2) Automated execution phase: The system calls the automated execution engine (e.g., UIAutomator2) to execute each short test case and the transition operations between them step by step according to the complete test sequence composed of LLM and completed by the system. During the execution process, the system collects multi-dimensional runtime data in real time, including GUI state snapshots, control response events, page load latency, exception stack information and method calls, etc., to construct a detailed execution trajectory and time sequence record, which serves as the basis for subsequent evaluation and defect localization.

[0039] Step S5: Coverage assessment and defect discovery.

[0040] After the sequence execution is complete, the system first performs a business consistency verification of the endpoint state. This involves comparing the final actual state of the test sequence with the expected endpoint state defined based on business semantic reasoning or design specifications to determine whether the sequence has achieved the expected business objective (e.g., "remaining on the address entry page and displaying an error message when no address is entered" should be the expected endpoint, not "successful settlement" or "application crash"). We evaluate the effectiveness of the method using the following metrics: a. Method coverage: Measures the extent to which a long test sequence covers application method or function calls.

[0041] b. Activity coverage: Reflects the access range of the test sequence in different page or functional states.

[0042] c. Defect detection rate: Used to evaluate the ability of a long test sequence to trigger abnormal or inconsistent states, including logical errors, unresponsive operations, or unexpected states.

[0043] By analyzing the above indicators, the method of this invention can systematically verify the reachability and coherence of long test sequences in UTG, and provide a quantitative evaluation framework for deep business logic coverage and defect identification. While maintaining high semantic consistency and execution observability, this method achieves end-to-end automated test construction for complex business processes, significantly improving coverage of deep business logic that is difficult to reach with traditional testing, and more efficiently discovering defects triggered by complex business logic.

[0044] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An LLM semantic composition enhancement method for GUI-based automated exploratory testing, characterized in that, Includes the following steps: (1) Construct a semantically enhanced user transition graph (UTG) based on LLM; the nodes of the UTG are defined as the static page states of the GUI, and the edges are represented as executable user operations connecting different states; (2) The semantically enhanced UTG is refined, functionally equivalent state transitions are merged, redundant paths in the graph are identified and eliminated, and then the UTG after eliminating redundancy is decomposed into a series of minimal, state-independent, composable test units. (3) Using LLM as a state combiner, the decomposed test units are combined in a goal-oriented manner to generate long test sequences that simulate real user journeys and complex multi-step business workflows. (4) Complete the path of long test sequences to form a complete execution path, and automatically execute and monitor the complete execution path; (5) The coverage of deep business logic is evaluated by collecting logs and status data during the execution phase, and the ability to trigger application defects is evaluated by analyzing the deviation between the actual execution results and the expected behavior.

2. The LLM semantic composition enhancement method for GUI-oriented automated exploratory testing according to claim 1, characterized in that, In step (1), a semantically enhanced user transformation graph (UTG) is constructed based on LLM. The specific process is as follows: (1-1) A hybrid exploration loop is adopted, in which the automated exploration tool is responsible for performing specific UI operations, while the LLM dynamically guides the exploration direction; (1-2) At each step of the exploration loop, LLM performs semantic analysis and priority sorting on the GUI components of the newly discovered page, and dynamically generates test inputs and resource creation test cases with high semantic value. (1-3) The automated exploration tool performs specific operations based on the priority ordering provided by the LLM, test inputs with high semantic value, and resource creation test cases; during the execution process, the UTG is dynamically expanded and semantically enhanced in a structured manner. (1-4) Iterate the above steps until the preset coverage or time threshold is met, and finally obtain the semantically enhanced user conversion graph UTG.

3. The LLM semantic composition enhancement method for GUI-oriented automated exploratory testing according to claim 2, characterized in that, In steps (1-3), all newly discovered GUI nodes and edges are recorded in real time during the execution process.

4. The LLM semantic composition enhancement method for GUI-oriented automated exploratory testing according to claim 1, characterized in that, The specific process of step (2) is as follows: (2-1) Apply the automatic test case extraction algorithm to perform graph structure analysis on the semantically enhanced UTG, and identify and extract acyclic paths with path simplicity and state independence as constraints. (2-2) The extracted acyclic path is deduplicated and its state is normalized, and it is defined as the smallest atomic unit of state transition, i.e., the test unit. (2-3) Define the pre-state required for execution and the post-state after execution in a structured manner for each test unit to ensure its reusability and composability.

5. The LLM semantic composition enhancement method for GUI-oriented automated exploratory testing according to claim 4, characterized in that, In step (2-1), each acyclic path corresponds to a short test case with independent execution logic.

6. The LLM semantic composition enhancement method for GUI-oriented automated exploratory testing according to claim 1, characterized in that, The specific process of step (3) is as follows: (3-1) LLM uses semantically enhanced UTG as the semantic and structural basis, and performs state matching and connection based on the pre-state and post-state of each test unit; (3-2) Dynamically select and sort test units, and combine them into a test chain with high exploration value and continuous execution, forming a long test sequence.

7. The LLM semantic composition enhancement method for GUI-oriented automated exploratory testing according to claim 1, characterized in that, The specific process of step (4) is as follows: (4-1) Perform strict connectivity verification on the subsequent and preceding states of adjacent test units in a long test sequence; (4-2) If the states do not match, the necessary transition operations are automatically identified and injected based on the topology information, control semantics and historical transfer trajectory of the semantically enhanced UTG, so as to generate a logically coherent and state-synchronized complete execution path. (4-3) Call the automated execution engine to execute each test unit and the transition operations between them step by step according to the complete execution path; among them, the transition operations include cross-page jump, context synchronization, intermediate input or confirmation actions, and page condition waiting; (4-4) Collect GUI state snapshots, control response events, operation logs and exception stack information in real time to build a detailed execution trajectory.

8. The LLM semantic composition enhancement method for GUI-oriented automated exploratory testing according to claim 7, characterized in that, The specific process of step (5) is as follows: (5-1) Verify the business consistency of the endpoint state based on the execution trajectory, that is, compare the final actual state of the test sequence with the expected endpoint state based on business semantic reasoning; (5-2) Based on the collected logs and status data, conduct an in-depth evaluation of status coverage, transition coverage, and especially the coverage of deep business logic; (5-3) The ability to trigger application defects is assessed by analyzing the deviation between the actual execution results and the expected behavior.

Images