Automated Exploratory Testing Methods for Graphical User Interfaces Based on Large Language Models
By acquiring snapshots of the graphical user interface state and historical test records, combining them with a large language model to generate single-step test actions, and optimizing action generation through a verification and feedback mechanism, the problem of low test coverage and reliability under dynamic interface changes in existing technologies is solved, realizing intelligent, high-coverage, and high-reliability automated exploratory testing.
Patent Information
- Application Number
- CN202611612600.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing automated testing methods for graphical user interfaces struggle to achieve high coverage and reliable action generation when faced with dynamic interface changes and complex business pages. Furthermore, existing large language models fail to be effectively embedded into the real-time decision-making loop of exploratory testing.
By acquiring graphical user interface state snapshots and historical test records, key control sets and exploration boundaries are extracted, and single-step test actions are generated by combining a large language model. The action generation is optimized through verification and feedback mechanisms, including structured description of state snapshots, key control extraction, exploration boundary identification, action generation based on multi-source constraint information, and alternative test action mechanisms.
It achieves intelligent, high-coverage, and high-reliability automated exploration testing in complex interface scenarios, solving the problem of balancing the executability of large language model generation actions and the sufficiency of exploration.
Smart Images

Figure CN123114985A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of software automation testing technology, and more specifically, to an automated exploratory testing method for graphical user interfaces based on a large language model. Background Technology
[0002] Automated testing of graphical user interfaces (GUIs) is a crucial aspect of ensuring software quality. Traditional GUI automation testing methods primarily include script-driven testing and record-and-playback testing. Script-driven testing relies on testers pre-writing test scripts, specifying operation steps, target controls, and expected results one by one. This results in high maintenance costs, and the scripts become invalid when the interface layout or interaction flow changes. Record-and-playback testing lowers the barrier to entry by recording and replaying manual operations, but it cannot proactively explore uncovered interaction paths. Furthermore, while exploration testing methods based on random strategies or model traversal do not require predefined scripts and can automatically traverse interface controls, their exploration process lacks an understanding of the interface semantics, often generating a large number of invalid operations. In complex business pages, this results in low coverage and low efficiency.
[0003] With the significant capabilities of large language models in natural language understanding and reasoning, they have been explored for use in assisting in test script generation. However, existing solutions mostly use large language models for single-cycle generation or post-analysis, failing to deeply embed them into the real-time decision-making loop of exploratory testing. This fails to resolve the contradiction between the reliability of large language model generation and the adequacy of exploration, and also makes it difficult to cope with the continuous adjustment of test behavior in scenarios with dynamically changing interfaces. To address these issues, existing technologies urgently need improvement. Summary of the Invention
[0004] The purpose of this application is to provide an automated exploration and testing method for graphical user interfaces based on a large language model. This method has the advantages of enabling intelligent interface exploration and testing driven by a large language model, thereby improving test coverage and the reliability of action generation.
[0005] This application provides an automated exploration and testing method for graphical user interfaces based on a large language model, the technical solution of which is as follows: Obtain a snapshot of the graphical user interface state and historical test records of the application under test; extract a set of key controls based on the state snapshot; and identify a set of exploration boundaries based on the state snapshot and the historical test records. Generate current action constraint information based on the state snapshot and the historical test records; Input the state snapshot, the set of key controls, the set of exploration boundaries, the current action constraint information, and the historical test records into the large language model to generate a single-step test action; The single-step test action is validated; if the validation fails, a rejection record is generated and fed back to the input of the large language model to regenerate the single-step test action; if the number of consecutive rejections for the current state snapshot reaches a threshold, an alternative test action is generated according to a preset priority. Execute the single-step test action or the alternative test action, collect a state snapshot after execution, compare the state snapshots before and after execution, determine whether a page state transition is triggered, and update the historical test record based on the comparison result.
[0006] Furthermore, the state snapshot includes the control's unique identifier, control text, control type, control hierarchy path, control coordinates, control size, visibility state, enabled state, focus state, pop-up state, input value, error message, and page hierarchy information.
[0007] Furthermore, the extracted key control set includes: The controls are obtained from the state snapshot. Invisible and obscured controls are removed, disabled and non-interactive controls are filtered out, and non-interactive controls used only for page display are filtered out to obtain a candidate control set. The buttons, input boxes, checkboxes, radio buttons, drop-down lists, menu items, tabs, links, list items, pagination controls, scroll containers, pop-up buttons, and error message areas in the candidate control set are mapped to a unified control description to obtain the key control set; The set of exploration boundaries to be identified includes: identifying exploration boundaries based on access history, page structure changes, new control status, scroll position, and page navigation relationships.
[0008] Furthermore, the extraction of the key control set also includes: labeling each key control with its function, candidate actions, prerequisites, and coverage status.
[0009] Furthermore, the historical test records include recent test records from multiple rounds and long-term test summaries; The recent multi-round test records include action descriptions, target controls, action parameters, pre-execution state summaries, post-execution state summaries, feedback logs, execution results, and correction suggestions for the most recent rounds of tests. The long-term test summary is stored in segments based on the current state, recent important events, early milestones, persistent failure records, and the focus of the next test, and is incrementally updated after each new recent multi-round test record is added.
[0010] Furthermore, generating current action constraint information based on the state snapshot and the historical test records includes: When the status snapshot indicates the presence of a blocking pop-up, the current action constraint information includes the single-step test action determining the target within the controls of the blocking pop-up; When the state snapshot indicates the existence of a control with an empty input value, the current action constraint information includes that the single-step test action targets the control and the action type is fill-in; When the state snapshot indicates that there is a control that is in focus and has an input value, the current action constraint information includes that the single-step test action targets the control and the action type is submit, save, continue input or cancel; When the historical test record indicates that the control in the state snapshot did not trigger a page state transition, the current action constraint information includes that the single-step test action does not target that control; When the historical test record indicates that the state snapshot before and after execution has no change in multiple consecutive rounds of comparison, the current action constraint information includes a loop detection warning. The loop detection warning includes the single-step test action targeting controls corresponding to different page identifiers or different control types. When multiple conditions are met at the same time and the corresponding constraints conflict, the constraint priority is determined according to the order of the conditions listed above.
[0011] Furthermore, the verification of the single-step test action includes: Verify whether the single-step test action is empty and whether the action type is supported; Verify whether the control identifier corresponding to the single-step test action exists in the state snapshot, and whether the visible and enabled states of the control identifier in the state snapshot indicate that the control can be operated; Compare whether the action parameters of the single-step test action match the control type corresponding to the control identifier in the state snapshot, and whether the action parameters meet the format requirements; Verify whether the single-step test action conflicts with the current action constraint information, and whether the action has been recorded in the rejection record or whether there is an action record in the historical test record that targets the same control and does not trigger a page state transition.
[0012] Furthermore, the rejection record includes the rejected action, the reason for rejection, and a correction suggestion; the large language model regenerates the single-step test action based on the rejection reason and correction suggestion in the rejection record.
[0013] Furthermore, the threshold is dynamically set based on the density of historical invalid actions; The preset priorities are as follows: handling blocking pop-ups, filling in required input fields, correcting error message related fields, submitting the currently valid form, accessing uncovered navigation entries, expanding unaccessed folded areas, scrolling to low-coverage areas, entering the next page of a pagination or list, and selecting interactive controls that are different from recently invalid actions. The generation of the alternative test action includes: determining candidate actions according to the preset priority, and using scoring rules to score the candidate actions based on their comprehensive action advancement value, coverage value, safety and executability, and repetition penalty, and selecting the candidate action with the highest score as the alternative test action.
[0014] Furthermore, the comparison of state snapshots before and after execution includes comparing page identifiers, window or pop-up states, control sets, control properties, focus states, input values, error messages, success messages, list data, navigation paths, and scroll positions. When the differential result indicates a page jump, the addition or disappearance of a control, a change in the error message, a successful data submission, the opening or closing of a pop-up window, a change in list data, or an expansion of the coverage area, a page state transition is triggered. When the difference result is empty or there are only non-functional changes such as timestamp updates, it is determined that the page state transition has not been triggered, and a correction suggestion is generated and written into the historical test record.
[0015] As can be seen from the above, the automated exploration testing method for graphical user interfaces based on a large language model provided in this application solves the problem of balancing the executability of actions generated by the large language model and the sufficiency of exploration by embedding the large language model into the real-time decision-making closed loop of the test, combined with the structured description of state snapshots, key control extraction and exploration boundary identification, action generation guidance mechanism based on multi-source constraint information, trial and error correction closed loop of deterministic rule verification and rejection feedback, and alternative test action generation mechanism. It has the advantages of achieving intelligent, high-coverage, and highly reliable automated exploration testing in complex interface scenarios. Attached Figure Description
[0016] The above and other objects, features, and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0017] Figure 1 This is a flowchart illustrating the steps of the automated exploration and testing method for a graphical user interface based on a large language model disclosed in an embodiment of this application. Figure 2 This is a flowchart illustrating the steps of the method for extracting a set of key controls and identifying a set of exploration boundaries disclosed in an embodiment of this application. Figure 3 This is a flowchart illustrating the method steps for verifying single-step test actions disclosed in an embodiment of this application. Detailed Implementation
[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments belong; the terminology used herein and in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit these embodiments; the terms "comprising" and "having," and any variations thereof, in the specification of these embodiments and the foregoing drawings, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification of these embodiments and the foregoing drawings are used to distinguish different objects, not to describe a particular order.
[0019] The implementation details of the technical solution in this embodiment are described in detail below: This application proposes an automated exploratory testing method for graphical user interfaces based on a large language model, such as... Figure 1 As shown, the method includes: S101, Obtain a snapshot of the graphical user interface state and historical test records of the application under test, extract a set of key controls based on the state snapshot, and identify a set of exploration boundaries based on the state snapshot and the historical test records; In practice, at the beginning of each test cycle, the testing system collects the raw control tree and page metadata of the current interface of the application under test through an automated driver interface or an accessibility interface, and converts the raw data from different platforms into a structured state snapshot. Simultaneously, it reads historical test records associated with the current test task from local or remote storage. The state snapshot reflects the objective state of the interface of the application under test at the current moment. The historical test records include recent multi-round test records and long-term test summaries, both of which serve as input for the current test cycle. Based on the state snapshot, the testing system reduces the complete control set to a set of key controls through visibility filtering, interactivity filtering, and semantic importance filtering. Based on the state snapshot and historical test records, it identifies unvisited page entry points, low-coverage area entry points, pagination entry points, scrolling area entry points, and collapsed area entry points from access history, page structure changes, control addition states, scroll positions, and page jump relationships. This constitutes an exploration boundary set, which guides the subsequent action generation module to prioritize targets with coverage expansion value.
[0020] Furthermore, the state snapshot includes the control's unique identifier, control text, control type, control hierarchy path, control coordinates, control size, visibility state, enabled state, focus state, pop-up state, input value, error message, and page hierarchy information.
[0021] In practical implementation, state snapshots use a unified data structure to standardize the description of the original interface information. The unique identifier for a control can be a resource identifier or a hash value based on control attributes. The control text is the currently displayed text. The control type describes the interaction category, such as a button or input box. The control hierarchy path is the sequence of nodes from the root node to the target control. Control coordinates and dimensions are represented in pixels or proportional coordinates. The visibility state indicates whether the control is rendered within the visible area. The enabled state indicates whether the control is interactive rather than grayed out or read-only. The focus state indicates whether the control has input focus. The pop-up state indicates whether there are modal or non-modal pop-ups and their occlusion relationships on the current interface. The input value records the current value of the input control. The error message records validation errors or exception messages. Page hierarchy information includes the page identifier, window identifier, and page hierarchy depth. The pseudocode for constructing a state snapshot is as follows: Iterate through each control in the original interface data, extract the identifier, text, type, coordinates, size, and hierarchical path, determine visibility, enabled state, and focus state, identify the context of the pop-up window and extract the input value and associated error message, assemble the above information into a standardized control description object; after all controls have been processed, combine them into a control list, extract the page identifier, pop-up state, and page hierarchy depth, and encapsulate it into a state snapshot output.
[0022] Furthermore, the extraction of the key control set, such as Figure 2 As shown, it includes: S2001, Obtain controls from the state snapshot, remove invisible and obscured controls, filter out disabled and non-interactive controls, and filter out non-interactive controls used only for page display, to obtain a candidate control set. In practice, the first stage of key control extraction is the generation of a candidate control set. The test system retrieves all controls from the state snapshot and performs three layers of filtering sequentially.
[0023] In practical implementation, the first layer is visibility filtering: controls whose visible state is invisible are directly removed; for visible controls, the occlusion situation is further checked. Let the rectangular area of control i be Ri, and the rectangular area of the pop-up be Rp. When the occlusion rate (the intersection of the area Ri and Rp divided by the area Ri) is greater than a preset occlusion threshold α, the control is removed. α is usually taken as 0.5 to 1.0. The second layer is interactivity filtering: controls whose enabled state is disabled and read-only or other non-interactive controls are removed. The third layer is semantic importance filtering, which aims to filter out controls without interactive functions. Specifically, it is determined control by control according to the following rules: Rule 1 (Plain Text Label Determination). If a control's type is text or label, and the control is not associated with any interactive event properties in the original control tree (such as onclick, onfocus, etc. event handlers in the Web platform, the clickable property not being true in the Android platform, and isAccessibilityElement being true and having no accessibility actions in the iOS platform), then the control is determined to be a plain text label and will be filtered out. If a text or label control is associated with interactive event properties, then it is determined to be an interactive text control and will be retained.
[0024] Rule 2 (Determination of Decorative Images). If a control is an image and simultaneously meets all three of the following conditions, it is considered a decorative image: the control is not associated with any interactive event attributes; the control text is an empty string; and the control does not contain any nested interactive child controls in its control hierarchy path. Image controls that do not meet any of the above conditions are considered functional icons and will be retained.
[0025] Rule 3 (Static Layout Container Determination). If a control is a container or group, and its subtree rooted at the control does not contain any child controls identified as interactive or functional in Rules 1 and 2, then the control is determined to be a static layout container and is filtered out. If interactive child controls exist in the subtree, the container control is retained to ensure the integrity of its child control's hierarchical path. Simultaneously, the container control's control type is marked as a penetrating container—meaning that the container itself does not participate in action generation as a target control in subsequent steps, but its child controls participate normally.
[0026] In addition, semantic importance filtering maintains an exemption list based on control type tags. Control types on the exemption list are directly retained if the interactivity filtering has passed, without triggering the judgment of rules one through three. Example entries on the exemption list include: button, input, select, textarea, and a elements on the web platform; android.widget.Button, android.widget.EditText, android.widget.CheckBox, and android.widget.Spinner on the Android platform; and UIButton, UITextField, UISwitch, and UISegmentedControl on the iOS platform. The exemption list can be dynamically configured according to the target platform of the application under test.
[0027] S2002, the buttons, input boxes, check boxes, radio buttons, drop-down boxes, menu items, tabs, links, list items, pagination controls, scroll containers, pop-up buttons, and error message areas in the candidate control set are mapped to a unified control description to obtain the key control set; In practice, buttons and pop-up buttons are mapped to submit or confirm controls, input boxes to inputtable controls, checkboxes to toggle controls, radio buttons to selectable controls, dropdown lists to expandable selection controls, menu items to navigation controls, tabs to page switching controls, links to page navigation controls, list items to detail entry controls, pagination controls to pagination navigation controls, scrollable containers to scrollable area controls, and error message areas to feedback display controls. Each unified control description includes a unique control identifier, a unified control type, control text, control coordinates, and control hierarchy path. Through this mapping, heterogeneous controls across different platforms are converted into semantically consistent description objects.
[0028] S2003, the identification of the exploration boundary set includes: identifying the exploration boundary based on access history, page structure changes, control addition status, scroll position and page jump relationship.
[0029] In practice, the system retrieves the set of visited page identifiers and the set of manipulated control identifiers from historical test records. Navigation entries, tabs, and links not appearing in the visited page identifier set in the current state snapshot are identified as unvisited page entry boundaries. Based on the current page number p and the total number of pages P of the pagination control, the next page button is identified as a pagination entry boundary when p is less than P. For scrolling areas, unreached areas are determined based on the scroll position and identified as scrolling area boundaries. For collapsed areas, unexplored collapsed area boundaries are determined based on historical expansion records. For page structure changes, the system compares the differences between the control sets in the current and previous state snapshots and marks the areas associated with newly added controls as possible exploration boundaries. For page jump relationships, a page jump graph is constructed based on historical jump records, and the entry controls corresponding to unexplored branches are identified as exploration boundaries.
[0030] The coverage value of exploring boundary j is Vj = w1 × (1 The formula is: Cj) + w2 × D'j + w3 × N'j, where each parameter is defined as follows: Cj is the coverage rate, which is the ratio of the number of accessed controls associated with this boundary in historical test records to the total number of controls associated with the boundary. Cj ranges from 0 to 1. D'j is the normalized level depth, D'j = Dj / Dmax, where Dj is the level depth of the page corresponding to this boundary in the page transition graph (starting from the root node level of 0 in the page transition graph), and Dmax is the maximum level depth among all explored boundaries. N'j is the normalized number of newly added controls, N'j = Nj / Nmax, where Nj is the number of newly added controls associated with this boundary in the current state snapshot (i.e., the number of controls that exist in the current state snapshot but have not appeared in previous state snapshots), and Nmax is the maximum value of Nj among all explored boundaries. w1, w2, and w3 are weight coefficients, satisfying w1 + w2 + w3 = 1. A larger Vj indicates that the boundary is more worthy of priority exploration. When all exploration boundaries Dmax or Nmax are zero (e.g., there is no historical comparison data in the first round of testing), the corresponding normalized components D'j or N'j are set to 0, and Vj is calculated only based on the defined components.
[0031] Furthermore, the extraction of the key control set also includes: labeling each key control with its function, candidate actions, prerequisites, and coverage status.
[0032] In practice, after generating a set of key controls, the testing system adds four semantic annotations to each key control.
[0033] The control function description refers to the functional role of the control in the interface interaction, which is inferred from the control's unified control type and control text. For example, the function of the submit button is to submit a form, the function of the input box is to input text, the function of the navigation link is to jump to a new page, the function of the collapse control is to expand or collapse the content area, and the function of the pop-up close button is to close the current pop-up.
[0034] The candidate actions list the set of action types supported by the control, determined based on the control's uniform control type. For example, the candidate actions for a submit control are click, for an input control are enter text and clear content, for a toggle control are check and decheck, for an expandable selection control are expand list and select option, and for a scrollable area control are scroll up and scroll down.
[0035] Preconditions describe the conditions that must be met before operating the control, and are determined based on the control's context and current state snapshot. For example, the precondition for a submit button is that all required input fields are filled in and the agreement checkbox is checked; the precondition for a control within a pop-up window is that the pop-up window is open and not closed; and the precondition for the next page button in a pagination control is that the current page number p has not reached the total number of pages p.
[0036] The coverage state describes the operation status of the control in the historical test record, including whether it has been operated on, the number of historical operations, the round number of the most recent operation, and whether the page state transition was triggered after the operation. The coverage state is extracted from the historical test record and updated in real time with each round of testing.
[0037] By labeling each key control with the above four pieces of information, the large language model can simultaneously perceive the semantic function of the control, the set of executable actions, the operation constraints, and the historical coverage when generating single-step test actions, thereby generating more targeted and valuable test actions. At the same time, the rule verification module can also perform constraint verification on the actions based on the preconditions and coverage status.
[0038] Furthermore, the historical test records include recent test records from multiple rounds and long-term test summaries; The recent multi-round test records include action descriptions, target controls, action parameters, pre-execution state summaries, post-execution state summaries, feedback logs, execution results, and correction suggestions for the most recent rounds of tests. The long-term test summary is stored in segments based on the current state, recent important events, early milestones, persistent failure records, and the focus of the next test, and is incrementally updated after each new recent multi-round test record is added.
[0039] In practice, historical test records are stored in a hierarchical structure, including two levels: recent multi-round test records and long-term test summaries.
[0040] The recent multi-round test log stores fine-grained execution information from the most recent K rounds of tests, where K is a positive integer that can be set according to the test task scale and the capacity of the large language model context window. Each round of test logs includes the following fields: Action Description: Records the type and intent of the action executed in this round; Target Control: Records the unique identifier and text of the key control that the action affects; Action Parameters: Records the specific parameters of the action, such as input text content, selection value, or scroll distance; Pre-Execution State Summary: Records the key features of the snapshot of the state before the action execution, including page identifier, number of visible controls, pop-up state, and error message content; Post-Execution State Summary: Records the corresponding key features of the snapshot of the state after the action execution; Feedback Log: Records the status codes, exception information, and time consumed returned by the automated driver interface during the action execution; Execution Result: Records whether the action was successfully executed and whether it triggered a page state transition; Correction Suggestions: Records directional guidance information provided for the next round of testing based on the execution result and the difference between the pre- and post-execution states. The recent multi-round test logs are managed using a first-in, first-out sliding window strategy. When the number of test rounds exceeds K, the oldest record is removed from the recent multi-round test logs and participates in the incremental update of the long-term test summary.
[0041] The long-term test summary is stored in segments, organized into five sections: Current Status, Recent Important Events, Early Milestones, Persistent Failure Records, and Next Test Focus. The Current Status section summarizes the current test progress of the application under test, including the number of pages covered, the number of controls manipulated, and the current page identifier. The Recent Important Events section records milestone events from recent rounds, such as the first entry into a new page, the first triggering of a pop-up, or the first discovery of an error message. The Early Milestones section records important exploration paths and verified functional flows completed in the early stages of testing. The Persistent Failure Records section records operations that failed after multiple attempts and their possible causes, preventing the system from repeatedly trying known ineffective operations. The Next Test Focus section records the subsequent exploration direction determined based on the current progress and known issues, guiding the large language model to prioritize actions with advancement value.
[0042] The incremental update process for long-term test summaries is as follows. When a new record is added to the recent multi-round test logs, the test system first determines whether the record contains important events or persistent failures. If the execution result in the record is the first triggering of a page jump or the first appearance of a new error message, the corresponding information is merged into the recent important events section. If the cumulative number of failures for a certain operation in the record reaches the preset failure count threshold F, the operation and its possible causes are written into the persistent failure record section. Then, the test system updates the current state section and the next test focus section based on the overall execution trend of the recent multi-round test logs. Specifically, if the proportion of interactive controls on a certain page that have been operated in the recent multi-round test logs reaches the preset coverage threshold β, the priority of that page is reduced in the next test focus section; if a control is operated for M consecutive rounds without triggering a page state transition, the control is marked as a low-value control and its priority as an alternative exploration direction is increased in the next test focus section. The failure count threshold F, coverage threshold β, and consecutive invalid rounds threshold M are all positive parameters that can be set according to the test task requirements. Through the aforementioned incremental update mechanism, long-term test summaries can continuously reflect the latest test progress, while avoiding excessively long inputs to large language models due to the infinite growth of historical records, thus maintaining the effectiveness and compactness of contextual information in long-process exploratory tests.
[0043] S102, Generate current action constraint information based on the state snapshot and the historical test record.
[0044] In practice, the current action constraint information maps the objective facts of the interface in the state snapshot and the execution experience in the historical test records into mandatory boundary conditions that must be followed in a single test action. Its generation relies on the pop-up state, input value, focus state field of the state snapshot, as well as the execution result and state difference field of the historical test records, and includes a total of five constraints.
[0045] Furthermore, the step of generating current action constraint information based on the state snapshot and the historical test records includes: (1) When the state snapshot indicates the existence of a blocking pop-up, the current action constraint information includes the single-step test action determining the target in the control within the blocking pop-up.
[0046] The first type is the blocking pop-up constraint. This is triggered when the pop-up status field of the status snapshot indicates the presence of a modal pop-up or overlay obscuring the underlying page. The constraint is: single-step test actions should target controls within the control hierarchy path of the blocking pop-up; controls on the underlying page are excluded from the candidate scope. For example, in a delete confirmation pop-up scenario, the action must target the confirm or cancel button within the pop-up.
[0047] (2) When the state snapshot indicates that there is a control with an empty input value, the current action constraint information includes that the single-step test action targets the control and the action type is fill; The second type is the required input constraint. It iterates through key controls of type "inputable" in the state snapshot, triggering when the input value field is empty. The constraint content is: the action should target this control and the action type should be "filled in". If multiple empty controls exist, multiple constraints are generated in the order they are listed.
[0048] (3) When the state snapshot indicates that there is a control that is in focus and has an input value, the current action constraint information includes the single-step test action targeting the control and the action type being submit, save, continue input or cancel; The third type is focus control constraint. This is triggered when the state snapshot indicates the presence of a control with true focus and a non-empty input value. The constraint states that actions should target this control and the action type should be limited to submit, save, continue input, or cancel to avoid context loss due to mid-input jumps.
[0049] (4) When the historical test record indicates that the control in the state snapshot has not triggered the page state transition, the current action constraint information includes that the single-step test action does not target the control; The fourth type is the invalid target exclusion constraint. Let Nc be the total number of times control c has been operated on in recent multi-round records, and Fc be the number of times a page state transition was not triggered. Then the invalid operation ratio Rc = Fc / Nc. An invalid operation is triggered when Rc is greater than a preset invalid threshold δ, where δ ranges from 0 to 1. The constraint is: actions should not target this control to avoid repeated operations on known invalid controls.
[0050] (5) When the historical test record indicates that the state snapshot before and after execution has no change in multiple consecutive rounds of comparison, the current action constraint information includes a loop detection warning. The loop detection warning includes the single-step test action targeting controls corresponding to different page identifiers or different control types. When the above conditions are met at the same time and the corresponding constraints conflict, the constraint priority is determined according to the order of the above conditions.
[0051] The fifth type is a loop detection warning. This is triggered when the state difference results for the most recent L consecutive rounds are all empty or only non-functional changes exist. L typically ranges from 3 to 10. The constraint is that actions should target controls corresponding to different page identifiers or different control types, guiding the system to break out of local loops.
[0052] When multiple constraints are triggered simultaneously and conflict, priority is determined in the following order: blocking pop-up constraints take precedence over required input constraints, required input constraints take precedence over focused control constraints, focused control constraints take precedence over invalid target exclusion constraints, and invalid target exclusion constraints take precedence over loop detection warnings. For example, when blocking pop-ups and loop detection warnings occur concurrently, blocking pop-up constraints take precedence, and page redirection requirements in loop detection are temporarily disabled.
[0053] The current action constraint information generation process is as follows: First, initialize the constraint list to empty. Second, read the pop-up status field; if a blocking pop-up exists, generate the corresponding constraint entry (priority 1). Third, iterate through input controls; generate mandatory input constraints for controls with empty input values (priority 2). Fourth, query the focus state; if a control has a focus state and a non-empty input value, generate a focus control constraint (priority 3). Fifth, iterate through key controls and query the Rc of each control; if Rc is greater than δ, generate an invalid target exclusion constraint (priority 4). Sixth, obtain the most recent L rounds of difference results from historical test records; if there are no functional changes for L consecutive rounds, generate a loop detection warning (priority 5). Seventh, check for conflicts in the constraint list; retain only the highest priority constraint entry and discard conflicting lower priority entries. Eighth, encapsulate the current action constraint information as output, serving as a high-priority component of the subsequent large language model input.
[0054] In this embodiment, the large language model can be a pre-trained model with natural language understanding and generation capabilities, such as the GPT series models or open-source large language models, which can receive structured prompt word input and return structured action output through API calls or local inference.
[0055] S103, input the state snapshot, the key control set, the exploration boundary set, the current action constraint information and the historical test record into the large language model to generate a single-step test action.
[0056] In its implementation, S103 organizes the five types of multi-source information generated in the preceding steps into a unified input format, which is then used by the large language model to generate structured single-step test actions. The organization of input information follows a preset priority ordering rule: the current action constraint information has the highest priority and is placed at the beginning of the prompt, ensuring that the large language model perceives mandatory boundary conditions first; the state snapshot follows immediately, providing a complete and objective description of the current interface; the key control set and exploration boundary set are arranged sequentially to provide optional controls and coverage directions; historical test records are placed at the end, with recent multi-round records arranged in reverse chronological order, and long-term summaries arranged in five text segments at the end. Since the large language model typically gives higher attention weight to the preceding information, this arrangement makes constraints and the current state easier to follow, with historical experience serving as a supplementary reference.
[0057] The input organization also needs to be included in token budget management. Before calculating the token count for each component, the token count estimation method must first be determined. There are two possible methods for token count estimation. If the large language model provider exposes a segmenter interface or segmenter library, the token count is obtained by encoding the text using that segmenter and then calculating the length of the encoded sequence. If the segmenter interface is unavailable (e.g., via API call and the API does not return a token count), an approximate estimation using text length is used: Token count = ceil(Number of Chinese characters × λcn + Number of English characters × λen + Number of digits and punctuation marks × λnum), where λcn, λen, and λnum are empirical coefficients based on the segmentation weights of common large language models. An example set of values is λcn = 1.5 (each Chinese character occupies an average of 1.5 tokens), λen = 0.3 (each English character occupies an average of 0.3 tokens), and λnum = 0.5 (each digit or punctuation mark occupies an average of 0.5 tokens). The number of Chinese characters refers to the number of characters with Unicode code points located in the CJK Unified Ideographs area (U+4E00 to U+9FFF) and its extended areas. The number of English characters refers to the number of characters with the letters AZ and az. The remaining characters are included in the number of numbers and punctuation marks. In scenarios where prompts are mainly in Chinese, this estimation method typically has an error of less than ±10% compared to the actual number of tokens, which is sufficient to meet the accuracy requirements of token budget management.
[0058] Let Tmax be the maximum number of input tokens for the large language model. The number of tokens estimated in the state snapshot, key control set, exploration boundary set, current action constraint information, recent multi-round records, and long-term summary as described above are Ts, Tc, Te, Ta, Tr, and Tl, respectively, with the constraint Ts + Tc + Te + Ta + Tr + Tl ≤ Tmax. When this limit is exceeded, a three-level truncation is performed: the first level compresses the long-term summary, retaining only the current state and the next test focus; the second level reduces the number of rounds K retained in the recent multi-round records to K' = max(1, ceil(K × Tmax / T)), where T is the original estimated total number of tokens before truncation; the third level simplifies each control in the state snapshot and key controls to retain only the control identifier, control type, control text, visible state, enabled state, and overridden state.
[0059] The structured single-step test actions generated by the large language model contain five fields. The action type field supports click, input, selection, submit, return, close pop-up, confirm, cancel, scroll, expand, collapse, drag, wait, and page navigation. The target control location field includes the control's unique identifier, control hierarchy path, and control text. The action parameter field is differentiated by action type: text content for input, direction and distance for scrolling, option value or index for selection, and condition or duration for waiting. The action reason field records the decision basis in brief text, and the expected interface change field describes the expected state change after execution, such as page jump or pop-up closure. The validation module and action execution bridging module read the required fields from the structured output and complete subsequent processing, eliminating the need for repeated parsing or secondary conversion.
[0060] The randomness of the output of the large language model is controlled by a temperature parameter τ, which ranges from 0 to 1. To achieve a dynamic balance between executability and exploration breadth, a dynamic temperature adjustment based on rejection records is adopted. Let the current cumulative number of rejections be Rr, and the historical average number of rejections per round be Ravg. When Rr is greater than Ravg, the step size of τ is decreased by Δτ to reduce randomness; when Rr is less than or equal to Ravg, τ is increased by Δτ to increase diversity. Δτ ranges from 0.05 to 0.2.
[0061] The execution flow of S103 is as follows: First, construct prompt words according to priority. Second, calculate the number of tokens T; if it exceeds Tmax, compress it using a three-level truncation method. Third, set the current τ value and call the large language model to generate the output text. Fourth, parse the output into a structured action A, extracting the action type At, target control location Ac, action parameter Ap, action reason Ar, and expected interface change Ae. If any required field is missing, it is marked as a failure and written to the rejection record. Fifth, pass the output A to the rule validation module.
[0062] S104, the single-step test action is verified; if the verification fails, a rejection record is generated and fed back to the input of the large language model to regenerate the single-step test action; if the number of consecutive rejections for the current state snapshot reaches a threshold, an alternative test action is generated according to a preset priority.
[0063] In practice, S104 is located after the large language model generates actions but before their execution. It checks each action one by one using multi-layered deterministic rules. If any layer of verification fails, a rejection record containing the rejection reason and correction suggestions is generated and fed back into the large language model for the next round of input to regenerate it. If the number of consecutive rejections under the same state snapshot reaches a dynamic threshold, the system no longer waits for the large language model and directly generates alternative test actions according to preset priorities and scoring rules.
[0064] Furthermore, the verification of the single-step test action, such as... Figure 3 As shown, it includes the following four sub-steps. Specifically, it includes: S3001, Verify whether the single-step test action is empty and whether the action type is supported; In practice, S3001 serves as the first filter in the validation process. First, it checks whether the single-step test action itself is null or an invalid output that failed to parse. If the large language model returns null output due to an anomaly, or if the structured action obtained after parsing the output text is an empty object, the validation fails directly. Second, it checks whether the action type field belongs to the system's predefined set of supported action types. The system's predefined set of supported action types includes click, input, selection, submit, return, close pop-up, confirm, cancel, scroll, expand, collapse, drag, wait, and page navigation, totaling fourteen types. Let action A have action type At, and the supported action type set be Stype. If Att is null or does not belong to Stype, the S3001 validation fails. The rejection reason is recorded as action type null or action type not supported, and the suggested correction is to generate a structured action containing a valid action type.
[0065] S3002, check whether the control identifier corresponding to the single-step test action exists in the state snapshot, and whether the visible state and enabled state of the control identifier in the state snapshot indicate that the control can be operated; In its implementation, S3002 performs a joint check on the existence and operability of a control. First, the unique identifier of the control is extracted from the target control location information field of the single-step test action, and this identifier is matched against the control list in the current state snapshot. If the unique identifier of the control does not exist in the control list of the current state snapshot, possible reasons include the large language model referencing a control from an invalid page or generating a non-existent control identifier out of thin air, resulting in a failed check. Let the set of control identifiers in the current state snapshot be Sid, and the unique identifier of the target control be idtarget. If idtarget does not belong to Sid, the rejection reason is that the target control does not exist in the current state snapshot. If the control identifier exists, the visible state field and enabled state field of the control in the current state snapshot are further read. The visible state must be visible, and the enabled state must be enabled; both conditions must be met to determine that the target control is operable in the current interface. If the visible state is invisible, the rejection reason is that the target control is currently invisible; if the visible state is visible but the enabled state is disabled, the rejection reason is that the target control is currently disabled. A suggested correction is to select a currently visible and enabled control as the target.
[0066] S3003, compare whether the action parameters of the single-step test action match the control type corresponding to the control identifier in the state snapshot, and whether the action parameters meet the format requirements; In practical implementation, S3003 is for verifying the compatibility between action parameters and control types. Different control types support different action types and parameter combinations. The test system maintains a valid mapping table between action types and control types. This mapping table records the set of valid action types supported by each control type and the parameter format constraints corresponding to each action type. An example of a valid mapping table is as follows: Submission controls: Click (no parameters), Wait (parameters are waiting conditions such as text appearance or duration, in milliseconds).
[0067] Input control: Input (parameter is a non-empty string), Clear (parameter is empty).
[0068] Toggleable control: Check (no parameters), Uncheck (no parameters).
[0069] Selectable control: Select (parameter is the option index, a non-negative integer).
[0070] Expandable selection control: Expand (no parameter), Select (parameter is the option value, which must belong to the candidate option set of this control), Collapse (no parameter).
[0071] Navigation controls: Click (no parameters).
[0072] Page switching control: Click (no parameters).
[0073] Page navigation control: Click (no parameters).
[0074] Details entry control: Click (parameters empty).
[0075] Pagination navigation control: Click (parameter empty).
[0076] Scrollable area control: Scroll up (parameter is a positive number or percentage value representing the scroll distance), Scroll down (parameter is a positive number or percentage value representing the scroll distance), Scroll left (parameter is the same as above), Scroll right (parameter is the same as above).
[0077] Feedback display controls: Waiting for disappearance (parameter is the waiting time), Click to close (parameter is empty).
[0078] Pop-up control: Click (no parameters).
[0079] Let the control type be Ct, the action type be At, and the action parameter be Ap. The valid combinations of Ct and At in the valid mapping table are denoted as M(Ct, At). When M(Ct, At) is empty, it indicates that the action type is incompatible with the control type, and the validation fails. When M(Ct, At) is not empty, further validation is performed to ensure that Ap conforms to the parameter format constraints corresponding to M(Ct, At). For example, for an input control paired with an input action, if Ap is an empty string, the validation fails because the input parameter cannot be empty. For a scrollable area control paired with an upward scrolling action, if Ap is a non-numeric string, the validation fails because the parameter format does not meet the requirements. The suggested correction in the rejection log is to match the valid action type and parameter format according to the control type.
[0080] S3004, verify whether the single-step test action conflicts with the current action constraint information, and whether the action has been recorded in the rejection record or whether there is an action record in the historical test record that targets the same control and does not trigger page state transition.
[0081] In practice, S3004 is a joint verification of semantic-level constraints and duplicate invalid operations, including three verification sub-items.
[0082] The first sub-item is constraint conflict verification. The target control and action type of the single-step test action are compared with each constraint entry in the current action constraint information C. If C contains a blocking pop-up constraint and the target control is not within the pop-up control hierarchy path, a conflict occurs. If C contains a required input constraint and the target control is not a null input control specified in the constraint entry, a conflict occurs. If C contains a focus control constraint and the action type is not one of submit, save, continue input, or cancel, a conflict occurs. If C contains an invalid target exclusion constraint and the target control belongs to the exclusion control list, a conflict occurs. When a constraint conflict occurs, the reason for rejection is that the action violates the current interface constraints, and the correction suggestion is to adjust the target control or action type according to the constraint content.
[0083] The second sub-item is rejection record duplication check. It checks whether the current single-step test action is duplicated in the rejection record set corresponding to the current state snapshot. Let Rset be the actions already recorded in the current rejection record set. If the action type and target control of the current action are exactly the same as the action type and target control of a rejected action in a record in Rset, then it is determined to be a duplicate rejection action. The reason for rejection is that the action has already been rejected, and the correction suggestion is to adjust according to the correction suggestions of the previous rejection record to avoid generating the same action again.
[0084] The third sub-item is historical invalid operation validation. It checks the recent multi-round records of the historical test logs to see if the current target control has a historical operation record. If it exists and the execution result of that historical operation record indicates that it did not trigger a page state transition, then the current action is considered a duplicate invalid operation. The reason for rejection is that the control's historical operation did not trigger a page state transition. The suggested correction is to select another control or verify whether the control's preconditions have been met.
[0085] The term "targeting the same control" refers to the target controls having the same unique identifier. If the unique identifier is not available, the control hierarchy path and control text will be matched in a downgraded manner.
[0086] The four sub-steps S3001 to S3004 are executed sequentially. If any sub-step fails verification, subsequent verification is terminated and a rejection record is generated. The rejection record is written to the set of rejected actions corresponding to the current state snapshot. The rejection record includes three fields: the rejected action field records the complete content of the rejected single-step test action, including action type, target control, and action parameters; the rejection reason field records the corresponding verification sub-step number and a specific description of the rejection reason; and the correction suggestion field records the direction of action correction for the rejection reason. This correction suggestion is directly directed to the large language model, informing the large language model how to adjust the next generation in the form of an operable instruction.
[0087] After a rejection record is generated, the testing system uses the set of rejected actions as incremental input information and feeds it back into the next round of input for the large language model. Specifically, in the next call to the large language model, the set of rejected actions is appended with high priority before or immediately after the current action constraint information. This allows the large language model to see which actions have been tried in this round, why they were rejected, and how to correct them when regenerating actions. This feedback mechanism creates a trial-and-error correction closed loop in the continuous generation process of the large language model. That is, the generated actions are checked layer by layer by the verification module. If an action fails, the reason for rejection and correction suggestions are recorded and fed back to the next round of input until all actions that pass the verification are generated.
[0088] Furthermore, the rejection record includes the rejected action, the reason for rejection, and a correction suggestion; the large language model regenerates the single-step test action based on the rejection reason and correction suggestion in the rejection record.
[0089] Furthermore, the threshold is dynamically set based on the density of historical invalid actions; the preset priorities are, in order, handling blocking pop-ups, filling in required input fields, correcting error message related fields, submitting currently valid forms, accessing uncovered navigation entries, expanding unaccessed folded areas, scrolling to low-coverage areas, entering the next page of a pagination or list, and selecting interactive controls different from recent invalid actions. The generation of the alternative test action includes: determining candidate actions according to the preset priority, and using scoring rules to score the candidate actions based on their comprehensive action advancement value, coverage value, safety and executability, and repetition penalty, and selecting the candidate action with the highest score as the alternative test action.
[0090] Specifically, in this embodiment, when the number of consecutive rejections for the same state snapshot reaches a threshold, the testing system no longer waits for the large language model to be regenerated, but directly initiates the alternative test action generation process. This threshold is not a fixed constant, but is dynamically set according to the density of historical invalid actions. Let the density of historical invalid actions be D, which is defined as the ratio of the number of actions that did not trigger page state transitions in the historical test records to the total number of actions, i.e., D = Ftotal / Ntotal, where Ntotal is the total number of actions in the historical test records, and Ftotal is the number of actions that did not trigger page state transitions. The testing system adjusts the rejection threshold according to the recent density of historical invalid actions. Let the base threshold be θ0, and the rejection threshold under the current state snapshot be θ, then the threshold calculation formula is θ = round(θ0 × (1 + λ × D)), where λ is an adjustment coefficient that controls the amplification of the threshold by the historical invalid density, and takes a positive value; round is the rounding function. When Ntotal = 0, that is, there are no historical test records, D is 0, and the threshold θ is the base threshold θ0.
[0091] When the density of recent invalid actions D is high, it indicates that invalid operations are concentrated in the current tested interface. The system appropriately relaxes the threshold to increase tolerance, allowing the large language model more opportunities to try. When the density of recent invalid actions D is low, the threshold is close to the basic threshold θ0, which shrinks the fault tolerance space of the large language model and triggers the generation of alternative actions more quickly to ensure testing efficiency. The basic threshold θ0 and the adjustment coefficient λ can be set according to the needs of the test scenario. For example, θ0 can be an integer between 3 and 5, and λ can be a constant between 0.5 and 2.
[0092] When the number of consecutive rejections reaches a threshold θ, the testing system generates alternative test actions based on preset priorities. The preset priorities are arranged into nine levels, from highest to lowest, according to the urgency of the action and its coverage advancement value: The first priority is to handle blocking pop-ups. That is, if there is a blocking pop-up on the current interface, the alternative test action will prioritize the operation of the confirmation, cancellation or close controls in the pop-up to eliminate the obstruction of the underlying interface by the pop-up. The second priority is to fill in required input fields. That is, if there is an input control with an empty value on the current interface, the test action will target that control to perform the filling operation. The third priority is to correct the fields associated with the error message. That is, if there is an error message on the current interface, the alternative test action will target the input control associated with the error message and correct the input value according to the semantics of the error message. The fourth priority is to submit the currently valid form. That is, if the required fields of the current form have been filled and there is no validation error message, the submit control is clicked to trigger the form submission process instead of the test action. The fifth priority is to access uncovered navigation entries, which means selecting navigation controls associated with boundaries of type "unaccessed page entry" from the set of exploration boundaries as targets.
[0093] The sixth priority is to expand unexplored folded areas, which means selecting the folded controls associated with the boundary of type unexplored folded area from the set of explored boundaries as the target to perform the expansion operation.
[0094] The seventh priority is to roll to the low-coverage area, which means selecting a rolling container of type "rolling area boundary" from the set of exploration boundaries as the target to perform the rolling operation.
[0095] The eighth priority is to enter the next page of a pagination or list, which means selecting the next page button of type pagination entry boundary from the set of exploration boundaries as the target to perform the click operation.
[0096] The ninth priority is to select interactive controls that are different from recent invalid actions. That is, select controls from the current set of key controls whose overlay state is not operated or has triggered a page state transition after operation as targets, and execute the default action among their candidate actions.
[0097] If a candidate action for a certain priority does not exist, it will be deferred to the next priority.
[0098] Alternative test actions are generated using a scoring rule to determine the optimal action from candidate actions. For a given set of candidate actions with a given priority, a comprehensive score is calculated for each candidate action. Let candidate action 'a' have a comprehensive score 'Sa', and the scoring formula be Sa = wp × Vp + wc × Vc + ws × Vs. wr × Rp, if Sa is negative, set it to 0. Vp is the action advancement value score, Vp = 1.0 0.1 × (L 1) Where L is the priority number of the candidate action (first priority L=1, second priority L=2, and so on up to ninth priority L=9), and Vp ranges from 0.1 to 1.0. Vc is the coverage value score, which is directly taken as the coverage value Vj associated with the exploration boundary of the candidate action. If the candidate action is not associated with any exploration boundary (such as blocking pop-up closing, required input, etc., which have no coverage extension meaning), then Vc is 0. Vs is the safety executability score, Vs = s / S, where S is the total number of safety executability evaluation items, and s is the number of evaluation items currently satisfied. Evaluation items include: target control visible, target control enabled, all preconditions of the target control satisfied, no blocking pop-up on the current interface, or the target control located within the current blocking pop-up. For example, when all four evaluation items are satisfied, Vs=1.0, and when only the first two are satisfied, Vs=0.5. Rp is the repetition penalty score, Rp = log2(Nr + 1), where Nr is the total number of times the target control of the candidate action has been operated on in recent rounds of testing. A logarithmic function is used to reduce the marginal increment of the repetition penalty: for the first operation, Nr=0, Rp=0, no penalty; after 3 operations, Rp=2.0, resulting in a moderate penalty; after 15 operations, Rp=4.0, resulting in a heavier penalty. This design allows the system to tolerate a small number of repetitive operations while suppressing frequent repetitions. wp, wc, and ws are the weighting coefficients for advancement value, coverage value, and safety executability, satisfying wp+wc+ws=1. A feasible set of values is wp=0.4, wc=0.3, and ws=0.3. wr is the independent coefficient of the repetition penalty, ranging from 0 to 1. A feasible set of values is wr=0.3. By separating wr from the other terms in the scoring formula, the strength of the repetition penalty term can be adjusted independently without affecting the relative proportions among the other three terms.
[0099] The testing system iterates through nine priority levels in descending order of priority. Within each priority level, it calculates and sorts the comprehensive score Sa for each candidate action, selecting the candidate action with the highest Sa as the representative action for that priority level. If the comprehensive score Sa of the representative action for a certain priority level is greater than the preset score threshold Smin, then that representative action is selected as the alternative test action, and the iteration terminates. If, after iterating through all nine priority levels, no candidate action has a comprehensive score exceeding Smin, then the action with the highest Sa among all candidate actions is selected as the fallback alternative test action. After the alternative test action is generated, the subsequent large language model regeneration and verification process is skipped, and the system directly proceeds to the action execution step.
[0100] S105, execute the single-step test action or the alternative test action, collect a state snapshot after execution, compare the state snapshots before and after execution, determine whether a page state transition is triggered, and update the historical test record according to the comparison result.
[0101] In its implementation, S105 converts structured test actions into native automated operations of the platform under test through the action execution bridging module. Taking a web application as an example, a click action locates the target element in the document object model tree using the control's unique identifier and calls the browser's automation interface to execute the click; an input action first locates the input box to gain focus, then inputs the action parameter text character by character before triggering a change event; a scrolling action reads direction and distance parameters and calls the scroll interface; a waiting action continuously monitors the interface until the waiting condition is met or a timeout occurs. Desktop and mobile platforms achieve similar adaptations through corresponding accessibility interfaces or automation driver interfaces. During execution, time consumption, status codes, and exception information are recorded as feedback logs.
[0102] After the action is executed, the interface may not be stable due to asynchronous rendering or page transition animations. Therefore, the test system performs a waiting period for interface stability after execution but before taking a snapshot: the interface state is continuously sampled at sampling intervals Δt, and the number of controls, the visibility of controls, and the state of pop-ups are compared between two consecutive samples; the interface is considered stable when the results of Q consecutive samples are consistent. The maximum waiting time is Δt × Q; if a timeout occurs, the last snapshot is used and the timeout information is recorded. Δt and Q are positive parameters, with Δt typically ranging from 0.5 to 2 seconds and Q ranging from 2 to 5. After the interface stabilizes, a snapshot of the state Safter after execution is taken using the same data structure as S101 to ensure semantic comparability between the snapshots before and after.
[0103] Furthermore, the comparison of state snapshots before and after execution includes comparing page identifiers, window or pop-up states, control sets, control properties, focus states, input values, error messages, success messages, list data, navigation paths, and scroll positions.
[0104] In practical implementation, the testing system performs an eleven-dimensional difference comparison between Sbefore and Safter. The comparison includes: Page identifier comparison: Differences in page identifiers before and after indicate a page jump has been triggered. Pop-up state comparison: Pop-ups are open, closed, or in a modal state. Control set comparison: The set of control identifiers before execution is Sidbefore, and after execution is Sidafter. Newly added controls are the difference between Sidafter and Sidbefore, and disappeared controls are the difference between Sidbefore and Sidafter. A non-empty difference indicates a change in the interface structure. Control attribute comparison: Coexisting controls are compared one by one in terms of type, text, visibility, enabled state, coordinates, and size. Focus state comparison: The system determines whether the interactive focus has shifted. Input value comparison: The system verifies whether the input operation modifies the control value to the value specified by the action parameter. Error message comparison: Addition, disappearance, or content changes indicate that business validation logic has been triggered. Success message comparison: The system identifies control text containing keywords such as "submission successful," "save successful," and "operation completed," and determines whether a success message appears based on changes in visibility. List data comparison: Compares changes in list row count, row text, and row identifiers. Navigation path comparison: Compares the complete path sequence from the homepage to the current page, including changes to both the endpoint and intermediate nodes. Scroll position comparison: Verifies whether scrolling shifts the position a specified distance in a specified direction and whether it brings invisible areas into the visible range.
[0105] When the differential result indicates a page jump, the addition or disappearance of controls, a change in error message, successful data submission, the opening or closing of a pop-up window, a change in list data, or an expansion of the coverage area, a page state transition is determined to be triggered. When the differential result is empty or there are only non-functional changes such as timestamp updates, a page state transition is determined not to be triggered, and a correction suggestion is generated and written to the aforementioned historical test record.
[0106] In practice, the page state transition determination adopts a seven-channel short-circuit mechanism, terminating the determination if any channel is satisfied in sequence. The first channel is page jump: if the page identifier is different, a transition is directly determined, which has the highest weight. The second channel is pop-up status: whether the pop-up is open or closed.
[0107] The third channel is for changes in control sets: After pre-filtering the newly added and disappeared control sets, if the number of remaining controls is greater than zero, the channel is considered valid. The pre-filtering rules are as follows: Rule 1 (Pre-filtering of time display controls): Controls whose text matches a preset time format regular expression are filtered out. These time format regular expressions include, but are not limited to, common date and time display formats such as HH:MM:SS, HH:MM, YYYY-MM-DD, HH:MM:SS, and YYYY / MM / DD. The content of such controls changes automatically with the system clock; their addition or disappearance does not represent a change in the interface's functional state. Rule 2 (Pre-filtering of dynamic background controls): Controls of image or media type, and whose parent node in the control hierarchy path has a carousel or animation attribute (such as the carousel or animation attribute value of a web platform), are filtered out. Such controls switch automatically with the carousel or animation; their changes do not represent a functional change caused by user operation. Rule 3 (Pre-filtering of purely decorative controls). If a control is a container, separator, or text containing only whitespace characters, and the control does not contain any child controls or child controls with interactive properties enabled in the control hierarchy path, it will be filtered out.
[0108] The fourth channel is for error message changes: error messages are added, disappear, or their content changes. The fifth channel is for successful data submission: a success message that did not exist before execution appears after execution. The sixth channel is for list data changes: row counts or content are updated. The seventh channel is for expanded coverage area: scrolling brings invisible controls into the visible state. If all seven channels are not satisfied and the difference is only a non-functional change such as a timestamp, a brief change in focus caused by cursor blinking, or a change in loading indicator status, it is determined that a page state transition has not been triggered.
[0109] When a transition is determined not to have occurred, the following correction suggestions are generated: If input is entered and the difference between the input values is zero, the correction suggestion is that the input operation has not taken effect, and it is recommended to check the read-only state or preconditions; if input has taken effect but other dimensions have not changed, the correction suggestion is that the input has been completed but not submitted, and it is recommended to proceed to the next step to submit; if the target control's state does not change after clicking, the correction suggestion is that the click has no response, and it is recommended to check the event binding or try other candidate actions; if scrolling is performed and the difference between the scroll positions is zero, the correction suggestion is that scrolling has not taken effect, and it is recommended to check the scrollable attributes or the upper limit of the range; if waiting is performed and all dimensions have not changed, the correction suggestion is that the wait has timed out, and it is recommended to adjust the waiting conditions or shorten the waiting time; if the cause cannot be clearly inferred, the correction suggestion is that the current action has not triggered the page state transition, and it is recommended to change the target control or action type.
[0110] The update process for historical test records is as follows: The current round record is appended to the recent multi-round test records, including eight fields: action description, target control, action parameters, pre- and post-execution state summary, feedback log, execution result, and correction suggestions. The feedback log records the time consumed, status code, and exception returned by the bridging module; the execution result records whether the execution was successful and the Pflag. Subsequently, the long-term test summary is updated according to the incremental update rules described in S101: the number of pages covered in the current state segment increases upon first entering a new page; the recent important event segment is updated during breakthrough operations; the persistent failure record segment is updated when the failure count threshold F is reached; and the next test focus segment adjusts its direction when the coverage threshold β or the continuous invalid threshold M is triggered.
[0111] The execution flow of S105 is as follows: First, the structured actions are passed to the bridging module for execution, and feedback logs are recorded. Second, the interface status is continuously collected at intervals of Δt. Stability is determined after Q consecutive consistent results; otherwise, the last snapshot is used. Third, Safter is collected after stabilization. Fourth, an eleven-dimensional difference comparison is performed on Sbefore and Safter to generate a difference result set D. Fifth, Pflag is determined based on a seven-channel short circuit. Sixth, if Pflag is not transferred, a correction suggestion is inferred based on D and the action type. Seventh, the current round record is constructed and appended to recent round records; when exceeding K rounds, the oldest record is removed. Eighth, each paragraph of the long-term summary is incrementally updated, completing the current test loop.
[0112] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. An automated exploratory testing method for graphical user interfaces based on a large language model, characterized in that, include: Obtain a snapshot of the graphical user interface state and historical test records of the application under test; extract a set of key controls based on the state snapshot; and identify a set of exploration boundaries based on the state snapshot and the historical test records. Generate current action constraint information based on the state snapshot and the historical test records; Input the state snapshot, the set of key controls, the set of exploration boundaries, the current action constraint information, and the historical test records into the large language model to generate a single-step test action; The single-step test action is verified; If the verification fails, a rejection record is generated and fed back to the input of the large language model to regenerate the single-step test action; if the number of consecutive rejections for the current state snapshot reaches a threshold, an alternative test action is generated according to a preset priority. Execute the single-step test action or the alternative test action, collect a state snapshot after execution, compare the state snapshots before and after execution, determine whether a page state transition is triggered, and update the historical test record based on the comparison result.
2. The method according to claim 1, characterized in that, The state snapshot includes the control's unique identifier, control text, control type, control hierarchy path, control coordinates, control size, visibility status, enabled status, focus status, pop-up status, input value, error message, and page hierarchy information.
3. The method according to claim 1, characterized in that, The extracted key control set includes: The controls are obtained from the state snapshot. Invisible and obscured controls are removed, disabled and non-interactive controls are filtered out, and non-interactive controls used only for page display are filtered out to obtain a candidate control set. The buttons, input boxes, checkboxes, radio buttons, drop-down lists, menu items, tabs, links, list items, pagination controls, scroll containers, pop-up buttons, and error message areas in the candidate control set are mapped to a unified control description to obtain the key control set; The set of exploration boundaries to be identified includes: identifying exploration boundaries based on access history, page structure changes, new control status, scroll position, and page navigation relationships.
4. The method according to claim 3, characterized in that, The extraction of the key control set also includes: labeling each key control with its function, candidate actions, prerequisites, and coverage status.
5. The method according to claim 1, characterized in that, The historical test records include recent test records and long-term test summaries; The recent multi-round test records include action descriptions, target controls, action parameters, pre-execution state summaries, post-execution state summaries, feedback logs, execution results, and correction suggestions for the most recent rounds of tests. The long-term test summary is stored in segments based on the current state, recent important events, early milestones, persistent failure records, and the focus of the next test, and is incrementally updated after each new recent multi-round test record is added.
6. The method according to claim 1, characterized in that, The step of generating current action constraint information based on the state snapshot and the historical test records includes: When the status snapshot indicates the presence of a blocking pop-up, the current action constraint information includes the single-step test action determining the target within the controls of the blocking pop-up; When the state snapshot indicates the existence of a control with an empty input value, the current action constraint information includes that the single-step test action targets the control and the action type is fill-in; When the state snapshot indicates that there is a control that is in focus and has an input value, the current action constraint information includes that the single-step test action targets the control and the action type is submit, save, continue input or cancel; When the historical test record indicates that the control in the state snapshot did not trigger a page state transition, the current action constraint information includes that the single-step test action does not target that control; When the historical test record indicates that the state snapshot before and after execution has no change in multiple consecutive rounds of comparison, the current action constraint information includes a loop detection warning. The loop detection warning includes the single-step test action targeting controls corresponding to different page identifiers or different control types. When multiple conditions are met at the same time and the corresponding constraints conflict, the constraint priority is determined according to the order of the conditions listed above.
7. The method according to claim 2, characterized in that, The verification of the single-step test action includes: Verify whether the single-step test action is empty and whether the action type is supported; Verify whether the control identifier corresponding to the single-step test action exists in the state snapshot, and whether the visible and enabled states of the control identifier in the state snapshot indicate that the control can be operated; Compare whether the action parameters of the single-step test action match the control type corresponding to the control identifier in the state snapshot, and whether the action parameters meet the format requirements; Verify whether the single-step test action conflicts with the current action constraint information, and whether the action has been recorded in the rejection record or whether there is an action record in the historical test record that targets the same control and does not trigger a page state transition.
8. The method according to claim 7, characterized in that, The rejection record includes the rejected action, the reason for rejection, and suggested corrections; The large language model regenerates the single-step test action based on the rejection reasons and correction suggestions in the rejection record.
9. The method according to claim 1, characterized in that, The threshold is dynamically set based on the density of historical invalid actions; The preset priorities are as follows: handling blocking pop-ups, filling in required input fields, correcting error message related fields, submitting the currently valid form, accessing uncovered navigation entries, expanding unaccessed collapsed areas, scrolling to low-coverage areas, entering the next page of a pagination or list, and selecting interactive controls that are different from recently invalid actions. The generation of the alternative test action includes: determining candidate actions according to the preset priority, and using scoring rules to score the candidate actions based on their comprehensive action advancement value, coverage value, safety and executability, and repetition penalty, and selecting the candidate action with the highest score as the alternative test action.
10. The method according to claim 2, characterized in that, The comparison of state snapshots before and after execution includes comparing page identifiers, window or pop-up states, control collections, control properties, focus states, input values, error messages, success messages, list data, navigation paths, and scroll positions. When the differential result indicates a page jump, the addition or disappearance of a control, a change in the error message, a successful data submission, the opening or closing of a pop-up window, a change in list data, or an expansion of the coverage area, a page state transition is triggered. When the difference result is empty or there are only non-functional changes such as timestamp updates, it is determined that the page state transition has not been triggered, and a correction suggestion is generated and written into the historical test record.
