An Automated Web UI Testing Method and System Based on Visual Language Model

Abstract

The application discloses an automatic Web UI test method and system based on a visual language model, which comprises the following steps: parsing a test task set containing operation actions and target objects from a test script; based on a Web interface screenshot and an accessibility tree, respectively extracting a visual interface element set and a structured interface element set through a visual language model and rule analysis; using a multi-modal semantic alignment algorithm to fuse the visual and semantic features of the two types of elements to generate a complete interface element set; inputting the test task and the complete element set as context prompts into the visual language model to drive it to generate interface observation, semantic reasoning and operation through iteration to execute the test. The application fuses multi-source information of vision, structure and semantics to realize end-to-end automatic mapping of natural language test tasks to accurate UI operations, and significantly improves the accuracy, robustness and generalization ability of the test in dynamic interfaces and cross-platform scenarios.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of software testing and artificial intelligence, specifically relating to an automated Web UI testing method and system based on a visual language model. Background Technology

[0002] In recent years, the development of Visual Language Models (VLMs) has provided new ideas for UI testing. Some methods attempt to directly parse interface screenshots using VLMs and generate operation sequences by combining them with natural language instructions. However, pure visual methods often struggle to accurately identify the semantic roles and states of interactive controls in scenarios with blurred text, occluded elements, or low contrast. Other methods, relying solely on structured information such as the DOM or accessibility trees, cannot address the semantic gaps caused by dynamic rendering and front-end framework encapsulation. Furthermore, existing large-model-based test agents typically employ open action generation mechanisms, lacking constraints on the operation space and prone to generating unexecutable or unsafe instructions. Simultaneously, in multi-step tasks, the lack of task-level isolation of historical action memories can easily lead to cross-task interference, reducing the stability and repeatability of the testing process. Therefore, there is an urgent need for an automated Web UI testing method that can integrate visual perception and structured semantic information, possess precise element alignment capabilities, and implement task-oriented iterative reasoning within a closed action space, balancing accuracy, robustness, and engineering feasibility. Summary of the Invention

[0003] This invention aims to address the shortcomings of existing Web UI automated testing methods in understanding the semantic alignment between natural language test scripts and interface elements, and provides an automated Web UI testing method based on a visual language model. This method parses a set of test tasks containing operational actions and target objects from the test script, extracts visual interface elements and structured interface elements by combining a visual screenshot of the current interface and an accessibility tree, and then uses a multimodal semantic alignment algorithm to fuse the two types of elements to generate a complete set of interface elements. Based on this, a dynamic prompt sequence containing an observation space, target tasks, and historical action memories is constructed to drive the visual language model to iteratively complete the Web UI automated testing process through automatic observation, logical reasoning, and operation generation. This method achieves end-to-end automated mapping from natural language test tasks to precise UI operations, significantly improving the accuracy, robustness, and generalization adaptability of testing in dynamic interfaces, cross-platform environments, and semantically ambiguous scenarios.

[0004] To achieve the aforementioned objectives, the technical solution adopted by this invention is as follows: An automated Web UI testing method based on a visual language model is provided, comprising the following steps:

[0005] S1. Obtain the test script And parse out the test task set from it. Each test task Includes an operation action and the target object to which the action is performed. ; S2. Based on screenshots of the current web application interface, extract a set of visual interface elements using a visual language model. Simultaneously, based on the accessibility tree corresponding to the web application, a set of structured interface elements is extracted using preset parsing rules. The visual interface elements include location information. Index information and semantic description information The structured interface elements include name information. Hierarchical path information and interaction status information ; S3. Employ a multimodal semantic alignment algorithm to fuse the set of visual interface elements. With the structured interface element set The corresponding elements in: This algorithm performs cross-modal matching based on visual and semantic features, and generates a fused complete set of interface elements based on the matching results; S4. The test task set With the complete set of interface elements As contextual prompts, the information is input into the visual language model; through the model's automatic observation of the current interface state, logical reasoning of task semantics, and generation and execution of operation sequences, the Web UI automated testing process is iteratively completed.

[0006] Furthermore, the specific method of step S1 includes the following sub-steps: S1-1. Use a sentence segmentation algorithm to divide the test script text. Split into a set of sentences For each sentence of text Based on a set of keywords for manually preset operation actions and target keyword set Extracting via keyword matching method Operations in and target object The extraction results include text content, text index, and sentence number information. The obtained operation actions and target objects are combined into operation action sets. With the target object collection ; S1-2, Based on the set of operation actions extracted from each sentence With the target object collection Based on the principle of proximity matching, the identified operation actions are paired with their nearest target objects to construct a set of test tasks. Each of the tasks Includes an operation action and a target object ,Right now .

[0007] Furthermore, the specific method of step S2 includes the following sub-steps: S2-1. Based on screenshots of the current web application interface, extract a set of visual interface elements using a visual language model. Each visual interface element Including location information Index information and semantic description information The location information Including the coordinates of the top left corner of the element and the coordinates of the bottom right corner The origin of the coordinate system is located at the top left corner of the web application interface: The index information is a unique identifier assigned to each visual interface element according to the traversal order from left to right and from top to bottom, used to distinguish different visual interface elements. The semantic description information is the text content generated by the visual language model that represents the visual interface element. S2-2. Based on the accessibility tree corresponding to the current web application interface, extract the set of structured interface elements using preset parsing rules. Each structured interface element Including name information Hierarchical path information and interaction status information Name information The accessible name declared by the element in the accessibility tree, and hierarchical path information. This provides the hierarchical path information from the root node to the element node, used to characterize its position in the interface structure and interaction state information. To determine whether the current interactive attribute is enabled based on the semantic role category of the element:

[0008] in, Representative elements Corresponding semantic role information

[0009] Furthermore, the specific method of step S3 includes the following sub-steps: S3-1, For each visual interface element Location information Extract the corresponding visual region feature vector. Then, a global average pooling operation is performed on it to obtain a normalized visual feature vector. : Where GAP(·) represents the global average pooling function, and ||·|| represents the L2 norm; S3-2, For each structured interface element , its name information With hierarchical path information Each input is fed into a pre-trained language model to generate semantic feature vectors of the same dimension. and The two are then concatenated along the feature dimension to form a joint semantic feature vector. :

[0010] Through linear projection layer splicing vectors Mapping to a space with the same dimension as the visual features yields semantic feature vectors.

[0011]

[0012]

[0013] in To obtain learnable parameters, the CrossModalAttn cross-modal attention module is used, combined with visual region feature vectors. semantic feature vectors Enhancement is performed to obtain the enhanced semantic feature vector. :

[0014]

[0015] in, For semantic feature vectors, For visual feature vectors, , , Let d be a learnable linear projection matrix, where d is the feature dimension, and "·" denotes matrix multiplication. The Softmax function operates along the row directions of the attention weight matrix, and then the semantic feature vector is augmented. L2 normalization is performed to obtain normalized enhanced semantic feature vectors. :

[0016] S3-3, Calculate the normalized enhanced semantic feature vector. With normalized visual feature vectors Cosine similarity between As a cross-modal matching score:

[0017] S3-4, Based on the aforementioned similarity score When the similarity score is greater than the set threshold The system considers element matching. When multiple visual interface elements match the same structured interface element, only the visual interface element with the highest similarity is retained to establish a one-to-one mapping relationship with that structured interface element. The remaining visual interface elements are marked as unmatched items. For successfully matched element pairs, their visual features, location information, index information, semantic description information, hierarchical path information, and interaction state information are fused to generate an interface element containing complete attributes. .

[0018] Furthermore, the specific method of step S4 includes the following sub-steps: S4-1. Construct a dynamic cue sequence P, which contains the following four parts: (a) Current observation space: consisting of the current web interface screenshot and its corresponding structured text explanation, which includes the name, hierarchical path and interaction state information of each interactive element, used to assist the model in understanding visual content; (b) Target task description: the set of test tasks extracted in step S1. (c) Historical Action Memory Space: Records the sequence of actions executed within the current test task. ; S4-2. Input the prompt sequence P into the visual language model VLM to drive it to generate the current time step. Intermediate inference results With operating instructions ,satisfy:

[0019] in This represents the current observation space, including screenshots of the web application interface and the complete set of interface elements. , Represents a set of test tasks. Represents the memory space of historical actions, and the intermediate reasoning results. It contains two semantic parts: (a) an environment-aware description: an objective statement of the current interface layout, the state of relevant elements, and the unfinished tasks; and (b) a logical deduction process: based on the test task set Tasks and the current observations. This involves analyzing which operation should be performed, evaluating the relevance of each candidate element, and logically deriving why the operation was selected. The operation instructions are then defined. The action is selected from a predefined action space, which includes: Click (element: int), Text (element: int, text_input: str), Hover (element: int), and Stop; where the element parameter is a unique identifier of the target element matched in the complete set of interface elements. S4-3, Generate the operation instructions The data is sent to the web browser's automated execution engine to simulate real user interaction behavior. After execution, the updated interface screenshot and accessibility tree are captured. Steps S2 to S3 are re-executed to generate a new set of complete interface elements, which is then used as the observation input for the next time step t+1. ; S4-4. Repeat S4-1 to S4-3 until the model outputs the Stop operation, or the test process ends when the preset maximum number of iterations is reached.

[0020] This application also provides an automated Web UI testing system based on a visual language model, the system being configured as follows: Get the test script And parse out the test task set from it. Each test task Includes an operation action and the target object to which the action is performed. ; Based on screenshots of the current web application interface, a set of visual interface elements is extracted using a visual language model. Based on the accessibility tree corresponding to the web application interface, a set of structured interface elements is extracted using preset parsing rules. The visual interface elements contain location information. Index information and semantic description information The structured interface elements include name information. Hierarchical path information and interaction status information ; Employing a multimodal semantic alignment algorithm, and fusing a set of visual interface elements. With structured interface element collection The algorithm performs cross-modal matching based on visual and semantic features, and generates a fused complete set of interface elements based on the matching results. ; The test task set With the complete set of interface elements As contextual prompts, the information is input into the visual language model; through the model's automatic observation of the current interface state, logical reasoning of task semantics, and generation and execution of operation sequences, the Web UI automated testing process is iteratively completed.

[0021] This application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described above.

[0022] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0023] The beneficial effects of this invention are as follows: 1. By parsing test task sets containing only operation actions and target objects from natural language test scripts, the reliance on vague or redundant information such as "expected results" is eliminated, significantly simplifying the complexity of task modeling and improving the efficiency and accuracy of mapping test instructions to executable operations.

[0024] 2. By simultaneously utilizing the visual language model and the barrier-free tree to extract visual interface elements and structured interface elements respectively, and integrating the position, semantics, hierarchy and interaction state information of the two, the limitations of single modality in element recognition are effectively overcome. In particular, it can maintain a high robust element localization capability in dynamic rendering, occlusion or low contrast interfaces.

[0025] 3. A cross-modal semantic alignment algorithm based on visual and semantic features is adopted to achieve one-to-one accurate matching between visual interface elements and structured interface elements. In the event of conflict, a similarity threshold and a unique mapping mechanism are used to ensure the determinism of the fusion result, thereby constructing a set of interface elements containing complete attributes and providing a reliable context for subsequent operations.

[0026] 4. By constructing a dynamic prompt sequence that includes the current observation space, target task description, and task-level isolated historical action memory, the visual language model is driven to perform an iterative closed loop of observation-reasoning-execution within a closed action space. This not only ensures the security and parsability of the operation but also avoids cross-task interference, significantly improving the success rate and generalization adaptability of Web UI automated testing in complex business processes. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall process of this method; Figure 2 Here is a screenshot example of a web application implementing this method. Detailed Implementation

[0028] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0029] like Figure 1 As shown in the technical approach, in one embodiment of the present invention, using Figure 2 The system screenshot and test script "Enter 'Zhang San' in the name text box. Click the search button." serve as an implementation example. The overall process includes: S1. Obtaining the test script and parsing out the test task set, where each test task includes an operation and the target object of that operation; S2. Based on the current web application interface screenshot, extracting the visual interface element set using a visual language model, and extracting the structured interface element set based on the accessibility tree corresponding to the web application using preset parsing rules; S3. Using a multimodal semantic alignment algorithm, fusing the corresponding elements in the visual interface element set and the structured interface element set, and generating a fused complete interface element set based on the matching results; S4. Inputting the test task set and the complete interface element set as context prompts into the visual language model. Through the model's automatic observation of the current interface state, logical reasoning of task semantics, and generation and execution of operation sequences, the web UI automated testing process is iteratively completed. Detailed steps include: S11. The test script "Enter 'Zhang San' in the name text box and click the search button" is processed into sentences to obtain a set of sentences. :

[0030] For each sentence in the sentence set, use the action keyword set. The extracted set of operations is:

[0031]

[0032] Use the target object keyword set The target object set extracted is:

[0033]

[0034] S12. Based on the extracted set of operation actions and the set of target objects, the first test task is generated according to the principle of proximity matching:

[0035] Similarly, processing The second test task was obtained:

[0036] Finally, the test task set .

[0037] S21, based on Figure 2 The screenshot of the web interface shown demonstrates the use of a visual language model for element detection, resulting in a set of visual interface elements. :

[0038]

[0039]

[0040] S22, based on Figure 2 The accessibility tree corresponding to the web interface screenshot shown is used to extract a set of structured interface elements using preset parsing rules. :

[0041]

[0042]

[0043] This yielded the set of visual interface elements and the set of structured interface elements corresponding to the web interface screenshot.

[0044] S31. Based on the set of visual interface elements and the set of structured interface elements extracted from the web interface screenshots in step S2, construct a mapping between the visual interface elements and the structured interface elements. Using the visual interface elements... and structured interface elements For example, first extract The screenshot area corresponding to the location information is cropped and input into the visual language model to obtain a visual feature vector:

[0045] Subsequently, Perform global average pooling and then L2 normalization:

[0046] S32. For structured interface elements The name information "Enter Name" and the hierarchical path information " / html / body / div[1] / input[1]" are respectively input into the pre-trained language model to obtain two semantic feature vectors with the same dimensions and consistent with the dimensions of the visual feature vector:

[0047]

[0048] The two are concatenated along the feature dimension to form a joint semantic feature vector:

[0049] Through linear projection layer splicing vectors Mapping to a space with the same dimension as the visual features yields the semantic feature vector:

[0050] Using the CrossModalAttn cross-modal attention module for semantic features Enhancement:

[0051] Finally, L2 normalization is performed on the enhanced semantic feature vector:

[0052] S33. Calculate the normalized visual feature vector. With normalization to enhance semantic feature vectors Cosine similarity between them: 0.21×0.25+(-0.14)×0.09+0.80×(-0.66)+...+0.59×0.52=0.92 The similarity score is significantly higher than the preset threshold. =0.72, indicating that the visual interface elements With structured interface elements High-level matching; the attributes of both are merged based on the matching results to generate complete interface elements. :

[0053]

[0054] S41, based on Figure 2 The web application screenshots shown and the complete set of interface elements generated by S3 constitute the observation space. The test task set Tasks extracted by S1 is used as the target task description, and the empty historical action memory space is combined to form a structured prompt sequence. ; S42, Prompt sequence The input is fed into the Visual Language Model (VLM), and the model outputs intermediate inference results. With operating instructions ,in: It includes an environment-aware description that "the page contains an editable input box named 'Name'" and a logical deduction process that "you need to enter 'Zhang San' in this box first to complete the first step of the task"; Select the Text operation from the predefined action space: Text(element=2, text_input="Zhang San"), where element=2 corresponds to the complete interface element generated by fusion in S3. .

[0055] S43, Operation Instructions The code is sent to the web browser's automation engine to simulate a user typing "Zhang San" into the "Name" input box. After execution, an updated screenshot of the interface and the accessibility tree are captured, and S2–S3 are re-executed to generate a new set of complete interface elements. .

[0056] S44, based on Construct a new prompt sequence P′, where the historical action memory space is updated to [Text(2,“Zhang San”)]. The model infers again, recognizes the “Search” button (element=5), and outputs the operation command. After clicking, the page refreshes, the task is completed, the model outputs the "Stop" command, and the test process terminates.

Claims

1. An automated Web UI testing method based on a visual language model, characterized in that, Includes the following steps: S1. Obtain the test script And parse out the test task set from it. Each test task Includes an operation action and the target object to which the action is performed. ; S2. Based on screenshots of the current web application interface, extract a set of visual interface elements using a visual language model. Based on the accessibility tree corresponding to the web application interface, a set of structured interface elements is extracted using preset parsing rules. The visual interface elements contain location information. Index information and semantic description information The structured interface elements include name information. Hierarchical path information and interaction status information ; S3. Employs a multimodal semantic alignment algorithm, integrating a set of visual interface elements. With structured interface element collection The algorithm performs cross-modal matching based on visual and semantic features, and generates a fused complete set of interface elements based on the matching results. ; S4. The test task set With the complete set of interface elements As contextual prompts, the information is input into the visual language model; through the model's automatic observation of the current interface state, logical reasoning of task semantics, and generation and execution of operation sequences, the Web UI automated testing process is iteratively completed.

2. The automated Web UI testing method based on a visual language model according to claim 1, characterized in that, Step S1 includes the following steps: S1-1. Use a sentence segmentation algorithm to divide the test script text. Split into a set of sentences For each sentence of text Based on a set of keywords for manually preset operation actions and target keyword set Extracting via keyword matching method Operations in and target object The extraction results include text content, text index, and sentence number information. The obtained operation actions and target objects are combined into operation action sets. With the target object collection ; S1-2, Set of Operation Actions Extracted from Each Sentence With the target object collection Based on the principle of proximity matching, the identified operation actions are paired with their nearest target objects to construct a set of test tasks. Each of the tasks Includes an operation action and a target object ,Right now .

3. The automated Web UI testing method based on a visual language model according to claim 1, characterized in that, Step S2 includes the following steps: S2-1. Based on screenshots of the current web application interface, extract a set of visual interface elements using a visual language model. Each visual interface element Including location information Index information and semantic description information The location information Including the coordinates of the top left corner of the element and the coordinates of the bottom right corner The origin of the coordinate system is located at the top left corner of the web application interface: The index information is a unique identifier assigned to each visual interface element according to the traversal order from left to right and from top to bottom, used to distinguish different visual interface elements. The semantic description information is the text content of the visual interface element extracted by the visual language model. S2-2. Based on the accessibility tree corresponding to the current web application interface, extract the set of structured interface elements using preset parsing rules. Each structured interface element Including name information Hierarchical path information and interaction status information Name information The accessible name declared by the element in the accessibility tree, and hierarchical path information. This provides the hierarchical path information from the root node to the element node, used to characterize its position in the interface structure and interaction state information. To determine whether the current interactive attribute is enabled based on the semantic role category of the element: in, Representative elements The corresponding semantic role information.

4. The automated Web UI testing method based on a visual language model according to claim 1, characterized in that, Step S3 includes the following steps: S3-1, For each visual interface element Location information Extract the corresponding visual region feature vector. Then, a global average pooling operation is performed on it to obtain a normalized visual feature vector. : Where GAP(·) represents the global average pooling function, and ||·|| represents the L2 norm; S3-2, For each structured interface element , its name information With hierarchical path information Each input is fed into a pre-trained language model to generate semantic feature vectors of the same dimension. and The two are then concatenated along the feature dimension to form a joint semantic feature vector. : Through linear projection layer splicing vectors Mapping to a space with the same dimension as the visual features yields semantic feature vectors. in To obtain learnable parameters, the CrossModalAttn cross-modal attention module is used, combined with visual region feature vectors. semantic feature vectors Enhancement is performed to obtain enhanced semantic feature vectors. : in, For semantic feature vectors, For visual feature vectors, , , Let d be a learnable linear projection matrix, d be the feature dimension, and "·" denote matrix multiplication. The Softmax function operates along the row directions of the attention weight matrix, and then the semantic feature vector is augmented. L2 normalization is performed to obtain normalized enhanced semantic feature vectors. : S3-3, Calculate the normalized enhanced semantic feature vector. With normalized visual feature vectors Cosine similarity between As a cross-modal matching score: S3-4, Based on the aforementioned similarity score When the similarity score is greater than the set threshold The system considers element matching. When multiple visual interface elements match the same structured interface element, only the visual interface element with the highest similarity is retained to establish a one-to-one mapping relationship with that structured interface element. The remaining visual interface elements are marked as unmatched items. For successfully matched element pairs, their visual features, location information, index information, semantic description information, hierarchical path information, and interaction state information are fused to generate an interface element containing complete attributes. .

5. The automated Web UI testing method based on a visual language model according to claim 1, characterized in that, Step S4 includes the following steps: S4-1. Construct a dynamic cue sequence P, which contains the following four parts: (a) Current observation space: consisting of the current web interface screenshot and its corresponding structured text explanation, which includes the name, hierarchical path and interaction state information of each interactive element, used to assist the model in understanding visual content; (b) Target task description: the set of test tasks extracted in step S1. (c) Historical Action Memory Space: Records the sequence of actions executed within the current test task. ; S4-2. Input the prompt sequence P into the visual language model VLM to drive it to generate the current time step. Intermediate inference results With operating instructions ,satisfy: in This represents the current observation space, including screenshots of the web application interface and the complete set of interface elements. , Represents a set of test tasks. Represents the memory space of historical actions, and the intermediate reasoning results. It contains two semantic parts: (a) an environment-aware description: an objective statement of the current interface layout, the state of relevant elements, and the unfinished tasks; and (b) a logical deduction process: based on the test task set Tasks and the current observations. This involves analyzing which operation should be performed, evaluating the relevance of each candidate element, and logically deriving why the operation was selected. The operation instructions are then defined. The action is selected from a predefined action space, which includes: Click (element: int), Text (element: int, text_input: str), Hover (element: int), and Stop; where the element parameter is a unique identifier of the target element matched in the complete set of interface elements. S4-3, Generate the operation instructions The data is sent to the web browser's automated execution engine to simulate real user interaction behavior. After execution, the updated interface screenshot and accessibility tree are captured. Steps S2 to S3 are re-executed to generate a new set of complete interface elements, which is then used as the observation input for the next time step t+1. ; S4-4. Repeat S4-1 to S4-3 until the model outputs the Stop operation, or the test process ends when the preset maximum number of iterations is reached.

6. An automated Web UI testing system based on a visual language model, characterized in that, The system is configured as follows: Get the test script And parse out the test task set from it. Each test task Includes an operation action and the target object to which the action is performed. ; Based on screenshots of the current web application interface, a set of visual interface elements is extracted using a visual language model. Based on the accessibility tree corresponding to the web application interface, a set of structured interface elements is extracted using preset parsing rules. The visual interface elements contain location information. Index information and semantic description information The structured interface elements include name information. Hierarchical path information and interaction status information ; Employing a multimodal semantic alignment algorithm, and fusing a set of visual interface elements. With structured interface element collection The algorithm performs cross-modal matching based on visual and semantic features, and generates a fused complete set of interface elements based on the matching results. ; The test task set With the complete set of interface elements As contextual prompts, the information is input into the visual language model; through the model's automatic observation of the current interface state, logical reasoning of task semantics, and generation and execution of operation sequences, the Web UI automated testing process is iteratively completed.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-5.

Images