A large model-based scene-aware user interface autonomous testing method and system

Abstract

The application discloses a scene-aware user interface autonomous testing method and system based on a large model. The method comprises the following steps: receiving a natural language use case script and static reference data, converting the natural language use case script and the static reference data into standardized test cases by using a large language model and a functional area experience library; collecting multi-modal features such as enhanced DOM, control tree, screenshot and OCR text in real time for Android and Web platforms respectively, constructing a structured element tree model and a functional area experience library, and forming a dual-platform adaptation dictionary through LLM classification; when executing the test cases, abstract element identifiers are converted into platform-specific positioning expressions through the dictionary; when element positioning fails, a difference tree is constructed and pruned through a tree edit distance, and the optimal matching path is determined by combining multi-dimensional similarity voting and automatically updating the script positioning. The application realizes natural language-driven cross-platform adaptive testing, effectively solves the problems of non-standard control recognition, use case logic loss and script vulnerability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, automated testing, and software quality assurance, specifically to a method and system for autonomous testing of scene-aware user interfaces based on a large model, applicable to graphical user interface testing on Web and Android platforms. Background Technology

[0002] Graphical user interfaces (UIs) have become the primary medium for software interaction with users. With the development of agile development and continuous integration, the frequency of software interface iterations is constantly increasing. Test objects also include custom rendering areas that lack standard element attributes, such as Canvas, SurfaceView, and custom Views. Traditional automated test scripts that rely on fixed element identifiers, CSS selectors, or XPath are prone to failure due to changes in interface structure or element attributes.

[0003] Existing cross-platform testing solutions typically call native testing tools from different platforms. Since the interface structure, element attributes, and location expressions differ between platforms, the same test logic is difficult to reuse directly. While image-based or OCR-based location solutions can identify some non-standard controls, the recognition results are usually independent of the native interface hierarchy, making it difficult to express the relationships between virtual controls and their container, surrounding elements, and business operations.

[0004] Using large language models to parse natural language test cases can reduce script writing costs, but natural language may contain ambiguous element descriptions, missing pre-processing steps, or omitted expected results. Furthermore, without constraints from interface structure, scene context, and existing operational experience, the executability and consistency of the generated results remain insufficient. In addition, existing script repair methods typically replace only a single positioning attribute, making it difficult to reliably determine replacement elements when element text, position, visual style, and hierarchical relationships change simultaneously. Therefore, a UI testing solution is needed that can uniformly represent native elements and virtual controls across different platforms, generate platform-independent test steps, and autonomously relocate when positioning fails. Summary of the Invention

[0005] Purpose of the invention: This invention provides a scene-aware UI autonomous testing method and system based on a large model to solve the problems of difficulty in unifying interface structures across different platforms, lack of structured representation for non-standard controls, incomplete logic in natural language test cases, and failure of element location due to interface changes.

[0006] Technical solution: A method for autonomous testing of scene-aware user interfaces based on large models, comprising the following steps:

[0007] The native interface hierarchy and interface images of different candidate test platforms are obtained, virtual controls that are not represented by the native interface hierarchy are identified, virtual nodes corresponding to each virtual control are attached to a custom rendering container node, and multimodal features of native elements and virtual controls are recorded according to a unified node structure to form a target enhancement element tree for each candidate test platform.

[0008] Based on the multimodal features of elements in each target enhancement element tree, their functional scenarios, and their positions in the functional operation sequence, elements with the same operational semantics in different candidate test platforms are mapped to the same abstract element identifier, and the abstract element identifier and its element location expression in different candidate test platforms are recorded in the mapping entries.

[0009] Using a multimodal large language model, based on natural language use case scripts, scenario description texts, enhancement element trees for each target, and functional operation sequences, we determine the abstract element identifiers corresponding to each operation step in the natural language use case scripts and fill in the missing pre-operation steps, generating standardized test cases that reference the abstract element identifiers.

[0010] The execution platform for this test is determined. Based on the mapping entries, the abstract element identifiers in the standardized test cases are converted into element location expressions for the execution platform and executed. When the target element is not found, the interface data of the execution platform is reacquired to form a real-time enhanced element tree. The range of candidate elements is determined based on the structural differences between the target enhanced element tree and the real-time enhanced element tree of the execution platform. The replacement element is determined based on the multimodal feature similarity between the candidate element and the target element. The mapping entries corresponding to the abstract element identifier of the target element are updated using the element location expressions of the replacement elements, and then execution continues.

[0011] Furthermore, the virtual nodes corresponding to each virtual control are attached to the custom rendering container node, including:

[0012] Get the container bounding box of each custom rendering container node and the control bounding box of each virtual control;

[0013] Define the custom rendering container node, whose container bounding box contains the control bounding box, as the parent node of the virtual control;

[0014] When a virtual control is located inside a nested custom rendering container node, the inner custom rendering container node is determined as the parent node of the virtual control.

[0015] Write the virtual node corresponding to the virtual control into the virtual child node set of the parent node, and retain the original native child node relationship of the parent node.

[0016] Furthermore, when the candidate test platform is a Web platform, the native interface hierarchy is a document object model tree, the custom rendering container nodes include Canvas nodes, and the corresponding target enhanced element tree is an enhanced document object model tree.

[0017] When the candidate test platform is Android, the native interface hierarchy is a control tree, the custom rendering container node includes SurfaceView node or custom View node, and the corresponding target enhancement element tree is an enhancement control tree.

[0018] The unified node structure includes node type, node identifier, parent node identifier, native attributes, text attributes, visual attributes, layout attributes, and interaction attributes. Virtual nodes also include virtual control categories and control bounding boxes.

[0019] Furthermore, multimodal features include text features, visual features, layout features, and interaction attribute features;

[0020] Text features are obtained from at least one of native text, optical character recognition text, and description tags of virtual controls;

[0021] Visual features are extracted from the local image regions corresponding to the elements;

[0022] The layout features are determined based on the element's position relative to its parent node, element size, hierarchy depth, local element density, and neighboring element density.

[0023] Interactive attribute features are used to characterize at least one of the following states of an element: visible state, available state, clickable state, and editable state.

[0024] Furthermore, the functional operation sequence includes ordered abstract operation steps corresponding to the same business function, multimodal interface features corresponding to each abstract operation step, and interface state switching markers.

[0025] Input the element type, parent element type, text content, serialized multimodal features, and the element's position in the functional operation sequence into the element semantic classification model to obtain the standardized interaction semantics of the element, and use the standardized interaction semantics as the abstract element identifier;

[0026] The mapping entries record the abstract element identifier and its element location expression under different candidate test platforms.

[0027] Furthermore, the functional operation sequence is constructed in the following way:

[0028] Multiple raw operation sequences are extracted from the application operation video. The raw operation sequences include ordered operation steps and the interface states corresponding to each ordered operation step.

[0029] Remove platform-specific location information from the original operation sequence, and group original operation sequences belonging to the same business function into the same sequence set according to operation order, element function, and interface semantics.

[0030] Based on the common operation steps in the sequence set, a platform-independent standard functional operation sequence is generated, and the differentiated operations of different candidate test platforms are recorded as platform adaptation information associated with the standard functional operation sequence.

[0031] When generating standardized test cases, the corresponding standard functional operation sequence is determined based on the scenario description text, and the missing pre-operation steps in the natural language test case script are supplemented by the standard functional operation sequence and its platform adaptation information.

[0032] Furthermore, the range of candidate elements is determined based on the structural differences between the target augmentation element tree and the real-time augmentation element tree of the execution platform, including:

[0033] Based on the tree edit distance between the target augmented element tree and the real-time augmented element tree, determine the structural difference regions where node insertion, deletion, replacement, or movement occurs.

[0034] Determine the target node in the target augmentation element tree corresponding to the failed element location expression;

[0035] Starting from the target node, trace upwards to the parent node and downwards to its direct child nodes, pruning irrelevant top-level containers and static display nodes, and identifying the pruned elements as candidate elements.

[0036] Furthermore, replacement elements are determined based on the multimodal feature similarity between candidate elements and target elements, including:

[0037] Calculate the text similarity, visual similarity, and spatial similarity between each candidate element and the target element;

[0038] The comprehensive matching score of each candidate element is calculated based on text similarity, visual similarity, and spatial location similarity, and the candidate elements are sorted according to the comprehensive matching score.

[0039] Perform interactive operations on the candidate elements according to the sorting results, and verify the interactive attributes of the candidate elements and whether the coordinates of the candidate elements are within the valid range of the page;

[0040] The candidate element that passes the verification is identified as the replacement element. The element positioning expression of the replacement element is associated with the abstract element identifier corresponding to the target element, and the mapping entry is updated.

[0041] This invention also provides an autonomous test and control system for scene-aware user interfaces based on a large model, comprising:

[0042] The enhanced element tree construction module is configured to obtain the native interface hierarchy and interface images of different candidate test platforms, identify virtual controls that are not represented by the native interface hierarchy, attach the virtual nodes corresponding to each virtual control to the custom rendering container node, and record the multimodal features of native elements and virtual controls according to a unified node structure to form the target enhanced element tree for each candidate test platform.

[0043] The cross-platform mapping module is configured to map elements with the same operational semantics in different candidate test platforms to the same abstract element identifier based on the multimodal features, functional scenarios, and positions of elements in the functional operation sequence of each target enhancement element tree, and to record the abstract element identifier and its element location expression in different candidate test platforms in the mapping entry.

[0044] The test case generation module is configured to use a multimodal large language model to determine the abstract element identifiers corresponding to each operation step in the natural language test case script and fill in the missing pre-operation steps based on the natural language test case script, scenario description text, each target enhancement element tree and functional operation sequence, and generate standardized test cases that reference the abstract element identifiers.

[0045] The execution and repair module is configured to determine the execution platform for this test. Based on the mapping entries, it converts the abstract element identifiers in the standardized test cases into element location expressions for the execution platform and executes them. When the target element is not found, it reacquires the interface data of the execution platform to form a real-time enhanced element tree. Based on the structural differences between the target enhanced element tree and the real-time enhanced element tree of the execution platform, it determines the range of candidate elements. Based on the multimodal feature similarity between the candidate elements and the target element, it determines the replacement element. After updating the mapping entries corresponding to the abstract element identifiers of the target element using the element location expressions of the replacement elements, it continues execution.

[0046] The present invention also provides an electronic device, comprising: a processor; a memory; and a computer program, the computer program being stored in the memory and configured to be executed by the processor, wherein the computer program, when executed by the processor, implements the autonomous testing method for scene-aware user interfaces based on a large model as described above.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the autonomous testing method for scene-aware user interfaces based on a large model as described above.

[0048] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the autonomous testing method for scene-aware user interfaces based on a large model as described above.

[0049] Beneficial Effects: This invention unifies the native interface hierarchy of different candidate testing platforms with virtual controls identified from a custom rendering container to construct a target enhanced element tree. This enables non-standard controls lacking native element entities to possess searchable hierarchical relationships, text attributes, visual attributes, layout attributes, and interactive attributes. By establishing mapping entries between platform-independent abstract element identifiers and element location expressions under different candidate testing platforms, the differences in element location across different platforms are isolated, allowing natural language testing logic to be converted into standardized test cases that can be reused across platforms. When element location fails, the range of candidate elements is determined based on the structural differences between the target enhanced element tree and the real-time enhanced element tree. Substitute elements are determined by combining the textual similarity, visual similarity, and spatial similarity between candidate elements and target elements. After updating the corresponding mapping entries, execution continues, thereby improving the testability of non-standard controls, the cross-platform reusability of test cases, and the adaptability of test scripts to interface changes. Attached Figure Description

[0050] Figure 1 This is the overall design roadmap of the present invention;

[0051] Figure 2 This is a schematic diagram of the distributed scheduling framework in this invention;

[0052] Figure 3 This is a flowchart of the method of the present invention;

[0053] Figure 4 This is a schematic diagram of the multimodal data acquisition process in this invention;

[0054] Figure 5 This is a schematic diagram of the cross-platform modeling technology in this invention;

[0055] Figure 6 This is a schematic diagram of the present invention;

[0056] Figure 7 This is a schematic diagram of the dynamic execution and repair technology in this invention. Detailed Implementation

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the technical concept of the present invention and are not intended to limit the scope of protection of the present invention.

[0058] This invention addresses the problems in existing UI automated testing, such as insufficient cross-platform adaptation, difficulty in recognizing non-standard controls, lack of natural language test case logic, and script failure caused by interface changes. It sequentially completes multimodal data collection, target enhancement element tree construction, cross-platform element mapping, standardized test case generation, test execution, and element relocation.

[0059] Figure 1 The overall design roadmap of this invention is illustrated. The input layer receives natural language use case scripts and scenario description text; the data acquisition layer acquires the native interface hierarchy and interface images of the Web and Android platforms, and extracts the text, visual, layout, and interaction attributes of elements; the cross-platform modeling layer uses a visual model to detect non-standard controls in the interface images, and uses a GUI screenshot / OCR module to complete screenshot processing, OCR text extraction, and coordinate transformation. The detected virtual nodes are merged with native nodes to form a target augmented element tree for each candidate test platform, and a mapping entry is established between abstract element identifiers and element location expressions for each platform. The large language model combines the natural language use case scripts, scenario description text, target augmented element tree, and functional operation sequence to determine the abstract element identifiers referenced in each operation step.

[0060] The execution and repair layer comprises a planning module, a dynamic execution module, and a result analysis module. The planning module generates execution instructions based on abstract element identifiers, mapping entries, and the execution platform. The dynamic execution module invokes the execution engine corresponding to the execution platform to perform operations such as clicks, inputs, swipes, and assertions, and sends the execution results to the result analysis module. When an element location expression fails, the result analysis module compares the target enhanced element tree with the real-time enhanced element tree, determines a replacement element within the candidate element range defined by structural differences, updates the mapping entries, and continues executing standardized test cases. The output layer summarizes the step execution status, exception logs, exception interface images, element relocation records, and resource status to generate a test report.

[0061] This method can be derived from... Figure 2 The user interface test control system shown is implemented. The test control system includes a master node and multiple execution nodes. The master node is responsible for task reception, parsing, allocation, resource management, and progress monitoring, while the execution nodes are responsible for interface data acquisition, test operations, and execution data feedback.

[0062] In a distributed deployment, Android execution nodes can integrate Appium and UIAnalysisKit, while web execution nodes can integrate Selenium and Chrome DevTools. The master node allocates tasks based on platform capabilities, node load, and health status, and transfers unfinished tasks to available nodes on the same platform in case of node failure. Nodes can transmit control commands via RESTful interfaces and report status through Netty-based TCP long connections and protobuf serialization protocol.

[0063] In this paper, the candidate test platform is the platform that participates in the construction of the enhancement element tree and cross-platform mapping, and can include the Web platform and the Android platform; the execution platform is the platform determined from the candidate test platforms that is actually executed for this standardized test case. (Refer to...) Figure 3 The method specifically includes the following steps.

[0064] S1. Obtain natural language test case scripts, scenario description texts, and baseline data for candidate test platforms. Natural language test case scripts are written by testers using natural language to describe the operations to be performed, input data, and expected results. These scripts may include complete steps, or vague descriptions of elements, or omit certain preliminary steps. Scenario description texts describe the test objectives, candidate test platforms, browser or device constraints, platform selection constraints, and business scenario constraints.

[0065] The benchmark data includes benchmark interface images and benchmark native interface hierarchical structures. After virtual control identification and node fusion in step S2, the benchmark native interface hierarchical structure forms a target enhanced element tree, serving as a reference for cross-platform modeling, standardized use case generation, and subsequent relocation. The native interface hierarchical structure of the Web platform is a DOM tree, while the native interface hierarchical structure of the Android platform is a control tree.

[0066] The baseline interface image and the baseline native interface hierarchy correspond to the same interface state. Natural language test case scripts and scenario description text serve as inputs to the test tasks and are associated with the corresponding baseline data.

[0067] S2. Obtain the native interface hierarchy structure and interface image of different candidate test platforms respectively, identify virtual controls that are not represented by the native interface hierarchy structure, attach the virtual nodes corresponding to each virtual control to the custom rendering container node, and record the multimodal features of native elements and virtual controls according to a unified node structure to form the target enhancement element tree of each candidate test platform.

[0068] like Figure 4 As shown, the specific methods for multimodal data acquisition and multimodal feature extraction are as follows.

[0069] (1) Data collection on the Android platform

[0070] For the Android platform, the native control type, resource identifier, content description, text content, boundary coordinates, parent-child relationship, and interactive attributes such as visibility, availability, clickability, and editability of native controls are obtained through the control extraction interface to form a native control tree. Since the native control tree usually lacks visual information such as color, texture, and shape, the corresponding interface image is also obtained, and local image areas of each control are cropped from the interface image based on the control boundary coordinates.

[0071] Interface images can be captured using layered acquisition and alpha fusion; for sensitive interfaces with a screen capture prohibition flag, images can be extracted from the terminal screen recording frames.

[0072] For custom rendering containers such as SurfaceView or custom Views, object detection models (e.g., the YOLOv8 lightweight object detection model) are used to detect buttons, sliders, checkboxes, dropdown menus, or other interactive areas. Contour detection is combined to determine the shape and bounding box, and OCR engines such as PaddleOCR are used to extract the control text. For virtual controls that have neither native text nor valid OCR text, descriptive labels are generated based on shape, icon category, and adjacent text. Each detection result is encapsulated as a virtual node.

[0073] (2) Web platform data collection

[0074] For web platforms, browser automation frameworks like Selenium are used to obtain the tag name, element identifier, style class name, text content, input values, and interactive attributes of standard DOM nodes. The layout tree is then obtained through interfaces such as DOMSnapshot in the Chrome DevTools protocol, recording the parent-child hierarchy, absolute position, relative position, element size, and display hierarchy of each DOM node. Standard DOM nodes can be positioned using element identifiers, CSS selectors, or DOM paths.

[0075] For custom rendering containers such as Canvas, the interface image of the Canvas area is obtained through methods such as the Selenium screenshot interface getScreenshotAs, ensuring that the capture resolution is consistent with the page rendering resolution. Object detection models such as YOLOv8 are used to identify visual objects such as buttons, sliders, checkboxes, menus, and pop-ups within the Canvas. Contour detection or Hough transform is used to correct the bounding boxes of controls with specific shapes, and OCR is combined to extract text labels. Controls detected but not represented by the standard DOM are identified as virtual controls.

[0076] (3) Unified node structure and virtual node connection

[0077] A unified node structure ensures consistent data standards for native DOM nodes on the Web platform, native control nodes on the Android platform, and virtual nodes obtained through image recognition. Native nodes must record at least the node type, node identifier, parent node identifier, native attributes, text attributes, visual attributes, layout attributes, and interaction attributes; virtual nodes also record the virtual control category, control bounding box, description label, and virtual node identifier.

[0078] For the Web platform, the Canvas virtual node is attached to the Canvas node; for the Android platform, the View virtual node is attached to the SurfaceView node or a custom View node. Specifically, the container bounding box of each custom rendering container node and the control bounding box of each virtual control are obtained. The custom rendering container node whose container bounding box contains the control bounding box is determined as the parent node of the virtual control. When the virtual control is located within a nested custom rendering container node, the inner custom rendering container node is determined as the parent node. The virtual node corresponding to the virtual control is written to the virtual child node set of the parent node, while preserving the original native child node relationship of the parent node.

[0079] Virtual child nodes and native child nodes are stored separately, thus preserving the parent-child relationships in the native UI hierarchy. Virtual child nodes can be sorted according to display hierarchy and vertical coordinates. After merging, native elements and virtual controls can be traversed, retrieved, and associated within the same enhanced element tree.

[0080] More specifically, native Web nodes can be represented using JSON or XML, with `nodeType` set to `html-native`, and fields such as `tagName`, `id`, `originAttr`, `visualMeta`, `layoutMeta`, `ocrText`, `interactMeta`, `childNodes`, and `virtualChildren` set. `visualMeta` records background color, text color, and shape; `layoutMeta` records absolute bounding box, normalized relative bounding box, and display hierarchy; `interactMeta` records interactive attributes such as `clickable` and `editable`. Canvas virtual nodes can have `nodeType` set to `canvas-virtual`, and fields such as `virtualId`, `shapeType`, `fillColor`, `borderColor`, `labelText`, `ocrBbox`, `layoutMeta`, and `interactMeta` set. Android native nodes use the same field definitions as virtual nodes; Android virtual nodes can have `nodeType` set to `view-virtual`.

[0081] Here is a JSON example of a native web button node:

[0082] {

[0083] "nodeType": "html-native",

[0084] "tagName": "button",

[0085] "id": "submit-btn",

[0086] "originAttr": {"class": "primary", "textContent": "Submit"},

[0087] "visualMeta": {"bgColor": "#1677ff", "textFontColor": "#ffffff", "shape": "rect"},

[0088] "layoutMeta": {"absBbox": [100, 320, 180, 360], "relativeBbox":[0.12, 0.41, 0.22, 0.46], "zIndex": 99},

[0089] "ocrText": "Submit",

[0090] "interactMeta": {"clickable": true, "editable": false},

[0091] "childNodes": [],

[0092] "virtualChildren": []

[0093] }

[0094] The following is a JSON example of a Canvas virtual control node:

[0095] {

[0096] "nodeType": "canvas-virtual",

[0097] "virtualId": "canvas-v-001",

[0098] "shapeType": "circle",

[0099] "fillColor": "#ffffff",

[0100] "borderColor": "#cccccc",

[0101] "labelText": "Agree to the privacy policy",

[0102] "ocrBbox": [110, 450, 140, 470],

[0103] "layoutMeta": {"absBbox": [105, 445, 135, 475], "relativeBbox":[0.13, 0.56, 0.17, 0.60]},

[0104] "interactMeta": {"clickable": true, "editable": false},

[0105] "childNodes": []

[0106] }

[0107] The following is an example of the JSON structure of the Canvas node after the virtual node is attached, where childNodes stores the original child nodes and virtualChildren stores the detected virtual nodes:

[0108] {

[0109] "nodeType": "html-native",

[0110] "tagName": "canvas",

[0111] "id": "canvas-privacy",

[0112] "originAttr": {"width": 800, "height": 1200},

[0113] "visualMeta": {"bgColor": "#f5f5f5"},

[0114] "layoutMeta": {"absBbox": [0, 300, 800, 1000]},

[0115] "childNodes": [],

[0116] "virtualChildren": [

[0117] {"nodeType": "canvas-virtual", "virtualId": "canvas-v-001", "shapeType": "circle", "labelText": "Agree to the agreement", "layoutMeta": {"absBbox":[105, 445, 135, 475]}, "interactMeta": {"clickable": true}},

[0118] {"nodeType": "canvas-virtual", "virtualId": "canvas-v-002", "shapeType": "rect", "labelText": "Next", "layoutMeta": {"absBbox": [220,600, 380, 650]}, "interactMeta": {"clickable": true}} ]

[0120] }

[0121] Android custom view virtual nodes use the same data structure, for example, setting nodeType to view-virtual, virtualId to view-v-001, and writing it to the virtualChildren collection of the SurfaceView node or custom view node.

[0122] (4) OCR text extraction

[0123] OCR text extraction comprises an interface layer, a recognition layer, and a post-processing layer. The recognition layer can be configured with PaddleOCR and EasyOCR: clear text is recognized by PaddleOCR, while blurry, skewed, or low-contrast text is recognized and fused by both engines after image enhancement; the post-processing layer merges adjacent discrete text boxes.

[0124] For web pages, text-containing areas can be cropped based on the layout tree, and then the text can be identified using exact matching or regular expression matching. The search scope can then be limited using region conditions. The OCR text box coordinate formula is: (absolute coordinates - coordinates of the top-left corner of the page) / page size, converted to normalized coordinates.

[0125] (5) Multimodal feature extraction

[0126] Multimodal features include text features, visual features, layout features, and interaction attribute features. Text features can be obtained by encoding native text, OCR text, or virtual control description tags using Sentence-BERT; visual features can include color, texture, shape, image embedding, contour moments, and color histograms, among which custom controls without text and Canvas virtual controls are mainly identified based on visual features.

[0127] In one embodiment, the visual feature vector of the AugNet output element is concatenated with the outline moments and RGB histogram of the Canvas dynamic element; Boolean interactive attributes are encoded using one-hot encoding, and the coordinates and dimensions are normalized.

[0128] Layout features are obtained by enhancing the association between the element tree and the layout matrix. The enhanced element tree is recursively traversed, associating each node with its corresponding record in the layout matrix. This records the element's position relative to its parent node, element size, hierarchy depth, and interaction type. The interface is divided into multiple grids based on normalized coordinates. The number of elements in each grid is counted, and the local element density of the element's grid and the neighboring element density of adjacent grids are written back to the corresponding nodes. Finally, the relative position, size, hierarchy depth, local element density, and neighboring element density are combined to form the layout features. The web platform can also record layout parameters such as Flex orientation or Grid templates.

[0129] Specifically, the forward mapping from the control tree to the layout matrix is ​​achieved by recursively traversing all nodes. The unique nodeId of each control is used as the row index of the layout matrix, and the matrix columns are standardized to store relativeX, relativeY, layerDepth, control size, interaction type, and Flex or Grid layout parameters of the Web platform. The reverse mapping from the layout matrix to the control tree uses the nodeId as the primary key, and the coordinates and density statistics in the matrix are synchronously written into the layout meta-attributes of the corresponding nodes. The 2D layout heatmap uses the normalized coordinates of all controls within the layout matrix as spatial points, divides the page into a uniform grid, and counts the number of controls in each grid. The resulting gridDensity field is simultaneously populated back into the layout matrix and the enhanced element tree. For a single control, its grid location is determined based on its relativeX and relativeY, the local density is read, and the average density of its surrounding 3×3 neighboring grids is calculated as the neighborhood distribution characteristic.

[0130] The aforementioned layout features are used to characterize the parent-child hierarchy of controls, page layout rules, and global spatial clustering distribution. For Flex layout and Grid layout on the Web platform, layout directions such as row and column, as well as Grid template information such as 3×2 grid, can also be recorded, and together with relative position, hierarchy depth, local density, and neighborhood density, they constitute the single control layout features.

[0131] Interactive attributes are used to characterize the visible, available, clickable, and editable states of an element. Display elements that are neither clickable nor editable are not considered target elements for actions, but they remain in the enhanced element tree as assertion objects, context elements, or hierarchical reference elements.

[0132] (6) Data alignment, quality verification and degraded acquisition

[0133] Interface images, native interface hierarchy, layout data, and OCR results within the same acquisition cycle are associated through acquisition batch identifiers; when deployed across nodes, the acquisition time is recorded using the synchronous system time, and the sequence is recorded using the monotonic clock within the node.

[0134] When data acquisition fails, a limited number of retries are performed at incremental time intervals. After a retry fails, the web platform is downgraded to full-screen image and full OCR acquisition, while the Android platform is downgraded to basic attribute and visual feature acquisition. Data quality verification includes checks on root nodes, coordinate ranges, image integrity, and multimodal feature integrity. Low-confidence OCR text, control data with coordinates outside the page range, or abnormal sizes are filtered out. Sensitive information in the acquired data is protected using encryption.

[0135] S3. Based on the multimodal features of elements in each target enhancement element tree, their respective functional scenarios, and their positions in the functional operation sequence, map elements with the same operational semantics in different candidate test platforms to the same abstract element identifier, and record the abstract element identifier and its element positioning expression in different candidate test platforms in the mapping entry.

[0136] like Figure 5 As shown, cross-platform modeling specifically includes:

[0137] (1) Construction of structured element tree model

[0138] A structured element tree is constructed based on the parent-child hierarchy, spatial location, and interactive attributes of the target enhanced element tree. Elements can be divided into window elements, clickable elements, input elements, and display elements. Window elements include the main interface, child windows, and pop-up windows; clickable elements include buttons, radio buttons, checkboxes, and links; input elements include text boxes, password boxes, and date input controls; and display elements include labels, lists, and icons.

[0139] The structured element tree records operational dependencies while preserving the original hierarchical relationships. For a valid terminal node that serves as a core interactive control, controls that are siblings of that valid terminal node and have the attribute of input are identified as condition nodes, and the condition nodes are associated with the predecessor path of the valid terminal node. For example, the preconditions of the login submit button may include an account input box and a password input box, thereby forming the operational dependency of entering account - entering password - clicking login.

[0140] (2) Construction of functional operation sequence

[0141] The functional operation sequence includes ordered abstract operation steps corresponding to the same business function, multimodal interface features corresponding to each abstract operation step, and interface state switching markers. The functional operation sequence is extracted from pre-recorded or offline collected application operation videos, also known as the functional area experience library.

[0142] For operation videos, frame rate standardization is performed first, then touch indicators are detected and click, long press and swipe actions are identified, and continuous keyboard input is aggregated into input actions; the boundaries of functional operation sequences are determined based on the similarity between continuous interface frames.

[0143] Multiple original operation sequences are extracted from different applications or candidate test platforms. Platform-specific positioning information is removed, and original operation sequences belonging to the same business function are grouped into the same sequence set based on operation order, element function, and interface semantics. Platform-independent standard functional operation sequences are generated based on the common operation steps in the set, and the differentiated steps required for execution on different candidate test platforms are recorded as platform adaptation branches associated with the standard functional operation sequences. Platform-differentiated steps are not directly deleted to avoid the standard sequences failing to execute on specific platforms.

[0144] Sequence normalization includes preprocessing encoding, semantic aggregation, standard sequence fusion, and mapping binding. A language model trained on business function classification and action sequence alignment can be used to categorize similar original sequences into corresponding business functions, generating standard function operation sequences. Sequence identifiers, business function types, ordered operation steps, interface multimodal features, and platform-adaptive branches are then stored in a function operation sequence library.

[0145] (3) Construction of abstract element identifiers and mapping entries

[0146] The element semantic classification model takes element type, parent element type, text content, serialized multimodal features, and sequence position as input, and outputs standardized operational semantics and control semantics. Sequence position can be divided into start, mid, and end, used to distinguish elements that are similar in text or visually but have different business positions. The functional operation sequence classification model and the element semantic classification model are fine-tuned separately. They can share pre-trained weights, but their training samples, optimization objectives, and inference outputs are distinct.

[0147] The standardized interactive semantics output by the element semantic classification model serve as a unified abstract element identifier across platforms. For example, submit buttons in different candidate test platforms can be mapped to the same abstract element identifier representing a common submit operation.

[0148] Elements with the same standardized interaction semantics across different candidate testing platforms are mapped to the same abstract element identifier. Each mapping entry records the abstract element identifier and its element location expression under different candidate testing platforms. The element location expression for the Web platform may include the element identifier, CSS selector, or DOM path; the element location expression for the Android platform may include the resource identifier or control hierarchy path.

[0149] When an element that has not yet been recorded is collected, its platform-native localization expression and multimodal features are input into the element semantic classification model to obtain the corresponding abstract element identifier. If the abstract element identifier already exists in the mapping entry, the element localization expression under the corresponding candidate test platform is added; otherwise, a new mapping entry is created. Thus, the same standardized test case can obtain specific localization expressions under different candidate test platforms by querying the mapping entry.

[0150] S4. Using a multimodal large language model, based on the natural language use case script, scenario description text, the enhancement element tree of each target, and the functional operation sequence, determine the abstract element identifier corresponding to each operation step in the natural language use case script, fill in the missing pre-operation steps, and generate standardized test cases that reference the abstract element identifier.

[0151] Reference Figure 6 The test case generation method is as follows.

[0152] (1) Prompt template and context

[0153] The prompt template includes role and task constraints, input data and context descriptions, output structure, and validation rules. Role and task constraints limit the model task to UI automation test case parsing; input data and context descriptions define the input format of natural language test case scripts, scenario description text, target enhancement element trees, interface images, and functional operation sequences; output structure and validation rules define the output structure of test case identifiers, target platforms, preconditions, operation steps, operation parameters, and expected results.

[0154] During element matching, the matching priority is determined according to the order of text, element category, color, and coordinates. For vague descriptions such as "click payment-related controls" or "enter a valid mobile phone number," the business scope is determined using the scenario description text, the actual candidate elements are identified using the target-enhanced element tree, and the corresponding operation logic is completed using the functional operation sequence.

[0155] (2) Keyword-driven and scenario-adaptive

[0156] The keyword system includes element positioning keywords, operation keywords, scenario keywords, and natural language mapping keywords. Element positioning keywords are used to express abstract element identifiers or element filtering conditions; operation keywords are used to express clicks, inputs, or triggered operations; scenario keywords are used to retrieve corresponding functional operation sequences; and natural language mapping keywords are used to map natural language actions to standard operation keywords. Common scenarios such as login and search generate steps according to the corresponding functional operation sequences, while customized scenarios such as privacy protocol verification and sensitive interface triggering generate steps in conjunction with specific rules.

[0157] (3) Step completion and test case verification

[0158] Complete the missing prerequisite steps based on the functional operation sequence and the operation dependencies in the structured element tree. For example, if a natural language use case only describes "click to log in and verify to enter the homepage", and the current interface has not yet entered an account and password, then complete the account input and password input steps according to the login function operation sequence, and determine the corresponding abstract element identifier for each step.

[0159] Standardized test cases should include at least a test case identifier, preconditions, test steps, and expected results. Each test step should include an abstract element identifier, operation type, and operation parameters. After generation, logical verification, element verification, and cross-platform adaptation verification should be performed, and the verification results should be output.

[0160] S5. Determine the execution platform for this test. Based on the mapping entries, convert the abstract element identifiers in the standardized test cases into element location expressions for the execution platform and execute them. Rebinding after the element location expressions become invalid is completed in this step, not during the natural language parsing phase.

[0161] Reference Figure 7 Specifically, it includes:

[0162] (1) Failure capture and real-time enhancement of element tree

[0163] When the target element is not found according to the element location expression, capture the invalid element location expression, the corresponding abstract element identifier, the operation type, the execution time, the exception log, and the exception interface image. Reacquire the native interface hierarchy structure and interface image of the execution platform, and form a real-time enhanced element tree using the same virtual control recognition, node attachment, and multimodal feature extraction methods as in step S2.

[0164] (2) Structural differences and range of candidate elements

[0165] Based on the tree edit distance between the target augmented element tree and the real-time augmented element tree of the execution platform, determine the structural difference areas where node insertion, deletion, replacement, or movement occurs; determine the target node in the target augmented element tree corresponding to the failed element positioning expression; using the target node as the retrieval origin, trace upwards to the parent node and downwards to traverse the direct child nodes, prune irrelevant top-level containers and static display nodes, and determine the elements retained after pruning as candidate elements.

[0166] (3) Multimodal similarity matching

[0167] Calculate the text similarity, visual similarity, and spatial similarity between each candidate element and the target element. Text similarity is calculated based on character edit distance; visual similarity can be calculated based on local image feature matching or visual vector cosine similarity, for example, by performing Gaussian noise reduction on the control image, using the SIFT algorithm and image pyramid for multi-scale keypoint extraction, and using RANSAC to remove false matches; spatial similarity is calculated based on the normalized coordinates of the target element and the candidate element.

[0168] In a standard non-standard control matching example, the image pyramid layer number is set to 6, and the proportion of matched inliers after RANSAC processing is used as the visual similarity. In a Canvas virtual control matching example, the visual feature vector output by AugNet is concatenated with the contour moments and RGB three-color histograms to form a composite visual description vector, and the vector cosine similarity is calculated. The text similarity threshold can be set to 0.8, the Canvas visual similarity threshold can be set to 0.7, and the image pyramid layer number and similarity threshold can be adjusted according to the interface type and historical repair data.

[0169] In one embodiment, spatial location similarity is represented as

[0170]

[0171] in , The normalized coordinates of the center point of the target element. , These are the normalized coordinates of the center point of the candidate element. The width of the page canvas.

[0172] Text similarity can be represented as:

[0173]

[0174] in For the target element text, The text represents the candidate element, and EditDistance is the character edit distance.

[0175] Visual similarity can be represented as:

[0176]

[0177] Where A and B are the visual vectors of the target element and the candidate element, respectively.

[0178] The overall matching score can be expressed as:

[0179]

[0180] Where α, β, and γ are weighting coefficients, which in one embodiment are set to 0.4, 0.3, and 0.3, respectively. Candidate elements are sorted according to their overall matching scores.

[0181] (4) Validating substitute elements and updating mapping entries

[0182] Candidate elements are validated based on their overall matching scores, from highest to lowest. For the web platform, a hover operation is performed on the candidate elements; for the Android platform, a tap-to-wake operation is performed on the candidate elements. Simultaneously, the interactive attributes and coordinates of the candidate elements are verified to ensure they are within the valid range of the page. If the current candidate element fails validation, the next candidate element is validated until a replacement element is determined.

[0183] After identifying the replacement element, the element location expression of the replacement element is associated with the abstract element identifier corresponding to the target element. The mapping entry is updated, and standardized test cases continue to be executed. The relocation log records the failure location expression, the replacement location expression, the repair strategy, the repair time, and the subsequent execution results.

[0184] (5) Other abnormal handling

[0185] In addition to handling element location failures, it can also handle operation timeouts and application crashes. When an operation times out, it dynamically waits based on page loading progress and network status; when the application crashes, it restarts the application and restores the test scenario based on completed steps and the sequence of functional operations. The repair log records the failure type, original location expression, alternative location expression, repair strategy, repair time, and subsequent execution results.

[0186] Fatal anomalies such as device offline with no available replacement device or application unrecoverable trigger alarms and suspend tasks; recoverable anomalies trigger corresponding remediation strategies. In distributed deployment, tasks that have not started or are transferable can be moved to low-load nodes on the same platform.

[0187] S6. Obtain the actual execution results, logs, exception information, and repair records of standardized test cases. Compare the actual execution results with the expected results in the standardized test cases to determine the pass, fail, or non-execution status of each test step and generate a test report.

[0188] Test reports can include test case identifiers, candidate test platforms and execution platforms, execution progress, step results, failure reasons, abnormal interface images, element relocation records, operation timeout records, application recovery records, and resource usage. The visual monitoring interface can display task progress, execution environment status, anomaly statistics, and repair statistics in real time, and provides controls for pausing, terminating, or re-executing.

[0189] Obtain the actual execution results, logs, exception information, and repair records of standardized test cases. Compare the actual execution results with the expected results in the standardized test cases to determine the pass, fail, or not executed status of each test step, and generate a test report.

[0190] Test reports can include test case identifiers, candidate test platforms and execution platforms, execution progress, step results, failure reasons, abnormal interface images, element relocation records, operation timeout records, application recovery records, and resource usage. The visual monitoring interface can display task progress, execution environment status, anomaly statistics, and repair statistics in real time, and provides controls for pausing, terminating, or re-executing.

[0191] This invention also provides an autonomous testing system for scene-aware user interfaces based on a large model, comprising:

[0192] The enhanced element tree construction module is configured to obtain the native interface hierarchy and interface images of different candidate test platforms, identify virtual controls that are not represented by the native interface hierarchy, attach the virtual nodes corresponding to each virtual control to the custom rendering container node, and record the multimodal features of native elements and virtual controls according to a unified node structure to form the target enhanced element tree for each candidate test platform.

[0193] The cross-platform mapping module is configured to map elements with the same operational semantics in different candidate test platforms to the same abstract element identifier based on the multimodal features, functional scenarios, and positions of elements in the functional operation sequence of each target enhancement element tree, and to record the abstract element identifier and its element location expression in different candidate test platforms in the mapping entry.

[0194] The test case generation module is configured to use a multimodal large language model to determine the abstract element identifiers corresponding to each operation step in the natural language test case script and fill in the missing pre-operation steps based on the natural language test case script, scenario description text, each target enhancement element tree and functional operation sequence, and generate standardized test cases that reference the abstract element identifiers.

[0195] The execution and repair module is configured to determine the execution platform for this test. Based on the mapping entries, it converts the abstract element identifiers in the standardized test cases into element location expressions for the execution platform and executes them. When the target element is not found, it reacquires the interface data of the execution platform to form a real-time enhanced element tree. Based on the structural differences between the target enhanced element tree and the real-time enhanced element tree of the execution platform, it determines the range of candidate elements. Based on the multimodal feature similarity between the candidate elements and the target element, it determines the replacement element. After updating the mapping entries corresponding to the abstract element identifiers of the target element using the element location expressions of the replacement elements, it continues execution.

[0196] The present invention also provides an electronic device, comprising: a processor; a memory; and a computer program, the computer program being stored in the memory and configured to be executed by the processor, wherein the computer program, when executed by the processor, implements the autonomous testing method for scene-aware user interfaces based on a large model as described above.

[0197] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the autonomous testing method for scene-aware user interfaces based on a large model as described above.

[0198] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described autonomous testing method for a scene-aware user interface based on a large model.

Claims

1. A method for autonomous testing of scene-aware user interfaces based on a large model, characterized in that, Includes the following steps: The native interface hierarchy and interface images of different candidate test platforms are obtained, virtual controls that are not represented by the native interface hierarchy are identified, virtual nodes corresponding to each virtual control are attached to a custom rendering container node, and multimodal features of native elements and virtual controls are recorded according to a unified node structure to form a target enhancement element tree for each candidate test platform. Based on the multimodal features of elements in each target enhancement element tree, their functional scenarios, and their positions in the functional operation sequence, elements with the same operational semantics in different candidate test platforms are mapped to the same abstract element identifier, and the abstract element identifier and its element location expression in different candidate test platforms are recorded in the mapping entries. Using a multimodal large language model, based on natural language use case scripts, scenario description texts, enhancement element trees for each target, and functional operation sequences, we determine the abstract element identifiers corresponding to each operation step in the natural language use case scripts and fill in the missing pre-operation steps, generating standardized test cases that reference the abstract element identifiers. Determine the execution platform for this test, and convert the abstract element identifiers in the standardized test cases into element location expressions for the execution platform based on the mapping entries and execute them; If the target element is not found, the interface data of the execution platform is reacquired to form a real-time augmented element tree. The range of candidate elements is determined based on the structural differences between the target augmented element tree and the real-time augmented element tree of the execution platform. The replacement element is determined based on the multimodal feature similarity between the candidate element and the target element. The mapping entry corresponding to the abstract element identifier of the target element is updated using the element location expression of the replacement element, and then the execution continues.

2. The method according to claim 1, characterized in that, Attach the virtual nodes corresponding to each virtual control to the custom rendering container node, including: Get the container bounding box of each custom rendering container node and the control bounding box of each virtual control; Define the custom rendering container node, whose container bounding box contains the control bounding box, as the parent node of the virtual control; When a virtual control is located inside a nested custom rendering container node, the inner custom rendering container node is determined as the parent node of the virtual control. Write the virtual node corresponding to the virtual control into the virtual child node set of the parent node, and retain the original native child node relationship of the parent node.

3. The method according to claim 1, characterized in that, When the candidate test platform is a Web platform, the native interface hierarchy is a Document Object Model tree, the custom rendering container nodes include Canvas nodes, and the corresponding target augmented element tree is an augmented Document Object Model tree. When the candidate test platform is Android, the native interface hierarchy is a control tree, the custom rendering container node includes SurfaceView node or custom View node, and the corresponding target enhancement element tree is an enhancement control tree. The unified node structure includes node type, node identifier, parent node identifier, native attributes, text attributes, visual attributes, layout attributes, and interaction attributes. Virtual nodes also include virtual control categories and control bounding boxes.

4. The method according to claim 1, characterized in that, Multimodal features include text features, visual features, layout features, and interactive attribute features; Text features are obtained from at least one of native text, optical character recognition text, and description tags of virtual controls; Visual features are extracted from the local image regions corresponding to the elements; The layout features are determined based on the element's position relative to its parent node, element size, hierarchy depth, local element density, and neighboring element density. Interactive attribute features are used to characterize at least one of the following states of an element: visible state, available state, clickable state, and editable state.

5. The method according to claim 1, characterized in that, The functional operation sequence includes ordered abstract operation steps corresponding to the same business function, multimodal interface features corresponding to each abstract operation step, and interface state switching markers. Input the element type, parent element type, text content, serialized multimodal features, and the element's position in the functional operation sequence into the element semantic classification model to obtain the standardized interaction semantics of the element, and use the standardized interaction semantics as the abstract element identifier; The mapping entries record the abstract element identifier and its element location expression under different candidate test platforms.

6. The method according to claim 5, characterized in that, Functional operation sequences are constructed in the following way: Multiple raw operation sequences are extracted from the application operation video. The raw operation sequences include ordered operation steps and the interface states corresponding to each ordered operation step. Remove platform-specific location information from the original operation sequence, and group original operation sequences belonging to the same business function into the same sequence set according to operation order, element function, and interface semantics. Based on the common operation steps in the sequence set, a platform-independent standard functional operation sequence is generated, and the differentiated operations of different candidate test platforms are recorded as platform adaptation information associated with the standard functional operation sequence. When generating standardized test cases, the corresponding standard functional operation sequence is determined based on the scenario description text, and the missing pre-operation steps in the natural language test case script are supplemented by the standard functional operation sequence and its platform adaptation information.

7. The method according to claim 1, characterized in that, The candidate element range is determined based on the structural differences between the target augmentation element tree and the real-time augmentation element tree of the execution platform, including: Based on the tree edit distance between the target augmented element tree and the real-time augmented element tree, determine the structural difference regions where node insertion, deletion, replacement, or movement occurs. Determine the target node in the target augmentation element tree corresponding to the failed element location expression; Starting from the target node, trace upwards to the parent node and downwards to its direct child nodes, pruning irrelevant top-level containers and static display nodes, and identifying the pruned elements as candidate elements.

8. The method according to claim 7, characterized in that, Replacement elements are determined based on the multimodal feature similarity between candidate elements and target elements, including: Calculate the text similarity, visual similarity, and spatial similarity between each candidate element and the target element; The comprehensive matching score of each candidate element is calculated based on text similarity, visual similarity, and spatial location similarity, and the candidate elements are sorted according to the comprehensive matching score. Perform interactive operations on the candidate elements according to the sorting results, and verify the interactive attributes of the candidate elements and whether the coordinates of the candidate elements are within the valid range of the page; The candidate element that passes the verification is identified as the replacement element. The element positioning expression of the replacement element is associated with the abstract element identifier corresponding to the target element, and the mapping entry is updated.

9. A scene-aware user interface autonomous test and control system based on a large model, characterized in that, include: The enhanced element tree construction module is configured to obtain the native interface hierarchy and interface images of different candidate test platforms, identify virtual controls that are not represented by the native interface hierarchy, attach the virtual nodes corresponding to each virtual control to the custom rendering container node, and record the multimodal features of native elements and virtual controls according to a unified node structure to form the target enhanced element tree for each candidate test platform. The cross-platform mapping module is configured to map elements with the same operational semantics in different candidate test platforms to the same abstract element identifier based on the multimodal features, functional scenarios, and positions of elements in the functional operation sequence of each target enhancement element tree, and to record the abstract element identifier and its element location expression in different candidate test platforms in the mapping entry. The test case generation module is configured to use a multimodal large language model to determine the abstract element identifiers corresponding to each operation step in the natural language test case script and fill in the missing pre-operation steps based on the natural language test case script, scenario description text, each target enhancement element tree and functional operation sequence, and generate standardized test cases that reference the abstract element identifiers. The execution and repair module is configured to determine the execution platform for this test, and convert the abstract element identifiers in the standardized test cases into element location expressions of the execution platform according to the mapping entries and execute them. If the target element is not found, the interface data of the execution platform is reacquired to form a real-time augmented element tree. The range of candidate elements is determined based on the structural differences between the target augmented element tree and the real-time augmented element tree of the execution platform. The replacement element is determined based on the multimodal feature similarity between the candidate element and the target element. The mapping entry corresponding to the abstract element identifier of the target element is updated using the element location expression of the replacement element, and then the execution continues.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the autonomous testing method for scene-aware user interfaces based on a large model as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the autonomous testing method for scene-aware user interfaces based on a large model as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the autonomous testing method for scene-aware user interfaces based on a large model as described in any one of claims 1 to 8.

Images