UI tree acquisition method and device, electronic equipment and storage medium

Abstract

The invention provides a UI tree acquisition method and device, electronic equipment and a storage medium. The method comprises the steps that a UI component set in a current interface is obtained, attribute information of all UI components is extracted to generate an initial UI tree, initial UI tree nodes are in one-to-one correspondence with the UI components, and the attribute information at least comprises component names, area information and hierarchical relations; performing father-child node area cutting processing on the initial UI tree layer by layer to determine an effective response area of each node of the initial UI tree; and generating a cut target UI tree based on the effective response area. By directly acquiring the UI data, the limitation of image recognition is avoided, and the data acquisition efficiency is improved; according to the method, the effective response area of the UI component is dynamically adjusted through the hierarchical cutting algorithm, the accurate UI tree is generated, the problem that the response area of the nested UI element is overlapped is solved, reliable data support is provided for the UI test, and the test coverage rate and accuracy are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of software testing, and in particular to a UI tree acquisition method, device, electronic device and storage medium suitable for UI testing. Background Art

[0002] UI testing is software testing whose test target is the UI performance of the application. Currently, UI testing mainly adopts two methods: manual testing and automated testing based on image recognition. Manual testing requires testers to operate UI components one by one, which is inefficient and difficult to cover all test scenarios. Although automated testing based on image recognition has improved efficiency, it has the following technical defects: first, it relies on the visual features of the UI and cannot obtain the hierarchical structure and attribute information of the UI; second, the recognition effect of dynamic UI (such as animation, special effects) is poor; third, it is difficult to accurately handle the overlapping response areas of nested UI elements, which may cause the generated test events to be invalid. These technical defects make it difficult for existing testing methods to meet the efficiency and accuracy requirements of UI testing. Summary of the Invention

[0003] The present disclosure aims to provide a UI tree acquisition method to directly acquire UI data and perform hierarchical clipping, thereby improving the efficiency and accuracy of UI testing.

[0004] In a first aspect, the present disclosure provides a UI tree acquisition method, comprising: acquiring a set of UI components in a current interface, extracting attribute information of each UI component to generate an initial UI tree, wherein the nodes of the initial UI tree correspond one-to-one to the UI components, and the attribute information includes at least the component name, area information, and hierarchical relationship; performing parent-child node area cropping processing on the initial UI tree layer by layer to determine the effective response area of ​​each node of the initial UI tree; and generating a cropped target UI tree based on the effective response area.

[0005] In the second aspect, the present disclosure provides a UI tree acquisition device, including: a UI tree generation module, used to obtain a set of UI components in the current interface, extract the attribute information of each UI component to generate an initial UI tree; a node area cropping module, used to perform parent-child node area cropping processing on the initial UI tree layer by layer to determine the effective response area of ​​each node of the initial UI tree; a UI tree adjustment module, used to generate a cropped target UI tree based on the effective response area.

[0006] In a third aspect, the present disclosure provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to perform the steps in any of the above-mentioned UI tree acquisition methods.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements the steps in any of the above-mentioned UI tree acquisition methods.

[0008] The present invention avoids the limitations of image recognition and improves data acquisition efficiency by directly acquiring UI data; dynamically adjusts the effective response area of ​​UI components through a hierarchical cropping algorithm to generate an accurate UI tree, solves the problem of overlapping response areas of nested UI elements, provides reliable data support for UI testing, and significantly improves test coverage and accuracy.

[0009] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by practicing the above-mentioned technology of the present disclosure.

[0010] In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily apparent by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present invention are shown by way of example and not limitation, in which:

[0012] Figure 1 A flowchart of a UI tree acquisition method provided in an embodiment of the present disclosure;

[0013] Figure 2 A schematic diagram of a sub-node cropping result provided by an embodiment of the present disclosure;

[0014] Figure 3 A schematic diagram of another sub-node cropping result provided by an embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram of updating and regularizing a parent node provided by an embodiment of the present disclosure;

[0016] Figure 5 A schematic diagram of a layer-by-layer traversal sequence of a UI tree provided in an embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram of the structure of a UI tree acquisition device provided by an embodiment of the present disclosure;

[0018] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0020] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0021] In one embodiment of the present disclosure, the UI tree acquisition method can be run on a local terminal device or a server. When the UI tree acquisition method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0022] A server can be one or more servers, a server cluster consisting of several servers, or a cloud server on a cloud computing platform. A server is used to perform specific processing and interact with clients, including sending and receiving requests, instructions, and data.

[0023] Client devices can be personal computers, smartphones, tablets, e-book readers, wearable devices, devices that interact with information based on AR (Augmented Reality) / VR (Virtual Reality), etc. The client devices run a configured client program that executes applications and / or tasks within applications and interacts with the server, including, but not limited to, sending requests and data to the server and receiving data and instructions from the server.

[0024] In the field of software testing, Intelligent Monkey testing is an automated testing technology for mobile app and game UIs. Its core concept is to simulate random user actions (such as clicks, swipes, and long presses) to detect app stability, performance, and user experience issues. Unlike the completely random nature of traditional Monkey testing, Intelligent Monkey testing incorporates UI context (such as interactive elements and hierarchical structures) to generate more appropriate test events, thereby improving test effectiveness.

[0025] When generating test events for Smart Monkey testing, traditional methods typically rely on random clicks based on screen coordinates or simple component traversal. This coarse-grained event generation method is prone to two core issues: First, due to a lack of accurate perception of the hierarchical relationships and dynamic layout of UI components, random operations often fall into non-interactive areas (such as obscured components or blank spaces), resulting in invalid events; second, when UI components experience dynamic changes (such as pop-up window overlays and list scrolling), the statically acquired component coordinate information will be disconnected from the actual rendering location, causing false touches or missed tests.

[0026] Related technologies have proposed using image recognition technology to locate UI elements and generate test events. This method relies on the visual characteristics of the UI, making it difficult to handle dynamically changing UIs. It cannot obtain the UI's hierarchical structure and attribute information, has poor recognition effectiveness for dynamic UIs (such as animations and special effects), and cannot accurately handle the overlapping of nested UI elements, resulting in potentially invalid test events.

[0027] In view of this, according to the embodiment of the present invention, a UI tree acquisition method, device, electronic device and storage medium are proposed to efficiently acquire all UI data in dynamically changing scenes, and perform hierarchical clipping on the UI data to ensure that the acquired UI components do not overlap or cover each other in the page layout, thereby generating an accurate UI tree to serve the execution of intelligent MonkeyUI testing.

[0028] refer to Figure 1 As shown, in an optional implementation, the disclosed embodiment provides a UI tree acquisition method, which includes: acquiring a set of UI components in the current interface, extracting attribute information of each UI component to generate an initial UI tree, where the nodes of the initial UI tree correspond one-to-one to the UI components, and the attribute information includes at least the component name, region information, and hierarchical relationship; performing parent-child node region cropping processing on the initial UI tree layer by layer to determine the effective response region of each node in the initial UI tree; and generating a cropped target UI tree based on the effective response region. In this way, accurate acquisition and hierarchical cropping of all UI data in the current interface are achieved, providing reliable data support for intelligent Monkey testing.

[0029] Optionally, use the engine's provided interface to obtain all UI components and their attribute information in the current scene. This attribute information is used to generate an initial UI tree, which is a collection of UI components organized in a tree structure, with each node corresponding to a UI component. This ensures that the acquired UI data is complete and accurate, providing a foundation for subsequent UI tree trimming.

[0030] Optionally, the UI components and UI component properties of the current interface are obtained through the UI framework and technology stack of different platforms. For example, for Android applications, the root view (root View) can be obtained through Activity.getWindow().getDecorView() provided by the Android system, or the content view can be obtained using Activity.findViewById(android.R.id.content). Use ViewGroup.getChildCount() and ViewGroup.getChildAt() to recursively traverse the view tree and filter out nodes that are not ViewGroups and Views. Use instanceof to check the component type (such as Button / TextView) and getVisibility() to filter out invisible elements.

[0031] For the game engine Unreal Engine, use UWidgetTree::GetAllWidgets() to get the widget tree and GetPaintSpaceGeometry() to get the geometry information.

[0032] In the Unity engine, use the GameObject.FindObjectsOfType method to retrieve all active UI components in the scene. This method returns all game objects containing a RectTransform component, ensuring that the retrieved components are actual interactive UI elements. Direct retrieval through the engine interface is more efficient than image recognition technology, completing the collection of UI components across the entire scene in milliseconds.

[0033] Optionally, the attribute information of a UI component includes at least the component name, region information, and hierarchical relationship. The name is used to uniquely identify each UI component, the region information is used to determine the area range and size of each UI component on the screen, and the hierarchical relationship is used to determine the position of each node in the generated initial UI tree.

[0034] Optionally, during the initial UI tree generation process, the system establishes a bidirectional index structure. In addition to conventional parent-child pointers, a hash table fast-lookup structure is maintained, with the key being the component instance ID and the value being the tree node object. This design allows for both hierarchical traversal through the tree structure and rapid node location through the hash table during subsequent clipping, significantly improving processing efficiency.

[0035] Optionally, the region clipping of parent and child nodes uses a post-order traversal algorithm, starting from the leaf nodes and processing upwards. This processing order ensures that the clipping of all its child nodes is complete when the parent node region is calculated. The algorithm uses a stack structure to store pending nodes and combines it with a marker to prevent duplicate processing, enabling it to correctly handle complex UI tree structures containing tens of thousands of nodes.

[0036] Optionally, the region cropping process can calculate the overlapping region through the region information of the parent and child nodes, thereby excluding the overlapping portion from the respective occupied regions to obtain a valid response region.

[0037] Optionally, after pruning each node in the initial UI tree, the target UI tree contains accurate valid response areas for each node. This provides reliable data support for Smart Monkey testing, improving test coverage and efficiency.

[0038] The above method avoids the limitations of image recognition and improves data acquisition efficiency by directly acquiring UI data. It also dynamically adjusts the effective response area of ​​UI components through a hierarchical cropping algorithm to generate an accurate UI tree, solving the problem of overlapping response areas of nested UI elements. This provides reliable data support for UI testing and significantly improves test coverage and accuracy.

[0039] In an optional implementation, the set of UI components in the current interface is obtained, and the attribute information of each UI component is extracted to generate an initial UI tree. This includes: obtaining the root container of the current interface; traversing all child nodes of the root container and extracting UI components mounted on the child nodes of the root container; extracting the attribute information of each UI component and encapsulating the extracted attribute information of each UI component into a UI tree node; and generating the initial UI tree based on the hierarchical relationship between all UI tree nodes and each UI tree node. In this way, by recursively traversing all child nodes of the root node of the interface element and extracting attributes, a hierarchical UI tree is generated, laying the foundation for subsequent precise cropping.

[0040] Optionally, use the Unity engine's GameObject.FindObjectOfType method to obtain the root Canvas object in the current scene. In Unity, the Canvas object is the core component used to manage and render the UI (user interface). The Canvas object is the root container of the Unity UI system (UGUI), and all UI elements (such as buttons, text, and images) must exist as its child objects to render correctly.

[0041] Optionally, if there are multiple Canvases in the scene, you can use the Unity engine's Canvas.sortingOrder or other logic to determine the main Canvas of the current page.

[0042] Optionally, call the GetComponentsInChildren(true) method of the Transform component to traverse all child nodes of the root container. The true parameter includes inactive nodes, ensuring that even hidden UI elements can be detected. During the traversal, each node is specifically checked to see if it contains a RectTransform component, a required component for all UI elements in Unity. This filter can be used to exclude non-UI game objects, improving data processing efficiency. Optionally, each traversed child node is stored in a temporary list for subsequent UI component extraction.

[0043] Optionally, for each child node, use the GetComponent() method to check whether a UI component (such as Button, Text, Image, Slider, etc.) is mounted. If the node has a UI component mounted, it is stored in a UI component list. This ensures that the extracted UI components are complete and provides support for generating the initial UI tree.

[0044] Optionally, encapsulate the extracted properties of each UI component into a UI node object and store it in a UI node list. Traverse the UI node list and organize each UI node into a UI tree based on the parent property of the RectTransform component in the UI component. This generates a hierarchical UI tree where each node contains all the property information for the corresponding UI component, providing a foundation for subsequent UI tree pruning.

[0045] Optionally, the root container identification process includes an exception handling mechanism. When multiple Canvases with the same sorting order are detected in the scene, the system activates a secondary filtering based on rendering depth: The system calculates the screen coordinates of each Canvas's center point using Camera.WorldToScreenPoint and selects the Canvas closest to the center of the screen as the primary container. This fault-tolerant design effectively addresses situations where developers misuse sorting orders, ensuring accurate root container selection.

[0046] In an optional implementation, the attribute information of each UI component is extracted, specifically including: extracting the attribute information of each UI component's parent node and the order information of each UI component within its parent node to determine the hierarchical relationship of each UI component; generating a unique identifier containing a hierarchical path for each UI component as the component name of the UI component, where the hierarchical path is constructed by concatenating the parent node name of each UI component and an additional distinguishing identifier; and extracting the position coordinates and size information of each UI component to determine the area information of each UI component. In this way, the conflict problem of components with the same name is resolved through hierarchical path naming, ensuring the uniqueness of UI tree node identifiers while fully recording component spatial information.

[0047] Optionally, when extracting the parent node property information for each UI component, obtain the direct parent node reference through Unity's Transform.parent property, then recursively query all the way back to the root Canvas. For example, for the Button_Start component, if its parent node path is Canvas / MainPanel, the complete parent-child chain relationship is recorded. This establishes a clear hierarchical index, providing a topological foundation for subsequent cropping. Order information is obtained through Transform.GetSiblingIndex(), which obtains the order number of the UI component in the parent node as an index.

[0048] Optionally, a unique identifier is generated for each UI component by concatenating the parent node name of each UI component and appending a distinguishing identifier to form a hierarchical path. For example, for the UI component Button_Start, a path similar to Canvas / MainPanel / Button_Start is generated by concatenating the parent node names. If a child node with the same name exists under the same parent node, a sequential number is appended (such as Button_Start(1)). This ensures that the name of each UI component is unique, resolving conflicts between nodes with the same name.

[0049] Optionally, for dynamically generated UI (such as pop-up windows), a timestamp prefix (such as Popup_20240315 / OKButton) is added to ensure global uniqueness.

[0050] Alternatively, use the RectTransform.GetWorldCorners method to obtain the world coordinates of the four corners of the component, and use the width and height properties of RectTransform.rect to obtain the size of the UI component. This allows you to extract the area information of each UI component and provide support for generating the UI tree.

[0051] Optionally, for non-rectangular UI controls, the actual display area is determined by calculating the minimum bounding rectangle.

[0052] Optionally, the attribute information of a UI component also includes type information. The type of the UI component (such as Button, Text, etc.) is obtained through the GetType() method. The type of the component determines the interactive properties of the component. The enabled property of the component is used to determine whether the component is available. If it is True, it means that the current component is available, otherwise it is unavailable.

[0053] In an optional embodiment, parent-child node region cropping is performed layer by layer on the initial UI tree to determine the effective response area of ​​each node in the initial UI tree. This includes: starting from the second layer of the initial UI tree and traversing the layers one level down; performing parent-child node region cropping on each node at the same level in a predetermined traversal order; completing parent-child node region cropping in response to the last node of the last level, and obtaining the effective response area of ​​each node in the initial UI tree based on the updated region information of all nodes. In this way, the hierarchical cropping process ensures accurate calculation of the response area of ​​UI components and avoids region overlap in dynamic UI scenarios.

[0054] Optionally, start from the second-level nodes and traverse the layers in order from top to bottom. Starting from the second-level nodes is because the cropping is based on the recalculation of the parent node's area occupation based on the child nodes. The top-level nodes have no parent nodes, so there is no need to start traversing from the top-level nodes.

[0055] Optionally, for each node at the same level, the parent-child node region clipping process is performed in sequence according to a preset traversal order. The preset traversal order can be from left to right, or from right to left, to ensure that each node at the same level is processed. Figure 5 In the UI tree shown, starting from node 2 of the second layer, traverse nodes 2, 3, 4, ..., until node 10 in the order from left to right and then from top to bottom.

[0056] Optionally, after the last node is cropped, the system traverses the entire UI tree and updates the RectTransform.rect property of each node to the valid area after cropping, thereby generating an accurate target UI tree and ensuring that the acquired UI components do not overlap or cover each other on the page layout, providing a spatial positioning benchmark for subsequent test event generation.

[0057] In an alternative embodiment, for the current node to be processed, the region information of the current node and its parent node is dynamically calculated and updated based on the region information of the current node's parent node, the region information of the current node, and the region information of its sibling nodes. In this way, by dynamically calculating and updating the region information, the effective response region of each node is ensured to be accurate.

[0058] Optional, the node's region information refers to the rectangular region coordinates obtained through the RectTransform component. This includes the world coordinate positions of the top left, bottom left, top right, and bottom right vertices, as well as the size of the UI component. In the Unity engine, these coordinates are dynamically obtained through the RectTransform.GetWorldCorners method, accurately reflecting the actual position of the UI component on the screen. The size of the UI component is obtained by reading the width and height properties of RectTransform.rect.

[0059] Optionally, for the current node to be processed, the overlapping area of ​​the current node and the parent node can be determined based on the area information of the current node and the area information of the parent node of the current node. Since the sibling nodes under the same parent node may also overlap with the parent node, the area obtained by excluding these overlapping parts from the area occupied by the current node needs to be used as the effective recognition area of ​​the current node. Since the nodes are cropped and traversed layer by layer from top to bottom, the parent node of the current node does not consider the occlusion effect of the current node on the parent node when determining the effective recognition area. Therefore, when processing the current node, the effective recognition area of ​​its parent node needs to be dynamically adjusted.

[0060] In an optional embodiment, for the current node to be processed, the overlapping portion of the parent node's area and the current node's area is calculated based on the area information of the current node's parent node and the current node's area. If there is a processed sibling node at the same level, the area occupied by the processed sibling node is excluded from the overlapping portion based on the area information of the processed sibling node to obtain the updated area information of the current node. Otherwise, the updated area information of the current node is obtained based on the overlapping portion. The parent node's area information is then reversely adjusted and updated based on the updated area information of the current node. In this way, by accurately calculating and dynamically adjusting the area information of parent and child nodes, the accuracy of the UI component's response area is ensured, and false triggering of test events is avoided.

[0061] Optionally, the overlap between the parent node area and the current node area is calculated by a geometric intersection operation. Figure 2 As shown in the figure, the area occupied by the parent node of the current node is P, and the area occupied by the current node is C. By comparing the coordinate ranges of the two areas, their intersection area P∩C can be calculated. This intersection area represents the actual visible and interactive area of ​​the current node within the parent node. This ensures that subsequent cropping is performed only on the actual interactive area, improving test accuracy.

[0062] Optionally, when there are processed sibling nodes, the area occupied by the processed sibling nodes needs to be excluded from the overlapping part. This is achieved through the geometric difference operation, that is, subtracting the processed sibling node area from P∩C. Figure 3 As shown, the area occupied by the parent node of the current node is P, the area occupied by the current node is C, and the area occupied by the processed sibling nodes of the current node is S. Then the area of ​​the current node after cutting is P∩CS. This processing method can avoid test event conflicts caused by overlapping areas between nodes of the same level, and ensure that the response areas of each node are independent and do not interfere with each other. In this way, the accuracy and reliability of test events can be significantly improved. Since the cutting process is performed in sequence according to the preset traversal order for each node at the same level during layer-by-layer processing, it is only necessary to exclude the effective recognition area of ​​the processed sibling node from the effective recognition area of ​​the current node in the parent node to avoid affecting the cutting results of the processed sibling. When the preset traversal order is to process from left to right, the processed sibling node is the left sibling node of the current node, otherwise it is the right sibling node of the current node, and each sibling node has a common parent node.

[0063] Optionally, when deriving the updated region information for the current node based on the overlapped portion, if no sibling nodes at the same level have already been processed, P∩C is directly used as the updated region information for the current node. This approach simplifies the calculation process while ensuring the accuracy of the results. This improves processing efficiency while maintaining accuracy and is suitable for most simple UI layout scenarios.

[0064] Optionally, the reverse adjustment and update of the parent node's area information based on the updated area information of the current node is achieved through dynamic geometric operations. Specifically, the updated area of ​​the current node is subtracted from the parent node area P to obtain the new area of ​​the parent node, that is, the new area of ​​the parent node is obtained through the set operation P-(P∩CS) (when there are processed sibling nodes) or P-(P∩C) (when there are no processed sibling nodes). This dynamic adjustment mechanism ensures that the parent node area always reflects its actual interactive area, avoiding area calculation errors caused by factor node changes. In this way, the integrity and consistency of the UI tree structure can be maintained.

[0065] Alternatively, in practical applications, geometric operations can be performed using the Intersect and Subtract methods provided by Unity's Rect class. For example, the Rect.Intersect method calculates the intersection of two rectangles, while the Rect.Subtract method performs the difference of rectangles. These methods provide efficient geometric computation capabilities that meet the demands of real-time processing. This allows you to fully leverage the Unity engine's underlying optimizations, improving processing speed and performance.

[0066] In an optional embodiment, the parent node region of the current node is calculated based on the region information of the current node's parent node and the updated region information of the current node, excluding the remaining portion of the current node region. If the current node has no unprocessed sibling nodes of the same level, the remaining portion is regularized to generate a maximum regular geometric shape that can be completely contained by the remaining portion. Based on the region information of the maximum regular geometric shape, the region information of the parent node is updated. In this way, geometric regularization ensures that the parent node region always has an operational regular shape, avoiding test event positioning failure due to complex cropping.

[0067] Optionally, the calculation process of excluding the current node area from the parent node area uses geometric difference operations. In specific implementations, region inclusion is determined using Unity's RectTransform.rect property and the Contains / Overlaps methods provided by the Rect class, combined with vector operations to calculate the difference space of the two rectangular areas. When the parent node area is P and the child node area is C, the system first calculates the overlap between P and C, then compares vertex coordinates to determine the set of boundary points in P that are not covered by C. This allows for precise separation of the parent node area fragments that need to be retained.

[0068] Optionally, regularization specifically refers to the process of converting a complex polygonal area into a maximum inscribed rectangle. After the parent node area excludes the current node area, the area that may be obtained is irregular. Therefore, after all the child nodes of the parent node are cut, the Rect function can be used to update the area occupied by the parent node. The Rect(Region) function is used to obtain the largest rectangular area contained in the Region area, and use this area as the area occupied by the parent node to update the size and position of the parent node. Figure 4 As shown, the parent node includes two child nodes. The area occupied by the parent node of the current node is P, the area occupied by the current node is C, and the area occupied by the processed sibling node of the current node is S. Then, the area after the current node is cropped is P∩CS, and the updated area of ​​the parent node is P-(P∩CS). Since the current node is the last child node of the parent node, after updating the area of ​​the parent node, the Rect function is used to obtain the final rectangular area P' occupied by the parent node in the figure. In this way, the area information of the parent node can be ensured to be accurate and regular, improving the efficiency and accuracy of test event generation.

[0069] Optionally, after completing the parent-child node area cropping process for all nodes in the current layer, regularize the parent nodes corresponding to all nodes in the current layer. Specifically, regularize the remaining parts of all nodes in the previous layer of the current layer to generate the largest regular geometric shape that can be completely contained by the remaining parts, and update the area information of the previous layer nodes based on the area information of the largest regular geometric shape. After the traversal of the current layer nodes is completed, its parent nodes, that is, all nodes in the previous layer, have also completed the cropping process accordingly. At this time, the Rect function can be used to update the area occupied by the previous layer nodes, and use this area to update the size and position of the previous layer nodes.

[0070] In an optional embodiment, test events are generated based on the target UI tree. In this way, generating test events based on the cropped target UI tree can ensure that the tested UI components do not overlap or cover each other on the page layout, thereby improving test effectiveness.

[0071] Optionally, corresponding test events are generated based on the effective response area of ​​each node in the target UI tree. For example, for each UI component, test events such as click, slide, and long press are generated. In this way, it can be ensured that the response area of ​​each UI component can be tested, thereby improving the test coverage. The accuracy of the response area directly affects the success rate of event positioning. Since there may be overlap between the response areas of the components, if no cropping is done, the returned test event will be click A. However, due to the problem of area coverage, the area where A is clicked during execution may be responded to by other components. The target UI tree serves as the basic data structure for test event generation. Each node stores the component type and precise response area. Generating test events based on the target UI tree can cover more UI scenarios and improve test coverage.

[0072] Corresponding to the above method embodiment, the embodiment of the present invention provides a UI tree acquisition device, such as Figure 6 As shown, the device includes: a UI tree generation module, which is used to obtain a set of UI components in the current interface, extract the attribute information of each UI component to generate an initial UI tree, where the nodes of the initial UI tree correspond one-to-one to the UI components, and the attribute information at least includes the component name, area information and hierarchical relationship; a node area clipping module, which is used to perform parent-child node area clipping processing on the initial UI tree layer by layer to determine the effective response area of ​​each node of the initial UI tree; and a UI tree adjustment module, which is used to generate a clipped target UI tree based on the effective response area.

[0073] The above-mentioned device avoids the limitations of image recognition and improves data acquisition efficiency by directly acquiring UI data; it dynamically adjusts the effective response area of ​​UI components through a hierarchical cropping algorithm to generate an accurate UI tree, solves the problem of overlapping response areas of nested UI elements, provides reliable data support for UI testing, and significantly improves test coverage and accuracy.

[0074] In an optional embodiment, the UI tree generation module includes a root container acquisition unit for acquiring the root container of the current interface; a component traversal unit for traversing all child nodes of the root container and extracting UI components mounted on the child nodes of the root container; an attribute encapsulation unit for extracting attribute information of each UI component and encapsulating the extracted attribute information of each UI component into a UI tree node; and a tree structure construction unit for generating an initial UI tree based on all UI tree nodes and the hierarchical relationship of each UI tree node.

[0075] In an optional embodiment, the attribute encapsulation unit includes a hierarchical relationship determination sub-unit, which is used to extract the parent node attribute information and sequence information of each UI component to determine the hierarchical relationship; an identifier generation sub-unit, which is used to generate a unique identifier containing a hierarchical path as a component name for each UI component, and the hierarchical path is composed of splicing the parent node name and an additional distinguishing identifier; and an area extraction sub-unit, which is used to extract the position coordinates and size information of each UI component to determine the area information.

[0076] In an optional embodiment, the node area cropping module includes a hierarchical traversal unit for performing hierarchical traversal processing from the second layer of the initial UI tree to the next layer layer by layer, and for executing parent-child node area cropping processing on each node of the same layer in sequence according to a preset traversal order; an area update unit for obtaining a valid response area based on the updated area information of all nodes in response to the completion of the processing of the last node of the last layer.

[0077] In an optional embodiment, the node area clipping module further includes a dynamic calculation unit for dynamically calculating and updating the area information of the current node and the parent node based on the area information of the parent node, the area information of the current node, and the area information of sibling nodes at the same level.

[0078] In an optional embodiment, the dynamic calculation unit includes an overlapping calculation subunit, which is used to calculate the overlapping part based on the parent node area and the current node area; an area update subunit, which is used to exclude the area occupied by the processed sibling node from the overlapping part when there is a processed sibling node, and obtain the updated area information of the current node; otherwise, obtain the updated area information of the current node based on the overlapping part; and a reverse adjustment subunit, which is used to reversely adjust the area information of the parent node according to the updated area information of the current node.

[0079] In an optional embodiment, the reverse adjustment sub-unit includes a remaining area calculation sub-unit, which is used to calculate the remaining part of the parent node area excluding the current node area; a geometric regularization sub-unit, which is used to generate a maximum regular geometric shape that can be fully contained for the remaining part when there is no unprocessed sibling node of the current node; and a parent node update sub-unit, which is used to update the area information of the parent node based on the maximum regular geometric shape.

[0080] In an optional implementation, a test event generation module is further included, which is used to generate a test event based on the target UI tree.

[0081] The UI tree acquisition device provided in the embodiment of the present disclosure has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding content in the aforementioned method embodiment.

[0082] It should be noted that although several units / modules or sub-units / modules of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided and embodied by multiple units / modules.

[0083] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, the electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned UI tree acquisition method. The specific implementation can be referred to the above-mentioned method embodiment, which will not be repeated here.

[0084] Figure 7 1 is a schematic diagram of the structure of an electronic device. The electronic device 1100 includes a processor 1101 having one or more processing cores, a memory 1102 having one or more computer-readable storage media, and a computer program stored on the memory 1102 and executable on the processor. The processor 1101 is electrically connected to the memory 1102. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not limit the electronic device and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0085] The processor 1101 is the control center of the electronic device 1100. It uses various interfaces and lines to connect the various parts of the entire electronic device 1100. By running or loading software programs and / or modules stored in the memory 1102 and calling data stored in the memory 1102, it executes various functions of the electronic device 1100 and processes data, thereby monitoring the electronic device 1100 as a whole.

[0086] Optionally, the electronic device 1100 further includes: a touch screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107, respectively. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0087] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned UI tree acquisition method when executed by a processor. Specific implementation can be found in the above-mentioned method embodiment, which will not be described in detail here.

[0088] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, terminal device, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0089] Furthermore, although the operations of the method of the present invention are described in a particular order in the accompanying drawings, this does not require or imply that these operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0090] Although the spirit and principles of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features of these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A UI tree acquisition method, characterized in that: include: Obtain the UI component set in the current interface, extract the attribute information of each UI component to generate an initial UI tree, wherein the initial UI tree nodes correspond one-to-one to the UI components, and the attribute information includes at least the component name, region information, and hierarchical relationship; Performing parent-child node area clipping processing on the initial UI tree layer by layer to determine the effective response area of ​​each node in the initial UI tree; A cropped target UI tree is generated based on the effective response area.

2. The method according to claim 1, characterized in that The step of obtaining a set of UI components in the current interface and extracting attribute information of each UI component to generate an initial UI tree includes: Get the root container of the current interface; Traverse all child nodes of the root container and extract UI components mounted on the child nodes of the root container; Extract the attribute information of each UI component, and encapsulate the extracted attribute information of each UI component into a UI tree node; An initial UI tree is generated based on all the UI tree nodes and the hierarchical relationship between the UI tree nodes.

3. The method according to claim 2, characterized in that The extraction of attribute information of each UI component specifically includes: Extract the parent node attribute information of each UI component and the order information of each UI component in its parent node to determine the hierarchical relationship of each UI component; Generate a unique identifier including a hierarchical path for each UI component as the component name of the UI component, wherein the hierarchical path is formed by concatenating the parent node name of each UI component and an additional distinguishing identifier; The position coordinates and size information of each UI component are extracted to determine the area information of each UI component.

4. The method according to claim 1, wherein The performing parent-child node area clipping processing on the initial UI tree layer by layer to determine the effective response area of ​​each node of the initial UI tree includes: Starting from the second level of the initial UI tree, perform a hierarchical traversal process to the next level layer by layer; For each node at the same level, perform the parent-child node area clipping process in sequence according to the preset traversal order; In response to the last node of the last level completing the parent-child node region clipping process, the valid response region of each node of the initial UI tree is obtained based on the updated region information of all nodes.

5. The method according to claim 1, wherein The parent-child node region cropping process includes: For the current node to be processed, the region information of the current node and its parent node is dynamically calculated and updated based on the region information of the current node's parent node, the region information of the current node, and the region information of sibling nodes at the same level.

6. The method according to claim 5, characterized in that The method of dynamically calculating and updating the area information of the current node and its parent node based on the area information of the current node's parent node, the area information of the current node, and the area information of sibling nodes at the same level for the current node to be processed specifically includes: For the current node to be processed, based on the region information of the parent node of the current node and the region information of the current node, the overlapping part of the parent node region and the current node region is calculated; When there is a processed sibling node at the same level, based on the region information of the processed sibling node, the region occupied by the processed sibling node is excluded from the overlapping portion to obtain updated region information of the current node; Otherwise, obtaining updated area information of the current node based on the overlapping portion; Reversely adjust and update the parent node's region information based on the updated region information of the current node.

7. The method according to claim 6, characterized in that The reverse adjustment and updating of the parent node's area information according to the updated area information of the current node includes: Based on the region information of the parent node of the current node and the updated region information of the current node, the region of the parent node of the current node is calculated excluding the remaining part of the region of the current node; If the current node does not have any unprocessed sibling nodes at the same level, regularize the remaining portion to generate a maximum regular geometric shape that can be completely contained by the remaining portion; Based on the area information of the largest regular geometric shape, the area information of the parent node is updated.

8. A UI tree acquisition device, characterized in that: include: A UI tree generation module is used to obtain the UI component set in the current interface, extract the attribute information of each UI component to generate an initial UI tree, wherein the initial UI tree nodes correspond one to one with the UI components, and the attribute information includes at least the component name, region information and hierarchical relationship; A node area clipping module, configured to perform parent-child node area clipping processing on the initial UI tree layer by layer to determine an effective response area for each node in the initial UI tree; The UI tree adjustment module is configured to generate a cropped target UI tree based on the effective response area.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the UI tree acquisition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for obtaining a UI tree according to any one of claims 1 to 7 is implemented.

Images