Methods, devices, equipment, media, and program products for selecting user interface elements.

Abstract

This application discloses a method, apparatus, device, medium, and program product for selecting user interface elements, relating to the field of computer technology. The method includes: in response to a debugging instruction triggered during the execution of a target application, obtaining the frame type of each user interface element and a matching preset adapter; encapsulating each user interface element into an interface node with the same preset node abstract interface using the preset adapter; in response to a selection operation at a target location, determining associated candidate interface nodes; calling the preset node abstract interface to obtain attribute information of each candidate interface node; and determining the target user interface element from the candidate user interface elements corresponding to the candidate interface nodes based on the attribute information. By implementing the technical solution of this application, unified abstraction and intelligent selection of user interface elements running under different user interface frameworks can be achieved, thereby efficiently and accurately locating the target user interface element for the user's intended operation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to methods, apparatus, devices, media, and program products for selecting user interface elements. Background Technology

[0002] In software development, especially in the development and debugging of graphical user interface (GUI) applications, UI (User Interface) debugging is a core task. However, with the increasing complexity of applications, modern application interfaces are often built from a mixture of heterogeneous technology stacks. A single application may simultaneously contain user interface elements implemented based on different frameworks. These elements are dynamically generated at runtime, forming complex hierarchical relationships. Current debugging tools are typically designed independently for specific frameworks, forcing developers to frequently switch between multiple tools when dealing with multi-framework interfaces. Furthermore, these tools struggle to effectively capture the true state of the interface during program runtime, and when elements obscure each other or change dynamically, developers find it difficult to quickly and accurately select the target object from numerous candidate elements for inspection. These limitations make UI debugging inefficient and cumbersome. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, device, medium, and program product for selecting user interface elements to improve the efficiency and accuracy of user interface debugging.

[0004] In a first aspect, this application provides a method for selecting user interface elements, comprising: in response to a debugging instruction triggered during the execution of a target application, obtaining the user interface framework type to which each user interface element in the target application belongs, and a preset adapter matching each user interface framework type; using the preset adapter, encapsulating each user interface element as an interface node, wherein all interface nodes have the same preset node abstract interface; in response to a selection operation for a target location in the user interface, determining candidate interface nodes in the target application associated with the target location; corresponding to each candidate interface node, calling the preset node abstract interface to obtain first attribute information of each candidate interface node; and based on the first attribute information, determining the target user interface element from the candidate user interface elements corresponding to each candidate interface node.

[0005] In some optional implementations, a preset adapter is used to encapsulate each user interface element into an interface node, including: using the preset adapter to call the application programming interface of each user interface element to obtain the second attribute information of each user interface element; converting each second attribute information into the format defined by the preset node abstraction interface to generate each interface node; wherein, the second attribute information includes at least one of identification information, hierarchical relationship information, geometric attribute information, state attribute information, and extended information.

[0006] In some optional implementations, in response to a picking operation for a target location in the user interface, determining candidate interface nodes in the target application associated with the target location includes: in response to a picking operation for a target location in the user interface, constructing a user interface tree of the target application based on the parent-child relationship between the various interface nodes; and querying the user interface tree based on the target location to obtain candidate interface nodes associated with the target location.

[0007] In some optional implementations, the user interface tree is queried based on the target location to obtain candidate interface nodes associated with the target location, including: performing coordinate transformation on the target location to obtain the target coordinates of the target location in a preset standard screen coordinate system; obtaining the target bounding box of each interface node in the preset standard screen coordinate system based on a preset node abstraction interface; and determining the interface node corresponding to the target bounding box as a candidate interface node when the target coordinates are within the target bounding box.

[0008] In some alternative implementations, if any interface node is invisible, the query for the interface node and all its child nodes is skipped.

[0009] In some optional implementations, determining the target user interface element from the candidate user interface elements corresponding to each candidate interface node based on the first attribute information includes: determining the priority of each candidate interface node based on the first attribute information according to a predefined multi-dimensional priority strategy; and determining the target user interface element from each candidate user interface element based on the priority; wherein the first attribute information includes at least one of interactivity information, type information, distance information from the target location, depth information in the user interface tree, rendering order information, and size information.

[0010] In some optional implementations, the target user interface element is determined from the candidate user interface elements based on priority, including: sorting the candidate interface nodes based on priority to obtain a sorting result; and determining the candidate user interface element corresponding to the candidate interface node with the highest priority in the sorting result as the target user interface element.

[0011] In some alternative implementations, in response to consecutive picking operations at the same target location, candidate interface nodes are selected sequentially according to priority, and the candidate user interface elements corresponding to the candidate interface nodes are determined as the target user interface elements.

[0012] Secondly, this application provides a user interface element picking device, comprising: a first acquisition module, configured to acquire, in response to a debugging instruction triggered during the execution of a target application, the user interface frame type to which each user interface element in the target application belongs, and a preset adapter matching each user interface frame type; an encapsulation module, configured to encapsulate each user interface element into an interface node using the preset adapter, wherein all interface nodes have the same preset node abstract interface; a first determination module, configured to determine, in response to a picking operation for a target location in the user interface, a candidate interface node associated with the target location in the target application; a second acquisition module, configured to, corresponding to each candidate interface node, call the preset node abstract interface to acquire first attribute information of each candidate interface node; and a second determination module, configured to determine the target user interface element from the candidate user interface elements corresponding to each candidate interface node based on the first attribute information.

[0013] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the user interface element selection method of the first aspect or any corresponding embodiment described above.

[0014] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform a method for selecting user interface elements according to the first aspect or any corresponding embodiment described above.

[0015] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute a method for selecting user interface elements according to the first aspect or any corresponding embodiment described above.

[0016] The user interface element selection method provided in this application, in response to runtime-triggered debugging instructions, obtains the frame type and its matching preset adapter for each user interface element in the target application, and uses these adapters to uniformly encapsulate interface elements under different frames into interface nodes with the same preset node abstract interface, thereby achieving a generalized abstraction and unified representation of heterogeneous user interface frames. Based on this, in response to the user's selection operation on a target location in the interface, candidate interface nodes associated with that location are determined, and a unified node abstract interface is called to obtain the first attribute information of each candidate node. Based on this attribute information, the target user interface element is accurately selected from the candidate elements. This application effectively solves the problem of fragmented debugging tools caused by the diversity of frameworks in related technologies, eliminating the need for developers to build separate debugging tools for each framework or learn how to use multiple tools. Furthermore, since the selection is performed based on the actual interface state during program runtime, it can accurately capture dynamically generated elements and runtime hierarchical relationships, overcoming the limitation of static inspection failing to reflect runtime states. Furthermore, the target selection method based on attribute information enables efficient and accurate location of the interface elements that the user intends to operate in complex scenarios such as multiple overlapping elements, significantly improving the convenience and efficiency of UI debugging. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a first method for selecting user interface elements according to an embodiment of this application; Figure 2 This is a schematic diagram of a second method for selecting user interface elements according to an embodiment of this application; Figure 3 This is a schematic diagram of a third method for selecting user interface elements according to an embodiment of this application; Figure 4 This is a structural block diagram of a user interface element picking device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that the information (including but not limited to user input information, such as information entered by the user into input boxes), data (including but not limited to data used for analysis, stored data, and displayed data, such as context code, all code of the current project, the service pressure corresponding to operations performed on all code of the current project, and the code development status of the current project), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, the context code, operations performed on all code of the current project, the corresponding service pressure, and the code development status involved in this application were all obtained with full authorization.

[0021] In graphical user interface (GUI) application development, UI debugging is a crucial task that directly impacts development efficiency and user experience. As modern applications become increasingly complex, a single interface often contains dozens or even hundreds of user interface elements (UI elements). These elements have parent-child relationships, sibling relationships, and complex visual occlusion relationships, posing a significant challenge to developers' debugging efforts.

[0022] Traditional UI debugging methods suffer from several problems: First, static analysis capabilities are limited. The interface preview function provided by development tools cannot reflect the true state of the program at runtime. For example, key information such as dynamically created controls, attribute changes triggered by user interaction, and hierarchical relationships adjusted at runtime cannot be obtained statically. Second, framework differences lead to fragmented tools. Different UI frameworks, such as Qt, WPF, Android, iOS UIKit, and Web DOM, have completely different internal implementation mechanisms and APIs (Application Programming Interfaces). Related debugging tools are usually developed for a single framework. When a project involves multiple technology stacks, developers have to frequently switch between multiple tools, which seriously affects work efficiency. Third, click location is difficult. When developers want to inspect a visible element on the interface, it is often difficult to quickly locate the corresponding object instance at the code level, especially in scenarios where multiple elements overlap. Visual observation alone cannot accurately determine which element is at the top level or which element will respond to user clicks.

[0023] Several solutions have been developed to address the above problems, but each has its own significant drawbacks: The first category consists of dedicated debugging tools that come with each framework. Although these tools are deeply optimized for specific frameworks, they cannot be used across frameworks and depend on the development environment. They are not available in the production environment, and self-developed frameworks do not have tool support.

[0024] The second category is a general object traversal scheme based on reflection. These tools utilize the reflection capabilities provided by programming languages ​​to traverse the object tree at runtime and obtain the object's properties and methods. Although theoretically universal, they have high performance overhead, and traversing large-scale UI trees can cause significant lag. At the same time, the information obtained by reflection lacks semantic abstraction, making it difficult to understand the differences in concepts such as position and hierarchy in different frameworks, and thus difficult to achieve intelligent selection.

[0025] The third type involves injecting debugging code using Hook or AOP (Aspect-Oriented Programming) techniques. This method inserts monitoring code into key methods of the UI framework to collect events such as element creation, destruction, and attribute changes. While it allows for deep monitoring, it is highly invasive, prone to stability risks, and has poor compatibility, requiring re-adaptation after framework upgrades.

[0026] In summary, the core problems with the relevant technologies are: the lack of a universal UI tree abstraction method, making it impossible to represent the UI hierarchy structure of different frameworks in a unified way; the lack of a universal click-picking algorithm, making it unable to intelligently determine the user's desired selection when multiple UI elements overlap; and the lack of a standard pattern for framework adaptation, lacking a reusable methodology for quickly adapting to new UI frameworks. Therefore, a cross-framework, high-performance, and intelligent UI element picking method is needed to address the many shortcomings of the relevant technologies.

[0027] In view of this, the user interface element selection method provided in this application responds to runtime debugging instructions and obtains in real time the framework type of each UI element in the target application and its corresponding preset adapter. This directly breaks through the limitations of traditional static inspection and can capture the real state of dynamically created controls, runtime attribute changes, and real-time hierarchical relationships. Then, using these preset adapters, native UI elements that originally belonged to different technology stacks such as Qt, WPF, and Android and had heterogeneous internal representations are uniformly encapsulated into interface nodes with the same preset node abstract interface. This process fundamentally solves the tool fragmentation problem caused by framework differences. By standardizing abstraction, it shields the complexity of the underlying framework, allowing all subsequent operations to be performed based on a unified node model without concern for the specific implementation. It also avoids the performance overhead and semantic loss caused by reflection mechanisms, as well as the intrusive risks of Hook technology. Based on this, for the selection operation of a target position in the user interface, it can accurately locate all candidate interface nodes associated with that position, laying the foundation for handling complex visual occlusion relationships. By calling a unified preset node abstraction interface, the system efficiently obtains the first attribute information (such as position, level, size, etc.) of each candidate node. Based on this standardized attribute information, it performs intelligent comparison and judgment, thereby accurately selecting the target user interface element that the user truly intends to select from numerous overlapping candidate elements. This complete process not only achieves high-performance, cross-framework general UI tree abstraction, but also provides a standardized adaptation pattern, making it possible to quickly adapt to new UI frameworks. Furthermore, through an attribute-based selection algorithm, it intelligently solves the click positioning problem in scenarios with overlapping elements.

[0028] According to an embodiment of this application, a method for selecting user interface elements is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a method for selecting user interface elements, which can be used in electronic devices such as desktop computers and laptops. Figure 1 This is a flowchart of a method for selecting user interface elements according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: In response to a debugging instruction triggered during the execution of the target application, obtain the user interface frame type to which each user interface element in the target application belongs, and the preset adapter that matches each user interface frame type.

[0030] The target application refers to the currently running application that requires UI debugging. Debugging commands refer to user-triggered commands that enter debug mode. For example, this could be activated via a specific key combination (such as Ctrl + left mouse button). User interface elements (UI elements) refer to independently identifiable visual components in the application interface, such as buttons, images, text boxes, scroll views, and containers. They are the basic units that constitute the user interface. The user interface framework type refers to the technology stack or framework used to build the user interface. Examples include Cocos2d-x, Unity, and Android View. UI elements in different frameworks differ in implementation, coordinate system, and hierarchy management. Pre-defined adapters are adapter classes pre-implemented for each user interface framework (UI framework), responsible for encapsulating the framework-specific UI elements into a unified interface. They implement the mapping from framework-specific APIs to general node interfaces, resolving framework differences. Specifically, when a user initiates debug mode via a specific debugging command (such as pressing a shortcut key or key combination), the first step is to identify which different types of user interface frameworks are used in the currently running target application. This process can be accomplished by detecting the underlying libraries the application depends on, runtime environment characteristics, or interface information exposed by the framework itself. After identifying the framework type, the system finds and loads the pre-configured adapters corresponding to each framework type based on the pre-configured mapping relationships. These adapters are pre-developed for different frameworks and can convert framework-specific user interface elements into components of a unified format, thus laying the foundation for subsequent unified processing.

[0031] Step S102: Using a preset adapter, each user interface element is encapsulated as a UI node, and all UI nodes have the same preset node abstract interface.

[0032] A UI node refers to a unified object encapsulated through a predefined adapter. It shields the differences between underlying frameworks, providing consistent properties and methods, such as retrieving name, position, size, and visibility. The predefined node abstract interface (e.g., IUINode) refers to a unified interface definition containing core methods needed for debugging, such as getName(), getPosition(), getSize(), isVisible(), and getWorldBounds(). All UI nodes must implement this interface. Specifically, the process of encapsulating various user interface elements into UI nodes using predefined adapters essentially involves creating a corresponding wrapper object for each original UI element. This wrapper object holds a reference to the original element and exposes its various properties and behaviors by implementing a standardized set of methods. In this way, regardless of the original element's framework, after encapsulation through the adapter, they all present a completely consistent interface, allowing upper-level tools to manipulate these UI nodes like ordinary objects, without needing to consider the differences in the underlying framework. This unified encapsulation greatly simplifies the complexity of cross-framework user interface debugging.

[0033] Step S103: In response to the picking operation for the target location in the user interface, determine the candidate interface nodes in the target application that are associated with the target location.

[0034] The user interface (UI) refers to the visual interface of an application, serving as the window through which users interact with the program and containing various UI elements. The target location refers to the specific coordinates of the user's click on the screen (e.g., the mouse click point). A picking operation refers to the user's action in debug mode, where they click on a location on the screen to select a specific UI element. Candidate UI nodes refer to all UI nodes within the target location area that have been initially detected; these nodes may contain multiple overlapping elements. Specifically, when a user performs a picking operation at a specific location on the application interface, it's necessary to quickly determine which UI nodes are associated with that location. This process typically involves traversing or searching all currently encapsulated UI nodes, determining whether the location falls within the geometric extent of each node (e.g., the screen area it occupies). To improve efficiency, spatial partitioning and caching strategies can be employed, but the core logic is to filter out all candidate UI nodes that can match the target location for further analysis.

[0035] Step S104: For each candidate interface node, call the preset node abstraction interface to obtain the first attribute information of each candidate interface node.

[0036] First attribute information refers to various attributes used to determine user intent, such as interactivity, element type (button, text, etc.), and distance from the click point. Specifically, for each selected candidate interface node, the corresponding method is called through their unified abstract interface to obtain the first attribute information used for decision-making. The first attribute information includes the basic characteristics of the element represented by the node, such as what type of control it is, whether it is currently operable, and its relative hierarchy in the interface. Because all nodes follow the same interface, electronic devices can consistently obtain this information without needing to know the specific implementation framework behind each node, thus ensuring the accuracy and efficiency of information acquisition.

[0037] Step S105: Based on the first attribute information, determine the target user interface element from the candidate user interface elements corresponding to each candidate interface node.

[0038] The target user interface element refers to the UI element ultimately selected after analyzing the primary attribute information; it represents the optimal match to the user's intent. Specifically, based on the acquired primary attribute information, the target user interface element needs to be determined from multiple candidate interface nodes. This decision-making process comprehensively considers the attribute information of each candidate interface node, such as the importance of the node type, the node's visual hierarchy within the interface, and the node's size, comparing and sorting them according to preset rules. Finally, the node that best matches the user's intent is selected as the target, and its corresponding original user interface element is highlighted or presented to the user in another way, completing a precise selection operation.

[0039] The user interface element selection method provided in this application, in response to runtime-triggered debugging instructions, obtains the frame type and its matching preset adapter for each user interface element in the target application, and uses these adapters to uniformly encapsulate interface elements under different frames into interface nodes with the same preset node abstract interface, thereby achieving a generalized abstraction and unified representation of heterogeneous user interface frames. Based on this, in response to the user's selection operation on a target location in the interface, candidate interface nodes associated with that location are determined, and a unified node abstract interface is called to obtain the first attribute information of each candidate node. Based on this attribute information, the target user interface element is accurately selected from the candidate elements. This application effectively solves the problem of fragmented debugging tools caused by the diversity of frameworks in related technologies, eliminating the need for developers to build separate debugging tools for each framework or learn how to use multiple tools. Furthermore, since the selection is performed based on the actual interface state during program runtime, it can accurately capture dynamically generated elements and runtime hierarchical relationships, overcoming the limitation of static inspection failing to reflect runtime states. Furthermore, the target selection method based on attribute information enables efficient and accurate location of the interface elements that the user intends to operate in complex scenarios such as multiple overlapping elements, significantly improving the convenience and efficiency of UI debugging.

[0040] This embodiment provides a method for selecting user interface elements, which can be used in electronic devices such as desktop computers and laptops. Figure 2 This is a flowchart of a method for selecting user interface elements according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: In response to a debugging instruction triggered during the execution of the target application, obtain the user interface frame type to which each user interface element in the target application belongs, and the preset adapter matching each user interface frame type. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0041] Step S202: Using a preset adapter, each user interface element is encapsulated as a UI node, and all UI nodes have the same preset node abstract interface.

[0042] Specifically, step S202 includes: Step S2021: Using the preset adapter, call the application programming interface of each user interface element to obtain the second attribute information of each user interface element.

[0043] The second attribute information includes at least one of the following: identification information, hierarchical relationship information, geometric attribute information, state attribute information, and extended information.

[0044] Application Programming Interfaces (APIs) refer to the collection of functions or methods provided by each user interface framework itself for accessing and manipulating its internal elements. Preset adapters obtain various attribute information of the original user interface elements through these interfaces, forming the basis for cross-framework adaptation. Secondary attribute information refers to the raw, unformatted data directly obtained by the preset adapter from the original user interface elements. This data contains various characteristics of the elements and is the raw material for building unified interface nodes.

[0045] Identification information refers to basic information used to uniquely distinguish different user interface elements, such as the element's name, type, or unique number, facilitating the identification and referencing of specific elements during debugging. Hierarchical relationship information describes the organizational relationship of user interface elements within the overall structure, such as the containment and inclusion relationships between elements. Geometric attribute information describes the spatial characteristics of user interface elements on the screen, such as position, size, and possible transformations (e.g., rotation, scaling), used to determine the actual display area of ​​the elements. State attribute information reflects the current dynamic characteristics of user interface elements, such as whether they are visible, interactive, or enabled, helping to understand the real-time behavior of the elements. Extended information refers to additional data specific to a particular framework, beyond the aforementioned general attributes, such as custom attributes, style class names, or data binding information, used to provide more comprehensive debugging support.

[0046] Specifically, the process of using a preset adapter to call the application programming interfaces (APIs) of various user interface elements to obtain second attribute information is essentially an interaction between the preset adapter as an intermediary and the underlying framework. When the preset adapter is activated, it actively calls the standardized access interfaces provided by the adapted framework to query the internal state of the original user interface elements through these interfaces, based on the characteristics of the adapted framework. The preset adapter requests various raw data of the elements sequentially according to a predefined strategy, such as the element's basic identifier, its position in the hierarchical structure, its geometric description on the screen, its current dynamic state, and additional information specific to the framework. During this process, the preset adapter needs to handle possible exceptions, such as the element being destroyed or the API call failing, to ensure the stability of the acquisition process. All methods of the adapter employ exception handling mechanisms. When the original element is destroyed or attribute access fails, it returns reasonable default values ​​(such as destroyed, zero rectangle, etc.), thus ensuring that the debugger (i.e., the debugging tool or debugging function module running on the electronic device) does not crash due to individual element anomalies. Finally, the preset adapter gathers these diverse raw data collected from different framework interfaces to form a set of unformatted second attribute information about the element.

[0047] In cases where some UI frameworks may not support certain predefined interface methods (such as getTransform and getRenderOrder), the adapter can employ reasonable default values ​​or a fallback strategy. For example, it can try to estimate transformation information through other means, or return the identity matrix as the default transformation; the rendering order can be inferred from the creation order of elements or their parent-child relationships. In practical applications, most UI frameworks at least provide position and hierarchy information (such as Cocos' getPosition and getLocalZOrder), which is sufficient to meet basic debugging needs.

[0048] Step S2022: Convert each second attribute information into the format defined by the preset node abstract interface to generate each interface node.

[0049] The core step in standardizing and encapsulating raw data is converting various secondary attribute information into the format defined by the preset node abstract interface and generating interface nodes. Specifically, after obtaining the secondary attribute information, the preset adapter maps and transforms this information one by one according to the unified data structure and format requirements specified by the preset node abstract interface. For example, it unifies the way concepts such as position, size, and visibility are expressed in different frameworks into the general expression form defined by the abstract interface. This transformation process involves data type conversion, unit normalization, coordinate system adjustment, and other operations to ensure that all elements from different frameworks ultimately present a completely consistent data appearance. After this processing, each raw user interface element is encapsulated into an interface node that implements a unified interface. These nodes provide the same methods and properties, allowing upper-level debugging tools to traverse, query, and manipulate them in a unified way, without needing to know which framework they come from.

[0050] In some optional implementations, the default node abstract interface includes at least the following methods: getName() for getting the node name, getType() for getting the node type, getParent() for getting the parent node, getChildren() for getting the list of child nodes, getPosition() for getting the node's local position, getSize() for getting the node's size, getWorldBounds() for getting the node's bounding box in world coordinates, getTransform() for getting the node's transformation matrix, isVisible() for determining if a node is visible, isInteractive() for determining if a node is interactive, getOpacity() for getting the node's opacity, getRenderOrder() for getting the node's rendering order, and getProperties() for getting framework-specific extended information. These methods follow the principle of minimization, including only the functionality necessary for UI debugging, while maintaining semantic clarity so that developers of different frameworks can understand them.

[0051] In some alternative implementations, a caching mechanism can be introduced into the adapter to improve performance. For example, for the computationally expensive `getWorldBounds()` method, the calculation result can be cached internally within the adapter, and a version number mechanism can be used to determine the validity of the cache. When the UI tree structure changes or its properties change, the global version number is incremented, the cache is automatically invalidated, and the calculation is recalculated on the next call. This avoids repeatedly calculating the bounding box every time a selection is clicked, significantly improving efficiency.

[0052] The user interface element picking method provided in this application embodiment obtains multi-dimensional second attribute information such as identification information, hierarchical relationship information, geometric attribute information, state attribute information, and extended information by calling the application programming interface of each user interface element. This heterogeneous raw data is then uniformly converted into a standardized format defined by a preset node abstraction interface, thereby generating interface nodes with a consistent data structure. This not only ensures that the encapsulated interface nodes accurately and completely reflect the runtime state of the original UI elements, but also provides a rich and standardized data foundation for subsequent picking operations by normalizing the diverse attribute information of different frameworks into a unified format. This allows for comprehensive consideration of the element's identity, hierarchy, position, interactivity, and framework-specific details when identifying targets based on this attribute information, significantly improving the accuracy and adaptability of the picking algorithm. Furthermore, this process relies entirely on the adapter's call to the framework API, achieving non-intrusive data acquisition from the original code, ensuring both the real-time performance and reliability of the data, while maintaining the stability and scalability of the system.

[0053] Step S203: In response to a picking operation targeting a target location in the user interface, determine candidate interface nodes in the target application associated with the target location. See details below. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0054] Step S204: For each candidate interface node, call the preset node abstraction interface to obtain the first attribute information of each candidate interface node. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0055] Step S205: Based on the first attribute information, determine the target user interface element from the candidate user interface elements corresponding to each candidate interface node. For details, please refer to [link to relevant documentation]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0056] This embodiment provides a method for selecting user interface elements, which can be used in electronic devices such as desktop computers and laptops. Figure 3 This is a flowchart of a method for selecting user interface elements according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: In response to a debugging instruction triggered during the execution of the target application, obtain the user interface frame type to which each user interface element in the target application belongs, and the preset adapter matching each user interface frame type. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0057] Step S302: Using a preset adapter, each user interface element is encapsulated as a UI node, and all UI nodes have the same preset node abstract interface. For details, please refer to [link to relevant documentation]. Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0058] Step S303: In response to a picking operation for a target location in the user interface, determine candidate interface nodes in the target application that are associated with the target location.

[0059] Specifically, step S303 includes: Step S3031: In response to the picking operation for the target location in the user interface, construct the user interface tree of the target application based on the parent-child relationship between the various interface nodes.

[0060] Parent-child relationships refer to the hierarchical containment relationships between user interface elements. An element (parent node) can contain multiple other elements (child nodes), forming a tree structure. For example, if a panel contains multiple buttons, then the panel is the parent node, and the buttons are the child nodes. This relationship determines the nesting and transformation propagation of elements.

[0061] A user interface tree (UI tree) is a tree-like structure that organizes all user interface elements, reflecting their hierarchical relationships. The root node typically represents the entire window or canvas, while the leaf nodes are specific controls. This tree allows you to traverse all elements and understand its organizational structure. The UI tree is constructed using a lazy loading mechanism, dynamically loading child nodes when a node is selected, reducing the load on the interface display.

[0062] Specifically, when a user performs a selection operation, the first step is to organize these nodes into a complete user interface tree based on the hierarchical relationships between the pre-defined interface nodes. This construction process typically starts from the root node and recursively traverses from top to bottom using the methods provided by each node to retrieve its child nodes, thereby reconstructing the hierarchical structure of the entire application interface. Through this tree, the position of each element in the interface and its subordinate relationships with other elements can be clearly understood, laying the foundation for subsequent click location lookups.

[0063] Step S3032: Based on the target location, query the user interface tree to obtain candidate interface nodes associated with the target location.

[0064] After constructing the user interface tree, it's necessary to search for all potentially selectable nodes within the tree based on the user's clicked target location. Specifically, this process can employ depth-first or breadth-first traversal, checking each node layer by layer from the root node to determine if it's related to the clicked location. To improve efficiency, visibility checks and spatial range checks can be combined during the traversal, ensuring that only nodes that might contain the clicked point are further examined. Finally, all nodes meeting the initial criteria are collected, forming a candidate interface node list.

[0065] The user interface element selection method provided in this application constructs a complete user interface tree based on the inherent parent-child relationships between various interface nodes. This organizes the originally scattered interface nodes into a hierarchical tree model. Then, the user interface tree is traversed and retrieved based on the target location, ultimately selecting candidate interface nodes associated with the target location. By explicitly constructing the user interface tree, not only are scattered interface nodes incorporated into a unified hierarchical structure, providing a clear traversal path and parent-child context for subsequent selection operations, but the method of querying the tree structure based on the target location can quickly locate the set of nodes that may hit the target location. This avoids the performance overhead caused by blindly traversing all nodes, thus significantly improving the selection efficiency of candidate nodes while ensuring selection accuracy. Furthermore, since the tree structure is entirely based on the original parent-child relationships between nodes, there is no need to maintain complex spatial indexes or relationship mappings, maintaining the simplicity of implementation while ensuring the accuracy and real-time nature of hierarchical information.

[0066] In some optional implementations, step S3032 above includes: Step a1: Perform coordinate transformation on the target position to obtain the target coordinates in the preset standard screen coordinate system.

[0067] A preset standard screen coordinate system refers to a unified coordinate system defined to address the differences in coordinate systems among different UI frames. For example, the origin can be set to the bottom left corner of the screen, the X-axis to the right, and the Y-axis to the top, with units in physical pixels. All element coordinates within all frames need to be transformed to this coordinate system for consistent click detection and calculation. Specifically, since different UI frames may use different coordinate system definitions, such as differences in the origin position or axial direction, it is necessary to uniformly transform the original screen position clicked by the user into a preset standard screen coordinate system. This transformation process takes into account the characteristics of the original coordinate system, mapping it to a unified standard coordinate system through mathematical transformations, ensuring that all subsequent calculations are performed on the same spatial reference, thereby eliminating errors caused by coordinate system differences.

[0068] Different UI frameworks use different coordinate transformation methods. For example, for OpenGL / game engines (origin, bottom left corner), only resolution scaling needs to be considered; for Android / iOS / Web (origin, top left corner), Y-axis flipping is required; for Web, page scrolling and device pixel ratio also need to be considered. The adapter needs to implement the toScreenCoordinates() method for the specific framework to uniformly convert these coordinates from different sources into standard screen coordinates.

[0069] Step a2: Based on the preset node abstraction interface, obtain the target bounding box of each interface node in the preset standard screen coordinate system.

[0070] The target bounding box refers to the smallest rectangular area occupied by an interface node in the standard screen coordinate system, completely encompassing the visible range of the element. The bounding box allows for quick determination of whether a clicked point falls within an element's area, forming the basis of click picking. Specifically, obtaining the target bounding box of each interface node in the standard screen coordinate system relies on the geometric information provided by each node through a preset node abstraction interface. By calling methods defined in the preset node abstraction interface, the actual display range of each interface node in world space is obtained. This range is typically represented by the smallest rectangular area, i.e., the bounding box. Since all nodes follow the same interface, their accurate positions and sizes in the standard coordinate system can be consistently obtained, providing a reliable basis for click hit detection.

[0071] In some alternative implementations, to accurately calculate the bounding box of a transformed (e.g., rotated, scaled, skewed) UI element, the adapter needs to recursively apply the transformation matrices of all parent nodes when implementing `getWorldBounds()`. The specific steps include: first, obtaining the element's bounding box in the local coordinate system (considering anchor points); then, calculating the element's own transformation matrix (usually constructed in the order of scaling, rotation, and translation); then, recursively multiplying it upwards and left-wise by the transformation matrices of all parent nodes to obtain the world transformation matrix; applying this matrix to the four vertices of the local bounding box and projecting it onto the screen coordinate system; finally, calculating the smallest rectangle containing all vertices as the world bounding box. This process ensures accurate click detection even when the element undergoes complex transformations.

[0072] In the above implementation, by recursively applying the transformation matrices of all parent nodes and strictly following the complete process of obtaining the local bounding box, constructing its own transformation matrix, vertex transformation, screen projection, and calculating the minimum bounding box, the actual bounding box of the UI element in the world coordinate system after any complex transformation such as rotation, scaling, and tilting can be accurately calculated. This fundamentally solves the problem of click position offset or hit detection failure caused by transformation, ensuring that even in scenarios with multi-layer nested transformations or non-linear transformations, click picking can still accurately hit the real display area of ​​the target element. At the same time, by encapsulating the complex transformation processing in the adapter's getWorldBounds() method, the upper-level picking algorithm does not need to be aware of the underlying transformation details, thus maintaining the simplicity and versatility of the system while ensuring high accuracy, and significantly improving the reliability and practicality of the debugging tool in complex UI scenarios.

[0073] Step a3: If the target coordinates are within the target bounding box, determine the interface node corresponding to the target bounding box as a candidate interface node.

[0074] Once the target coordinates of the clicked location in the preset standard screen coordinate system are obtained, they are compared with the bounding box of each candidate node. If the target coordinates are inside the bounding box of a node, it means that the node is covered by the clicked location, and therefore the node is determined as a candidate interface node.

[0075] In the above implementation, the target location clicked by the user undergoes coordinate transformation to unify it into a preset standard screen coordinate system to obtain the target coordinates. Then, the bounding boxes of each interface node in the same standard coordinate system are obtained through a preset node abstraction interface. Finally, candidate interface nodes are filtered by determining whether the target coordinates are within the target bounding box. By introducing a unified coordinate transformation mechanism, the comparison obstacles caused by inconsistencies in the origin, direction, or units of coordinate systems between different UI frameworks are eliminated. This ensures that all interface nodes and click locations are spatially determined under the same metric. At the same time, hit detection is performed based on the containment relationship of the bounding box, which not only ensures the spatial accuracy of the picking operation but also makes the judgment logic simple and efficient. Furthermore, since both coordinate transformation and bounding box acquisition rely on the preset node abstraction interface, the entire process is completely decoupled from the specific implementation of the underlying framework.

[0076] In some optional implementations, step S303 above further includes: if any interface node is invisible, then skip the query of the interface node and all its child nodes.

[0077] An invisible state refers to a user interface element that is currently not visible on the screen, possibly due to being hidden, having zero opacity, being outside the screen area, or being clipped by its parent container. Skipping invisible nodes and their subtrees when traversing the UI tree can significantly improve performance and avoid unnecessary calculations.

[0078] All child nodes of a UI node refer to all lower-level nodes that the current UI node directly or indirectly contains in the user interface tree. If the current node is not visible, all its child nodes are usually also not visible, so they can be skipped during traversal.

[0079] Specifically, when traversing the user interface tree to find candidate nodes, an early termination optimization is used. That is, if a certain interface node is currently invisible, there is no need to continue checking that node and all its child nodes. This is because invisible nodes are not displayed on the screen, and their child nodes are usually also invisible. Therefore, skipping them directly can significantly reduce unnecessary calculations and improve query efficiency.

[0080] In the above implementation, when an invisible interface node is detected during traversal, the query of that node and all its child nodes is skipped directly, and no further position judgment or bounding box detection is performed on it. By using the visibility attribute of interface nodes as a pruning condition, invisible nodes and their entire subtrees are removed from the candidate range in the early stage of traversal. This avoids the performance waste caused by meaningless coordinate transformations and bounding box calculations for invisible elements, and significantly reduces the number of nodes that need to be traversed. This significantly improves the query efficiency of the user interface tree, especially when there are a large number of occluded or hidden UI elements. This early termination mechanism can effectively reduce the response latency of the picking operation.

[0081] In some alternative implementations, for very large UI trees (e.g., more than 1000 nodes), a spatial index structure (such as a quadtree) can be used to accelerate the search for candidate click points. However, considering that the number of nodes in a typical UI interface is usually in the tens to hundreds, and the frequency of click debugging operations is not high, in actual implementation, spatial indexing can be enabled or disabled as needed.

[0082] Step S304: For each candidate interface node, call the preset node abstract interface to obtain the first attribute information of each candidate interface node. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0083] Step S305: Based on the first attribute information, determine the target user interface element from the candidate user interface elements corresponding to each candidate interface node.

[0084] Specifically, step S305 includes: Step S3051: Determine the priority of each candidate interface node based on the first attribute information according to the predefined multi-dimensional priority strategy.

[0085] The first attribute information includes at least one of the following: interactivity information, type information, distance information from the target location, depth information in the user interface tree, rendering order information, and size information.

[0086] Multi-dimensional priority strategy is a method that calculates the priority of each candidate element by combining multiple factors (such as interactivity, type, distance, depth, rendering order, and size). Priority indicates the degree to which a candidate interface node is selected.

[0087] Interactivity refers to whether user interface elements are interactive, that is, whether they can respond to user input (such as clicks, touches, etc.). For example, interactive elements (such as buttons) usually have higher priority, which aligns with the user's intention to select operable controls during debugging.

[0088] Type information refers to the category of user interface elements, such as buttons, text boxes, images, containers, etc. Different types of elements have different weights in priority scoring; for example, buttons are more likely to be selected than ordinary containers.

[0089] The distance information from the target location refers to the distance between the click point (target location) and the center of the bounding box of the candidate element, which can be calculated using Manhattan distance (|dx| + |dy|). For example, the closer the distance, the higher the priority, reflecting the accuracy of the user's click intent.

[0090] Depth information in the user interface tree refers to the hierarchical depth of the candidate node within the UI tree (the root node has a depth of 0). For example, child elements typically have higher priority than parent elements because users often want to select a specific control rather than its container.

[0091] Rendering order information refers to the order in which elements are drawn on the screen; typically, elements rendered later are displayed on top of each other. For example, elements rendered later have higher priority, which aligns with visual intuition.

[0092] Size information refers to the area of ​​the element's bounding box. Text elements are not included in the area calculation to avoid misjudgment. For example, smaller elements have higher priority to avoid accidentally selecting large background areas and to align with users' usual habit of selecting specific small controls.

[0093] Specifically, when determining the priority of each candidate interface node, a predefined multi-dimensional priority strategy is used to comprehensively compare the primary attribute information provided by each candidate node. This strategy considers multiple key factors, such as whether the element represented by the node is interactive, the specific type of the element, the relative relationship between the click position and the node, the node's hierarchical depth in the user interface tree, the rendering order of the nodes, and the size of the nodes. By analyzing these factors, the rationality of selecting each candidate node in a specific scenario can be evaluated, thereby determining the priority of each node. This process does not simply rely on a single indicator, but rather uses a multi-dimensional comprehensive consideration to ensure that the priority determination truly reflects the user's operational intent.

[0094] Step S3052: Based on priority, determine the target user interface element from each candidate user interface element.

[0095] Based on the established priorities, the final target element is selected from among the candidate user interface elements. This selection process follows the principle of highest priority, that is, comparing the priority levels of all candidate nodes and choosing the one that best matches the user's expectations. This optimization mechanism ensures that among multiple overlapping elements, the element that the user is most likely to want to select can be accurately identified, providing a precise target for subsequent debugging operations.

[0096] The user interface element selection method provided in this application adopts a predefined multi-dimensional priority strategy. It comprehensively considers at least one of the following: interactivity information, type information, distance information from the target location, depth information in the user interface tree, rendering order information, and size information of each candidate interface node. A quantified priority is calculated for each candidate node. Then, based on the result of sorting or comparing these priorities, the target element that best matches the user's intent is accurately identified from multiple overlapping candidate user interface elements. This application introduces a multi-dimensional comprehensive scoring mechanism, abandoning the limitations of traditional single-dimensional judgments (such as relying solely on rendering order or distance). This allows the selection algorithm to fully simulate multiple considerations when identifying targets in complex interfaces, such as prioritizing interactive controls, favoring clickable areas, prioritizing child elements over parent elements, respecting visual hierarchical relationships, and avoiding large-area background interference. This significantly improves the accuracy of target identification and user satisfaction in complex scenarios with overlapping elements such as buttons, icons, text, and containers. Furthermore, since the priority calculation relies entirely on the first attribute information obtained through a unified interface, the entire process is independent of the specific implementation of the underlying framework.

[0097] In some optional implementations, step S3052 above includes: Step b1: Sort each candidate interface node according to priority to obtain the sorting result.

[0098] The sorting result refers to the list of candidate interface nodes arranged from highest to lowest (or lowest to highest) priority score. Specifically, to identify the highest priority node among the candidates, all candidate nodes are sorted according to their priority. The sorting is based on the results of multi-dimensional comparisons; for example, nodes with higher priority can be ranked first, and nodes with lower priority can be arranged sequentially. In this way, an ordered list of candidate nodes is obtained, which clearly reflects the priority order of each node in this click operation.

[0099] Step b2: The candidate user interface element corresponding to the candidate interface node with the highest priority in the sorting results is determined as the target user interface element.

[0100] After obtaining the sorting results, the candidate node with the highest priority in the list is directly selected, and its corresponding original user interface element is determined as the target user interface element. This node is considered to be the selection that best matches the user's intent under the current click action.

[0101] In the above implementation, candidate interface nodes are sorted according to priority to obtain a clear sorting result. Then, the user interface element corresponding to the candidate interface node with the highest priority in the sorting result is directly determined as the target element. This application introduces a sorting and selection mechanism to transform the priority of each candidate interface node into a clear and comparable linear order, and makes a final decision based on the principle of selecting the best. This ensures the certainty and predictability of the selection result, and makes the entire target determination process efficient. It does not require the introduction of additional loop switching or manual intervention mechanisms, thus enabling the element most likely to be desired by the user to be hit in one step in most scenarios.

[0102] In some optional implementations, step S3052 further includes: in response to continuous picking operations on the same target location, sequentially selecting candidate interface nodes according to priority, and determining the candidate user interface element corresponding to the candidate interface node as the target user interface element.

[0103] Continuous picking refers to a user clicking multiple times on the same screen location (such as repeatedly pressing Ctrl + left mouse button). Each click triggers the picking process, cycling through the candidate elements.

[0104] Sequential cyclic selection refers to selecting different candidate nodes sequentially according to priority when continuously selecting the same area. For example, the highest priority node is selected the first time, the next highest priority node is selected the second time, and so on, repeating in a loop, allowing users to quickly browse all overlapping elements.

[0105] Specifically, when a user performs multiple consecutive picking operations at the same target location, the electronic device recognizes this as repeated clicks on the same area. In this case, selecting the same element each time might not satisfy the user's need to view other overlapping elements. Therefore, it iterates through the candidate nodes in a cyclical manner according to the previously sorted priority order. That is, the first click selects the node with the highest priority, the second click selects the node with the next highest priority, and so on, until all candidate nodes have been traversed before starting again. This cyclical switching mechanism allows users to quickly browse and inspect all interface elements covered by clicks through simple, continuous clicks, greatly improving the flexibility and efficiency of debugging.

[0106] It should be noted that during the loop switching process, non-interactive elements are not skipped, as debugging may require checking the attributes of these elements (such as image path settings). Loop switching can be triggered by repeatedly clicking the same location, or by using modifier keys (such as Ctrl + left mouse button) for manual switching. This example uses repeated clicking as an example. Determining whether elements are in the same area can be done by comparing the difference between the current click coordinates and the previous click coordinates to see if it is less than a preset threshold, or directly based on whether the candidate element sets are the same.

[0107] In the above embodiments, when a user performs continuous picking operations on the same target location, each candidate interface node is selected sequentially according to priority, and the user interface element corresponding to the currently selected candidate node is determined as the target element. This application solves the correction requirement when the automatically selected optimal element is not what the user intended by introducing a loop switching mechanism. This allows the user to quickly traverse all overlapping candidate elements by continuously clicking the same location without needing to reposition or switch tools. This ensures the efficiency of automated picking while giving the user the final choice and a flexible operating experience. At the same time, since the loop order is strictly based on the existing priority sorting results, the entire process is completely consistent with the priority calculation logic, achieving both natural continuity of operation and avoiding the introduction of additional sorting rules or state maintenance mechanisms. This significantly improves the human-computer interaction friendliness and scenario adaptability of the debugging tool.

[0108] In some optional implementations, priority can be quantified using a priority score. Specifically, for each candidate interface node, specific weight coefficients are assigned to various factors in its first attribute information (such as interactivity, type, distance, depth, rendering order, size, etc.), and a priority score is calculated through a weighted summation. The weighting of each factor is designed such that a lower score indicates a higher priority for the node, meaning it better aligns with the user's intentions. For example, interactive elements typically receive a larger negative weight, significantly lowering their score and thus increasing their priority; elements closer to the click point, smaller in area, deeper in the tree, and later in the rendering order also receive lower scores due to their weighting, thus being assigned higher priority. This multi-dimensional quantitative evaluation measures the rationality of selecting each candidate node. When determining the target user interface element, the priority scores of all candidate nodes are compared, and the original user interface element corresponding to the node with the lowest score (i.e., the highest priority) is directly selected as the target. To facilitate sorting and subsequent possible loop switching, candidate nodes are arranged in ascending order of priority score, resulting in an ordered list from low to high scores, clearly reflecting the priority order of each node. In a single click, the first node in the list is directly selected as the target; while when the user clicks continuously in the same position, the next node is selected in turn according to the order of the list, thus enabling quick browsing and precise adjustment of all overlapping elements.

[0109] For example, the priority scoring formula can be: P = W1 × interactivity + W2 × type + W3 × distance + W4 × depth + W5 × rendering order + W6 × size. The weighting coefficients can be set as follows: Interactivity weight W1 = -20000 (interactive elements score much lower than non-interactive elements), type weight W2 = -120 multiplied by the type priority index (buttons, input boxes, etc., have smaller indices), distance weight W3 = +80, depth weight W4 = -300, rendering order weight W5 = -480, and size weight W6 = +40 multiplied by the square root of the area (size can be excluded for text elements). A lower score indicates higher priority. Type priority mappings are as follows: Button:1, TextField:2, Checkbox:3, ImageView:4, Text:5, Layout:6, ScrollView:7, ListView:8, Container:9, Node:10. These weights and mappings can be adjusted based on actual debugging experience to achieve optimal picking results.

[0110] In this embodiment, the above-described method for selecting user interface elements will be described by way of example using a complete click-and-pick process.

[0111] First, the user holds down Ctrl and left-clicks the screen to trigger the debug command. The screen coordinates are then converted to game world coordinates (if necessary). Starting from the root node, the UI tree is recursively traversed, prioritizing child nodes with ZOrder (also known as Z-axis order or stacking order) < 0, followed by those with ZOrder ≥ 0, ensuring consistency with the actual screen display order. Invisible nodes and their subtrees are skipped. For each visible node, the clicked point is checked using its world bounding box; if so, it is added to the candidate list. The priority of each candidate node is calculated (considering interactivity, type, distance, depth, rendering order, size, etc.). Nodes are sorted by priority; if the same area is clicked repeatedly, the next candidate is selected in a loop, otherwise the highest-priority candidate is chosen. Finally, the selected node is highlighted (e.g., by drawing a green border around it) and its detailed attributes (position, size, type, etc.) are displayed in the information panel. Simultaneously, the corresponding node is located and expanded in the tree list, completing a precise selection. This process ensures accurate selection of target UI elements during debugging, even in complex overlapping interfaces, allowing users to find the desired node through continuous clicks.

[0112] This embodiment also provides a user interface element picking device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0113] This embodiment provides a picking device for user interface elements, such as... Figure 4 As shown, it includes: The first acquisition module 401 is used to acquire, in response to a debugging instruction triggered during the running of the target application, the user interface frame type to which each user interface element in the target application belongs, and the preset adapter that matches each user interface frame type. The encapsulation module 402 is used to encapsulate each user interface element into an interface node using a preset adapter, and all interface nodes have the same preset node abstract interface. The first determining module 403 is used to determine, in response to a picking operation for a target location in the user interface, a candidate interface node in the target application associated with the target location; The second acquisition module 404 is used to call the preset node abstraction interface to obtain the first attribute information of each candidate interface node corresponding to each candidate interface node. The second determining module 405 is used to determine the target user interface element from the candidate user interface elements corresponding to each candidate interface node based on the first attribute information.

[0114] In some alternative implementations, the encapsulation module 402 includes: The Get submodule is used to utilize the preset adapter to call the application programming interface of each user interface element and obtain the second attribute information of each user interface element. The conversion submodule is used to convert each second attribute information into the format defined by the preset node abstract interface, and generate each interface node. The second attribute information includes at least one of the following: identification information, hierarchical relationship information, geometric attribute information, state attribute information, and extended information.

[0115] In some alternative implementations, the first determining module 403 includes: Construct submodules to respond to picking operations at target locations in the user interface, and build the user interface tree of the target application based on the parent-child relationships between various interface nodes; The query submodule is used to query the user interface tree based on the target location and obtain candidate interface nodes associated with the target location.

[0116] In some optional implementations, the query submodule includes: The transformation unit is used to perform coordinate transformation on the target position to obtain the target coordinates in the preset standard screen coordinate system; The acquisition unit is used to acquire the target bounding box of each interface node in the preset standard screen coordinate system based on the preset node abstract interface. The first determining unit is used to determine the interface node corresponding to the target bounding box as a candidate interface node when the target coordinates are located within the target bounding box.

[0117] In some alternative implementations, the first determining module 403 further includes: The Skip Submodule is used to skip queries on any interface node and all its child nodes if any interface node is invisible.

[0118] In some alternative implementations, the second determining module 405 includes: The first determination submodule is used to determine the priority of each candidate interface node based on the first attribute information according to a predefined multi-dimensional priority strategy. The second determination submodule is used to determine the target user interface element from each candidate user interface element based on priority; The first attribute information includes at least one of the following: interactivity information, type information, distance information from the target location, depth information in the user interface tree, rendering order information, and size information.

[0119] In some optional implementations, the second determining submodule includes: The sorting unit is used to sort each candidate interface node based on priority to obtain the sorting result; The second determining unit is used to determine the candidate user interface element corresponding to the candidate interface node with the highest priority in the sorting results as the target user interface element.

[0120] In some optional implementations, the second determining submodule further includes: The third determining unit is used to respond to continuous picking operations on the same target location by sequentially selecting candidate interface nodes according to priority, and determining the candidate user interface element corresponding to the candidate interface node as the target user interface element.

[0121] The user interface element picking device provided in this application embodiment can execute the user interface element picking method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0122] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0123] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0124] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0125] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from memory 508, or installed from ROM 502. When the computer program is executed by processor 501, it performs the functions defined in the user interface element selection method of embodiments of this application.

[0126] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0127] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the method for selecting user interface elements shown in the above embodiments is implemented.

[0128] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0129] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for selecting user interface elements, characterized in that, The method includes: In response to a debugging instruction triggered during the execution of the target application, the user interface framework type to which each user interface element in the target application belongs, and a preset adapter that matches each user interface framework type are obtained. Using the preset adapter, each of the user interface elements is encapsulated as an interface node, and all the interface nodes have the same preset node abstract interface; In response to a picking operation targeting a location in the user interface, candidate interface nodes associated with the target location in the target application are determined; For each of the candidate interface nodes, the preset node abstraction interface is called to obtain the first attribute information of each of the candidate interface nodes; Based on the first attribute information, the target user interface element is determined from the candidate user interface elements corresponding to each candidate interface node.

2. The method according to claim 1, characterized in that, The step of encapsulating each of the user interface elements into interface nodes using the preset adapter includes: Using the preset adapter, the application programming interface of each user interface element is called to obtain the second attribute information of each user interface element; Each of the second attribute information is converted into the format defined by the preset node abstract interface to generate each of the interface nodes; The second attribute information includes at least one of the following: identification information, hierarchical relationship information, geometric attribute information, state attribute information, and extended information.

3. The method according to claim 1, characterized in that, The step of determining candidate interface nodes associated with the target location in the target application in response to a picking operation for a target location in the user interface includes: In response to a picking operation targeting a location in the user interface, a user interface tree of the target application is constructed based on the parent-child relationship between the various interface nodes. Based on the target location, query the user interface tree to obtain candidate interface nodes associated with the target location.

4. The method according to claim 3, characterized in that, The step of querying the user interface tree based on the target location to obtain candidate interface nodes associated with the target location includes: The target position is transformed to obtain the target coordinates in a preset standard screen coordinate system; Based on the preset node abstraction interface, obtain the target bounding box of each interface node in the preset standard screen coordinate system; If the target coordinates are located within the target bounding box, the interface node corresponding to the target bounding box is determined as the candidate interface node.

5. The method according to claim 3, characterized in that, The method further includes: If any of the interface nodes is invisible, the query for that interface node and all its child nodes is skipped.

6. The method according to claim 1, characterized in that, The step of determining the target user interface element from the candidate user interface elements corresponding to each candidate interface node based on the first attribute information includes: According to a predefined multi-dimensional priority strategy, the priority of each candidate interface node is determined based on the first attribute information; Based on the priority, the target user interface element is determined from each of the candidate user interface elements; The first attribute information includes at least one of the following: interactivity information, type information, distance information from the target location, depth information in the user interface tree, rendering order information, and size information.

7. The method according to claim 6, characterized in that, The step of determining the target user interface element from the candidate user interface elements based on the priority includes: Based on the priority, each candidate interface node is sorted to obtain a sorting result; The candidate user interface element corresponding to the candidate interface node with the highest priority in the sorting result is determined as the target user interface element.

8. The method according to claim 7, characterized in that, The method further includes: In response to consecutive picking operations on the same target location, the candidate interface nodes are selected sequentially and cyclically according to the priority, and the candidate user interface elements corresponding to the candidate interface nodes are determined as the target user interface elements.

9. A picking device for user interface elements, characterized in that, The device includes: The first acquisition module is used to acquire, in response to a debugging instruction triggered during the execution of the target application, the user interface framework type to which each user interface element in the target application belongs, and a preset adapter that matches each user interface framework type. An encapsulation module is used to encapsulate each of the user interface elements into an interface node using the preset adapter, and all the interface nodes have the same preset node abstract interface. The first determining module is configured to determine, in response to a picking operation for a target location in the user interface, a candidate interface node in the target application associated with the target location; The second acquisition module is used to call the preset node abstraction interface to obtain the first attribute information of each candidate interface node, corresponding to each candidate interface node. The second determining module is used to determine the target user interface element from the candidate user interface elements corresponding to each of the candidate interface nodes based on the first attribute information.

10. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method for selecting user interface elements according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method for selecting user interface elements according to any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method for selecting user interface elements according to any one of claims 1 to 8.

Images