Target element positioning method and device based on interface structure perception

Abstract

The application discloses a target element positioning method and device based on interface structure perception, relates to the technical field of artificial intelligence and man-machine interaction, and realizes the collaborative optimization based on explicit structure reasoning and multi-source information by fusing multiple reasoning results. The positioning accuracy under the conditions of complex interface layout and complex natural language instructions is greatly improved, the interpretability and generalization ability of the method are enhanced, and the method provides more accurate target element positioning capability for a graphical user interface intelligent agent. The method comprises the following steps: in response to a positioning request of a target element, acquiring interface input data and a natural language instruction; constructing an interface relation graph based on the interface input data; performing multi-type reasoning on element objects in the interface relation graph to obtain multi-type reasoning results of the element objects; selecting at least two reasoning results in the multi-type reasoning results for fusion, and determining a positioning result of the target element according to a fusion result.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a method and apparatus for locating target elements based on interface structure perception. Background Technology

[0002] With the development of artificial intelligence, natural language processing, and human-computer interaction technologies, automated operation based on natural language command-driven graphical user interfaces (GUIs) has gradually become an important research direction. GUI agents need to understand user commands and accurately locate target elements within the current interface to complete operations such as clicking, inputting, and selecting. Since GUIs typically contain various elements such as buttons, menus, input boxes, lists, and pop-ups, and have complex layouts, numerous elements, and diverse types, different elements may have similar text, functions, or visual styles. Therefore, element localization has become a key technical problem affecting the performance of intelligent agents. Especially in multi-step task scenarios, localization errors in early steps may gradually accumulate in subsequent executions, thereby reducing the stability and success rate of the entire task chain.

[0003] In existing technologies, visual perception results from interface screenshots or flat element lists obtained through element detection are typically used as input. These are then combined with textual similarity, semantic similarity, or visual features between natural language commands and candidate elements to determine the target element. While this approach is effective in simple scenarios, it has limitations with complex interfaces and commands: firstly, flat element lists typically only provide element-level location results, making it difficult to support complex command parsing; secondly, when multiple semantically similar, textually identical, or functionally similar elements exist on the interface, relying on visual features or textual semantics for matching can easily lead to ambiguity and misjudgment of the target element.

[0004] To overcome the limitations of flat representations, some studies have attempted to introduce graph models to model interface structures. However, existing graph model methods typically only use graph structures as implicit feature inputs to classification models, making it difficult to directly express and enforce structural conditions. Summary of the Invention

[0005] In view of this, this application provides a target element localization method and apparatus based on interface structure awareness. The main purpose is to solve the problem that existing graph model methods usually only use graph structure as implicit feature input to the classification model, making it difficult to directly express and execute structural conditions.

[0006] The first aspect provides a target element localization method based on interface structure awareness, which includes: In response to the location request of the target element, obtain the interface input data and natural language instructions; Based on the interface input data, an interface relationship graph is constructed; the interface relationship graph is an interface graph structure constructed based on a set of structured containers. Perform multi-type reasoning on the element objects in the interface relationship graph to obtain the multi-type reasoning results of the element objects; At least two inference results are selected from the multiple inference results and fused together. The location result of the target element is determined based on the fusion result.

[0007] Secondly, a target element localization device based on interface structure awareness is provided, the device comprising: The acquisition unit is used to acquire interface input data and natural language instructions in response to the location request of the target element; The construction unit is used to construct an interface relationship graph based on the interface input data; the interface relationship graph is an interface graph structure constructed based on a set of structured containers. The reasoning unit is used to perform multi-type reasoning on the element objects in the interface relationship diagram to obtain multi-type reasoning results for the element objects. The fusion unit is used to select at least two inference results from the multiple types of inference results for fusion, and determine the location result of the target element based on the fusion result.

[0008] Thirdly, a target element localization device based on interface structure awareness is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the program, it implements the above-mentioned target element localization method based on interface structure awareness.

[0009] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-described target element localization method based on interface structure awareness.

[0010] By employing the above technical solutions, this application provides a target element localization method and apparatus based on interface structure perception, which achieves the following beneficial effects compared with existing methods: First, by constructing an interface relationship graph, container objects, element objects, and their relationships in the graphical user interface are explicitly modeled in the form of a graph structure, transforming interface structure information from implicit features into an explicit structural representation that can be directly manipulated and reasoned about; Second, by performing multi-type reasoning on the interface relationship graph, matching and filtering can be performed directly on the graph structure, making full use of the explicit relationships between elements for accurate localization, effectively narrowing the candidate range and significantly reducing the risk of mismatch due to visually similar or semantically similar elements; In addition, by fusing multiple reasoning results, collaborative optimization based on explicit structural reasoning and multi-source information is achieved, which not only greatly improves the localization accuracy under complex interface layouts and complex natural language instructions, but also enhances the interpretability and generalization ability of the method, providing graphical user interface agents with more accurate target element localization capabilities and improving the efficiency of agents in understanding and executing user tasks.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a target element localization method based on interface structure awareness in one embodiment of this application. Figure 2 yes Figure 1 A flowchart illustrating a specific implementation method for step 102; Figure 3 This is a schematic diagram of the construction process of a structured container set in one embodiment of this application; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method for step 203; Figure 5 This is a schematic diagram of the construction process of the interface relationship diagram in one embodiment of this application; Figure 6 This is a schematic diagram of the visual reasoning process in one embodiment of this application; Figure 7 This is a schematic diagram of the structural reasoning process in one embodiment of this application; Figure 8This is a schematic diagram of the reasoning process in one embodiment of this application; Figure 9 This is a schematic diagram of the target element positioning device based on interface structure perception in one embodiment of this application; Figure 10 This is a schematic diagram of the device structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0013] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0014] Typically, the visual perception results from a screenshot or a list of flat elements obtained through element detection are used as input. This is combined with natural language commands and matching based on textual similarity, semantic similarity, or visual features between candidate elements to determine the target element. While this method is effective in simple scenarios, it has limitations with complex interfaces and commands: firstly, the list of flat elements usually only provides element-level location results, making it difficult to support complex command parsing; secondly, when multiple semantically similar, textually identical, or functionally similar elements exist in the interface, relying on visual features or textual semantics for matching can easily lead to ambiguity and misjudgment of the target element.

[0015] To overcome the limitations of flat representations, some studies have attempted to introduce graph models to model interface structures. However, existing graph model methods typically only use graph structures as implicit feature inputs to classification models, making it difficult to directly express and enforce structural conditions.

[0016] To address this problem, this embodiment provides a target element localization method based on interface structure awareness, such as... Figure 1 As shown, it includes the following steps: 101. In response to the location request of the target element, obtain the interface input data and natural language instructions.

[0017] In this embodiment, the location request can be automatically triggered by the graphical user interface agent when it receives a user operation task, or it can be actively initiated by the user through a specific interaction method.

[0018] Upon responding to a location request, the interface data acquisition module acquires the current graphical user interface (GUI) input data. The methods for acquiring GUI input data include, but are not limited to, the following: capturing a complete screenshot of the currently displayed GUI as visual GUI input data; reading pre-annotated interface annotation data, which includes the position, type, and attribute information of each element in the interface; and calling the interface parsing model to perform real-time parsing of the interface and acquiring the element set data output by the interface parsing model, which records detailed information about all visible elements in the interface in a structured format. Accordingly, the GUI input data includes at least one of the following: GUI screenshot image, interface annotation data, and element set data output by the interface parsing model.

[0019] While acquiring interface input data, the system also receives natural language commands from the user via a command receiving module. Natural language commands are acquired primarily through two methods: voice input and text input. When using voice input, the voice recognition module converts the user's spoken commands into text-based natural language commands. When using text input, the system directly receives text commands entered by the user via a keyboard, touchscreen, or other input devices. Natural language commands describe the target operations that the user expects the graphical user interface agent to perform, such as clicking, inputting, selecting, confirming, returning, or jumping.

[0020] In practical applications, when a user wants a graphical user interface agent to perform a certain operation, such as clicking a button or selecting a menu item, the user first sends a natural language command containing a description of the operation to the agent. Upon receiving the command, the agent immediately triggers a target element location request. In response to this request, it simultaneously acquires a complete screenshot of the current interface and the user's natural language command. The screenshot is saved as a bitmap format, and the natural language command is converted into a uniformly encoded text string. This acquired data serves as the raw input for localization processing, laying the foundation for subsequent construction of the interface relationship graph and execution of localization inference.

[0021] 102. Based on the interface input data, construct an interface relationship diagram.

[0022] In this embodiment, the interface relationship diagram is an interface diagram structure built based on a structured container set; the structured container set includes container objects, element objects and their relationships; the relationships include the subordinate relationship between element objects and container objects, the order relationship between element objects, and the hierarchical relationship between container objects.

[0023] Specifically, after acquiring the interface input data, the data is parsed to extract the attribute and position information of all element objects in the interface. When the interface input data is a screenshot, the object detection model is invoked to identify all element objects in the image and extract element information such as bounding box coordinates, element type, and text content for each element object. When the interface input data is interface annotation data, the element information recorded in the annotation file is directly read. When the interface input data is a set of element data output by the interface parsing model, this data is directly used as the parsing result.

[0024] Furthermore, a structured container set is constructed based on the parsed element information. The construction process of the structured container set includes three steps: container object identification, element object identification, and relationship establishment. In the container object identification step, based on the type attributes and size characteristics of element objects, container components with containment functions in the interface are identified, such as windows, dialog boxes, panels, lists, tables, and menus, and these container components are created as container objects. In the element object identification step, basic interactive elements in the interface are identified, such as buttons, text boxes, checkboxes, radio buttons, dropdown lists, icons, and labels, and these basic elements are created as element objects. In the relationship establishment step, the subordinate relationship between container objects and element objects is established based on the spatial position and hierarchical nesting relationships between element objects. Whether an element object belongs to a container object is determined by whether its bounding box is completely contained within the bounding box of the container object. For multiple element objects within the same container object, the order relationship between element objects can be established based on their visual arrangement in the interface. Sequential indices are assigned to element objects within the same container object according to the visual reading order from left to right and top to bottom. For container objects with nested relationships, a hierarchical relationship can be established between the container objects based on their containment relationship, forming a tree-like hierarchical structure.

[0025] After constructing the structured container set, the interface relationship graph is further built. The interface relationship graph uses a graph data structure, where each container object and element object is mapped to a node in the graph. Specifically, different types of nodes are created for different types of objects; container objects are mapped to container nodes, and element objects are mapped to element nodes. Based on the relationships defined in the structured container set, corresponding relationship edges are established between nodes. Dependency relationships are mapped to dependency edges from container nodes to element nodes, order relationships are mapped to order edges connecting adjacent element nodes within the same container, and hierarchy relationships are mapped to hierarchy edges connecting parent and child container nodes. The completed interface relationship graph fully preserves the structured information of the interface, allowing all elements in the interface and their relationships to be explicitly represented and queried.

[0026] Furthermore, to ensure the completeness of nodes in the interface relationship graph, the construction process also includes setting node attributes and edge attributes. The attribute information of container objects and element objects is stored in the interface relationship graph as attributes of the corresponding nodes, including node type, name, position coordinates, size, etc. Additional information about relationships is also stored in the interface relationship graph as attributes of the corresponding edges, including edge type, weight, direction, etc.

[0027] In practical applications, when receiving interface input data containing 10 buttons, 3 text boxes, and 1 list, the process first identifies the list as a container object and the 10 buttons and 3 text boxes as element objects. It determines that 8 buttons and 2 text boxes are located within the list container, establishing their hierarchical relationship with the list container. Based on the order of the buttons and text boxes in the list, an order relationship is established between them. Then, the constructed container objects, element objects, and their relationships are combined into a structured container set. These objects are mapped to nodes, and the relationships are mapped to edges, ultimately constructing a complete interface relationship graph.

[0028] 103. Perform multi-type reasoning on the element objects in the interface relationship diagram to obtain the multi-type reasoning results of the element objects.

[0029] In this embodiment, the multi-type reasoning includes at least one of visual reasoning, structural reasoning, and graph reasoning. These three reasoning mechanisms analyze and evaluate the element objects from different perspectives.

[0030] The visual reasoning mechanism primarily analyzes the visual attribute information of element objects. First, it extracts visual attributes such as text content, image features, element type, and functional role for each element object in the interface relationship diagram. Then, it calculates the degree of matching between these visual attributes and natural language instructions, including multiple dimensions such as text semantic matching, image content similarity, and functional role compatibility. Finally, it comprehensively calculates the matching degrees across all dimensions to obtain the visual reasoning result for each element object. This result reflects the degree of visual compatibility between the element object and the natural language instructions.

[0031] The structural reasoning mechanism primarily analyzes the structural attribute information of element objects. It first extracts structural constraint information from natural language instructions, such as the relative position of elements, their container, and hierarchical relationships. Then, it obtains the structural attributes of each element object in the interface relationship graph, including its position in the interface hierarchy, its spatial relationship with adjacent elements, and its container region. By comparing the structural attributes of the element objects with the structural constraints in the natural language instructions, the degree of conformity is calculated, yielding the structural reasoning result for each element object. This result reflects the extent to which the element object meets the instruction's structural requirements.

[0032] The graph reasoning mechanism primarily relies on deep learning reasoning based on relational graph neural networks. This mechanism first converts the interface relational graph into a graph tensor representation suitable for neural network processing, including node features, edge relationships, and global context information. Then, the graph tensor is input into a pre-trained relational graph neural network, which learns complex association patterns between interface elements through multi-layer message passing and feature aggregation. The network ultimately outputs a probability prediction value for each element node as a target element, i.e., the graph reasoning result. This graph reasoning result reflects the intelligent prediction of the target element based on the deep learning model.

[0033] Through the coordinated work of the three reasoning mechanisms mentioned above, the degree of matching between each element object and the natural language instruction can be comprehensively evaluated from different dimensions, generating multiple types of reasoning results including visual reasoning results, structural reasoning results, and graph reasoning results, providing rich decision-making basis for subsequent target element localization.

[0034] 104. Select at least two reasoning results from the multiple types of reasoning results and fuse them to determine the location result of the target element based on the fusion result.

[0035] In one possible implementation, the fusion process of multiple types of reasoning results can employ a weighted fusion strategy. First, at least two of the following reasoning results are obtained for each element object: visual reasoning result, structural reasoning result, and graph reasoning result. Then, these reasoning results are weighted according to preset fusion weights to obtain a fusion score for each element object. The fusion weights can be configured according to the actual application scenario; for example, the weight of visual reasoning results can be increased when visual information is relatively reliable, and the weight of structural reasoning results can be increased when structural constraints are relatively clear.

[0036] After the fusion calculation is completed, the fusion scores of all element objects are compared and analyzed. The candidate element with the highest fusion score is selected as the final target element. This target element is the result of locating the interface element pointed to by the natural language command.

[0037] The target element localization method based on interface structure awareness provided in this application has the following beneficial effects compared with existing methods: First, by constructing an interface relationship graph, the container objects, element objects and their relationships in the graphical user interface are explicitly modeled in the form of a graph structure, transforming the interface structure information from implicit features into an explicit structural representation that can be directly manipulated and reasoned about; Second, by performing multi-type reasoning on the interface relationship graph, matching and filtering can be performed directly on the graph structure, making full use of the explicit relationships between elements for accurate localization, effectively narrowing the candidate range and significantly reducing the risk of mismatch due to visually similar or semantically similar elements; In addition, by fusing multiple reasoning results, collaborative optimization based on explicit structural reasoning and multi-source information is achieved, which not only greatly improves the localization accuracy under complex interface layouts and complex natural language instructions, but also enhances the interpretability and generalization ability of the method, providing the graphical user interface agent with a more accurate target element localization capability and improving the efficiency of the agent in understanding and executing user tasks.

[0038] In practical applications, the purpose of constructing interface relationship graphs is to transform unstructured interface information into a structured, computable graph data structure, providing a unified basic representation framework for subsequent tasks such as intelligent interaction, automated testing, and interface understanding. Interface relationship graphs can transform a collection of structured containers into a graph structure, thus clearly representing the nesting relationships between containers, the sequential relationships between elements, and the dependency relationships between elements and containers. Specifically, for example... Figure 2 As shown, step 102 above includes the following steps: 201. Obtain a set of interface elements based on the interface input data.

[0039] 202. Based on at least one of the spatial distribution, text information, and semantic attributes among the element objects in the interface element set, construct a structured container set.

[0040] 203. Convert the element objects, container objects and their relationships in the structured container set into a graph structure to obtain the interface relationship graph.

[0041] In this embodiment, the interface input data may include at least one of the following: interface screenshot image, interface annotation data, and element set data output by the interface parsing model.

[0042] When the input data is a screenshot, a visual analysis model can be used to process the screenshot, identify candidate regions in the interface, and convert these candidate regions into element objects. For each element object, at least one of its positional information, textual information, and semantic attributes is determined. Positional information may include bounding box coordinates, center point coordinates, or normalized coordinates; textual information may include the candidate element's own displayed text or text obtained through character recognition; semantic attributes may include element type, interaction role, structural role, or functional category.

[0043] When the input data is labeled data or element set data, information such as element identifier, element bounding box, element text, element type, and element role can be directly read and converted into element objects. In an optional implementation, coordinate information from different sources can also be uniformly processed. For example, normalized bounding box coordinates can be converted into pixel coordinates, or pixel coordinates can be converted into normalized coordinates relative to the viewport size, to ensure coordinate consistency in the subsequent structural modeling process.

[0044] In one alternative implementation, in order to reduce noise and false detection interference in interface analysis, element objects can be filtered according to a preset confidence threshold to remove element objects with low confidence.

[0045] In one alternative implementation, to improve the stability of text matching and instruction parsing, the text information of the element object can also be normalized, for example, by removing invalid symbols, unifying capitalization, merging synonyms, or cleaning up empty text.

[0046] After obtaining the set of interface elements, a structured container set can be constructed based on at least one of the spatial distribution, text information, and semantic attributes of the element objects in the set. The structured container set is used to elevate the original flat list of elements into a structural representation with container semantics, hierarchical relationships, and local order relationships. In other words, element objects no longer exist as independent interface objects, but are organized into different container objects; that is, element objects in the structured container set are organized into different container objects, and the container objects express the regional division and hierarchical relationships of the interface.

[0047] Specifically, for each element object in the collection of interface elements, aggregation can be performed based on its spatial position, alignment, proximity, visual arrangement, text semantics, and interactive attributes. Multiple element objects with a common area affiliation or structural relationship can be grouped into the same container object. For example, multiple menu items located on the left side of the interface and arranged vertically can be aggregated into a sidebar container; multiple navigation items located at the top and arranged horizontally can be aggregated into a navigation bar container; buttons, text boxes, and tooltips located inside pop-ups can be aggregated into a pop-up container; and multiple input boxes and submit buttons located in a form area can be aggregated into a form container.

[0048] In one optional implementation, a corresponding container information extraction strategy can be adopted according to the different types of interface input data: if the interface input data is labeled data, the container information is directly read and the mapping relationship between element objects and container objects is restored; if the interface input data is a flat element set output by the interface parsing model, spatial clustering is performed based on the center point of the element object to generate multiple element clusters, and then the container bounding box is generated based on the minimum enclosing area of ​​each element cluster. The container type and container role are inferred by combining element text, position area, layout features and interaction features.

[0049] In one optional implementation, container types include navigation containers, list containers, grid containers, form containers, and general containers, while container roles include headers, sidebars, footers, list areas, content areas, and form areas. It should be noted that the above container types and roles are merely illustrative examples, and those skilled in the art can adaptively extend or replace the container types and roles according to the specific interface type, application scenario, or annotation system requirements.

[0050] Specifically, in the process of establishing container hierarchy based on the spatial containment relationship between container objects, for any given container object, the container object with the smallest area that contains its bounding box can be found among other container objects as the parent container, forming a relatively stable parent-child hierarchy structure. It should be noted that for element objects that cannot be assigned to an existing container, they can be placed in the default container or the page-level container to avoid the element objects being lost during the subsequent construction of the interface relationship graph.

[0051] For a detailed explanation of the construction process of a specific structured container collection in practical applications, please refer to [link / reference]. Figure 3As shown, the interface input data is first parsed, and candidate elements are obtained through a dual path of image and data and then normalized. Then, container objects are generated by combining spatial layout and semantic information. Subsequently, a hierarchical structure is established based on the subordinate relationship between elements, and further integrated into a structured container set with clear boundaries, semantic types and internal composition features. This set not only retains the physical layout information of the interface, but also incorporates semantic hierarchy and functional division, supporting the rapid matching and reasoning of structural constraints in natural language instructions.

[0052] After generating a set of structured containers, an interface relationship graph is further constructed based on this set. The interface relationship graph is used to uniformly represent container objects, element objects, and their structural relationships as a graph structure, facilitating subsequent explicit structural constraint reasoning and learned graph reasoning. Specifically, such as... Figure 4 As shown, step 203 above includes the following steps: 301. Map the container objects in the structured container set to container nodes.

[0053] 303. Map the element objects in the structured container collection to element nodes.

[0054] 303. Establish relationship edges based on the association between the container nodes and / or element nodes.

[0055] 304. Convert the structured container set into a graph structure containing the container nodes, the element nodes, and the relationship edges to obtain the interface relationship graph.

[0056] In this embodiment, the relationship edge includes at least one of the following: parent-child relationship edge, spatial relationship edge, containment relationship edge, sequential relationship edge, and same role relationship edge.

[0057] Among them, parent-child relationship edges are used to represent the parent-child hierarchical relationship between container nodes; spatial relationship edges are used to represent the relative spatial position between container nodes and / or element nodes; containment relationship edges are used to represent that a container node contains an element node; order relationship edges are used to represent the visual ordering relationship between multiple container nodes with the same parent container node or multiple element nodes with the same container node; same role relationship edges are used to represent the semantic association relationship between container nodes with the same role type. Correspondingly, the process of constructing the interface relationship graph includes: adding container nodes; adding element nodes; establishing containment relationship edges between container nodes and element nodes; specifically, establishing parent-child relationship edges based on the containment relationship between containers; establishing spatial relationship edges based on the spatial overlap and relative position of the center point between container nodes and / or element nodes; establishing order relationship edges based on the visual ordering of sibling nodes; and establishing same-role relationship edges based on the same or similar roles.

[0058] In this way, container ownership, hierarchical relationships, spatial relationships, and local order relationships in the interface can be explicitly expressed. Thus, the interface relationship graph can serve both as the execution object for explicit structural constraint reasoning and as the input basis for graph neural network models.

[0059] For a detailed process of constructing the interface relationship diagram in practical applications, please refer to [link / reference needed]. Figure 5 As shown, taking a set of structured containers as input, it is first mapped to container nodes and element nodes; then, containment relationship edges between containers, parent-child relationship edges between elements and containers, spatial proximity relationship edges between elements or containers, and order relationship edges in the interface layout are established respectively; finally, all nodes and relationship edges are integrated to output a unified interface relationship graph that integrates structural, semantic, and layout information.

[0060] Understandably, a single reasoning method often struggles to handle complex and ever-changing interface scenarios and diverse user commands. To comprehensively capture the feature information of element objects across different dimensions and improve the accuracy of understanding natural language commands, it is necessary to combine these multiple types of reasoning. This allows for a comprehensive evaluation of element objects across complementary dimensions, thereby improving the accuracy of element location. Specifically, step 103 above includes the following steps: Visual reasoning is performed on the element objects in the interface relationship diagram based on the visual attribute information of the element objects to obtain the visual reasoning result of the element objects; the visual attribute information includes at least one of the element object's text content, bounding box coordinates, element type, and screenshot area image. Structural reasoning is performed on the element objects in the interface relationship graph based on the structural constraint information of the element objects to obtain the structural reasoning result of the element objects; the structural constraint information includes at least one of sequence constraints, container constraints, parent level constraints and spatial position constraints. The graph tensor representation based on the interface relationship graph performs graph reasoning on the element objects in the interface relationship graph to obtain the graph reasoning result of the element objects; the graph tensor representation is a node embedding vector obtained by encoding the interface relationship graph through a graph neural network.

[0061] The aforementioned visual reasoning primarily measures whether candidate elements match natural language instructions in terms of visual appearance, textual semantics, and element role. Specifically, such as... Figure 6 As shown, the visual reasoning process includes the following steps: 401. Obtain the bounding box of each element object in the interface relationship diagram, and locate the corresponding region image of each element object from the interface input data based on the bounding box.

[0062] 402. Perform multi-dimensional visual analysis on the region image to obtain multi-dimensional visual attribute information of the element objects.

[0063] 403. Based on the multi-dimensional visual attribute information of the element object, calculate the degree of visual matching between the element object and the natural language instruction in different dimensions.

[0064] 404. The visual matching degree of the element object and the natural language instruction in different dimensions is weighted and fused to obtain the visual reasoning result of the element object.

[0065] In this embodiment, for each element object in the interface relationship diagram, a corresponding region image can be cropped from the interface screenshot based on the bounding box of the element object. The region image is then combined with the text information, element type, element role, and resolution confidence of the element object to calculate the degree of matching between the element object and the natural language instruction.

[0066] For example, when the natural language instruction is "click the submit button", if the text of an element object is "submit", that element object can obtain a high text matching score. When the natural language instruction is "click the search icon", even if the element object itself has no text, if its corresponding image area is displayed as a magnifying glass icon, it can also obtain a high visual candidate score through image semantic matching.

[0067] Specifically, multi-dimensional visual attribute information includes text information, element type, element role, and resolution confidence. Correspondingly, the visual matching degree between the element object and the natural language instruction across different dimensions can be composed of one or more of the following: text matching degree, image semantic matching degree, action role compatibility matching degree, and resolution confidence matching degree. Text matching degree measures the degree of matching between the candidate element object's text and the target element description in the natural language instruction, and can be calculated using methods such as exact match, keyword inclusion, fuzzy match, edit distance, synonym matching, or text vector similarity. Image semantic matching degree measures the visual semantic similarity between the element object's screenshot area and the target element description in the natural language instruction, and can be output by a visual language model, icon recognition model, image classification model, or interface resolution model. Action role compatibility matching degree determines whether the action type in the natural language instruction is compatible with the element object's role; for example, "click" usually corresponds to a button, link, or icon, "input" usually corresponds to an input box, and "select" usually corresponds to a checkbox, radio button, or dropdown list. The confidence level of the resolution can be derived from the output of a visual resolution model, an OCR model, or an interface element detection model, and is used to indicate the reliability of the element object itself being correctly detected or identified.

[0068] In one alternative implementation, the visual reasoning result can be obtained using the following weighted fusion method:

[0069] in, Represents element object The results of visual reasoning; Indicates the degree of text matching; Indicates the degree of semantic matching between images; Indicates the degree of compatibility and matching between action characters; Indicates the degree of confidence matching in parsing; , This indicates the corresponding weight. The visual matching degree in each dimension can be normalized to the [0,1] interval before weighted fusion. If an element object does not have a visual matching degree in a certain dimension, for example, if the element object has no text, the corresponding text matching degree can be excluded from the calculation, or the text matching degree can be set to a neutral value.

[0070] In summary, visual reasoning results represent the degree of matching between an element object and a natural language instruction in the visual or semantic dimensions. Essentially, it evaluates whether an element object resembles the target element described by the natural language instruction based on its appearance, text, role, and parsing reliability.

[0071] The aforementioned structural reasoning is primarily used to assess whether element objects within the interface structure satisfy the regional, sequential, hierarchical, or spatial constraints contained in natural language instructions. This process requires extracting structural constraint information of the target element from the natural language instructions. This embodiment employs a combination of rule matching and semantic parsing to extract structural constraint information. In the rule matching stage, a structural constraint keyword library is maintained, containing keywords and phrase patterns related to various structural constraints. Natural language instructions are matched against the keyword library to identify the explicitly expressed structural constraint information within the instructions. In the semantic parsing stage, a pre-trained language understanding model is used to perform semantic analysis on the natural language instructions, understanding the implicit structural constraint information within them. Semantic parsing effectively supplements rule matching, especially when the natural language instructions are ambiguous or indirect.

[0072] The structural constraint information includes at least one of the following: sequence constraints, container constraints, parent level constraints, and spatial location constraints.

[0073] When extracting order constraints, the main focus is on identifying words and phrases in natural language instructions that indicate sequential position. Keywords for order constraints include words indicating relative order such as first, second, third, last, middle, front, back, left, and right. For example, when a natural language instruction includes phrases like "click the first button" or "select the last item in a list," the order constraint information is extracted, and the relative position of the target element among its sibling elements is recorded.

[0074] When extracting container constraints, the main focus is on identifying words and phrases in natural language instructions that indicate the scope of the container. Keywords related to container constraints include prepositional phrases indicating spatial scope, such as "in a window," "in a dialog box," "in a menu," "in a list," and "on a panel." For example, when a natural language instruction includes statements like "click the OK button in a login window" or "select an option in a settings menu," the container constraint information is extracted, and the container scope of the target element is recorded.

[0075] When extracting parent level constraints, the main focus is on identifying words and phrases in natural language instructions that indicate hierarchical paths. Keywords for parent level constraints include modifiers indicating hierarchical relationships, such as those found in the main menu, second-level pages, and sub-menus. For example, when a natural language instruction includes statements like clicking the "Save" option in the "File" sub-menu under the main menu or selecting the "Network Settings" item in the settings page, the parent level constraint information is extracted, and the parent container hierarchy of the target element is recorded.

[0076] When extracting spatial constraints, the main focus is on identifying words and phrases in natural language instructions that indicate spatial location. Keywords for spatial constraints include directional words indicating relative position, such as top-left, top-right, bottom-left, bottom-right, middle, above, below, left, right, top, and bottom. When natural language instructions include phrases like "click the close button in the top-right corner" or "select the middle position," spatial constraint information is extracted, and the spatial position of the target element relative to the interface boundary or other reference elements is recorded.

[0077] In practical applications, when a user inputs a natural language command and clicks the first checkbox under the Network tab in the settings dialog box, the natural language command is first semantically parsed to identify key information such as the settings dialog box, the Network tab, "first," and the checkbox. Then, container constraints are extracted from the settings dialog box and the Network tab, including the first order constraint and the element type constraint checkbox. This constraint information is then integrated into a structured constraint set, clearly defining the multiple constraints that the target element needs to satisfy.

[0078] Specifically, extracting structural constraint information of target elements from natural language instructions includes the following steps: Extract the order constraints representing the relative order of element objects within the sibling element set based on the natural language instructions; and / or Based on the natural language instructions, extract container constraints representing the container object or region to which the element object belongs; and / or Based on the natural language instructions, extract the parent hierarchy constraints representing the element object as it lies below the specified parent container object or ancestor container object; and / or Based on the natural language instructions, extract the spatial position constraints representing the relative position of element objects in the interface or container object; In one alternative implementation, structural constraint information can be extracted using rule matching. For example, sequence constraints, container constraints, and spatial location constraints can be identified based on ordinal numbers, region terms, directional terms, and hierarchical relation terms. Structural constraint information can also be extracted using a semantic parsing model. Furthermore, structural constraint information can be extracted by combining rule matching with a semantic parsing model to improve the accuracy and robustness of constraint extraction under complex instructions.

[0079] In one optional implementation, to address the semantic inconsistency between region descriptions in natural language instructions and container role labels in the interface relationship diagram, a role alias mapping relationship can be established. This mapping relationship can map diverse role descriptors in natural language instructions to one or more container roles in the interface structure system. Specifically, when a natural language instruction contains a descriptor indicating navigation, it can be mapped to possible container roles such as header, sidebar, or footer; when a natural language instruction contains settings or privacy-related descriptions, it can be mapped to candidate container roles such as form area, settings area, or content area. This role alias mapping process can effectively alleviate the semantic gap between natural language expression and the structural label system, significantly improving the system's ability to understand diverse user instructions.

[0080] Specifically, such as Figure 7 As shown, the structural reasoning process includes the following steps: 501. Obtain the structural attribute information of each element object in the interface relationship diagram.

[0081] 502. Based on the structural attribute information of the element object, determine the degree of matching between the element object and different constraint conditions in the structural constraint information.

[0082] 503. The matching degree of the element object with different constraints in the structural constraint information is fused to obtain the structural reasoning result of the element object.

[0083] In this embodiment, the structural attribute information includes the hierarchical path of the element object, the list of sibling elements, the parent container object, and spatial coordinates.

[0084] For each element object, its container, parent container, ancestor container chain, sibling element set, visual arrangement order, and relative spatial position can be determined in the interface relationship diagram. Based on these structural attribute information, it can be determined whether the element object satisfies the structural description in the natural language instruction.

[0085] For example, when the natural language instruction is "Click on Settings in the left menu," there may be multiple element objects with the text "Settings" on the interface. Relying solely on visual reasoning may make it difficult to distinguish the true target element. In this case, the interface relationship diagram can be used to determine whether each "Settings" element object is located within the left menu container. "Settings" element objects located within the left menu container or its ancestor containers will receive a higher degree of matching for container constraints, while "Settings" element objects located in the main text area, pop-up area, or other areas will receive a lower degree of matching for container constraints.

[0086] For example, when the natural language instruction is "Select the second product," the product element objects can be sorted according to visual order within the product list container. The element object in the second position will have a higher degree of matching for the order constraint. When the natural language instruction is "Click 'Edit' in the account information area," the chain of parent containers for each "Edit" button can be searched upwards. If the ancestor container of a "Edit" button matches the "account information area," then that element object will have a higher degree of matching for the parent level constraint. When the natural language instruction is "Click the icon to the right of the search box," the relative spatial position between the element object and the search box can be used to determine whether the icon is located to the right of the search box, thereby calculating the degree of matching for the spatial position constraint.

[0087] In one alternative implementation, the structural reasoning result can be obtained by weighted fusion of the matching degrees of the above-mentioned multiple constraints:

[0088] in, Represents element object The structural reasoning results; Indicates the degree of matching of container constraints; Indicates the degree of matching of sequential constraints; Indicates the degree of matching of parent level constraints; Indicates the degree of matching of spatial location constraints; , This indicates the corresponding weight.

[0089] In another alternative implementation, the structural reasoning result can be obtained by multiplying and fusing the matching degrees of the above-mentioned multiple constraints:

[0090] It should be noted that when using the weighted summation method, some structural constraints can be incompletely satisfied but still retain a certain score. When using the multiplicative fusion method, if a key structural constraint is clearly not satisfied, the final structural inference result will be significantly suppressed, making it more suitable to treat certain constraints as strong constraints. If a certain type of structural constraint is not present in the natural language instruction, the matching degree of that type of structural constraint can be excluded from the calculation or set to a neutral value. For example, for simple instructions like "click submit," if the natural language instruction does not contain a clear description of the region, order, hierarchy, or space, the matching degree of the corresponding structural constraint can be set to a neutral value to avoid unnecessary interference with the structural inference result.

[0091] Therefore, the structural reasoning result represents the degree of matching between the element object and the structural constraint information in the natural language instruction. Its essence is to start from the structural position of the candidate element in the interface relationship diagram and evaluate whether the candidate element is in the position, region, order or hierarchy required by the natural language instruction.

[0092] Specific graph reasoning is output by a relational graph neural network after learning and reasoning based on the interface relational graph. This output represents the probability that a candidate element is predicted as the target element. Specifically, for example... Figure 8 As shown, the graph reasoning process includes the following steps: 601. Convert the interface relationship graph into a graph tensor representation processed by the relationship graph neural network.

[0093] 602. Input the graph tensor representation into the relational graph neural network for multi-round message passing reasoning.

[0094] 603. In the output stage of the relational graph neural network, target classification is performed on the element objects to obtain the graph reasoning results of the element objects.

[0095] In this embodiment, the graph tensor representation includes a node feature matrix, edge index, edge type, node type, element node index, node text features, node geometric features, and natural language instruction features. Node types can include container nodes and element nodes, and edge types can include containment edges, parent-child edges, spatial edges, sequential edges, and edges with the same role.

[0096] In the reasoning process of a relational graph neural network, different types of relational edges use different parameters or message passing methods, so that element nodes receive regional semantic information from their container nodes, order information from sibling element nodes, and spatial relationship information from adjacent element nodes. Container nodes aggregate the information of their internal elements to form a structured representation of the local interface area. After multiple rounds of message passing, the representation of each element node not only includes its own text, position, type, and role information, but also integrates information such as its container, surrounding elements, sibling order, parent-child hierarchy, and spatial context.

[0097] Since the ultimate goal is to locate the operable interface elements, rather than the container itself, target classification can be performed only on the element nodes during the output phase. For each element node, the relational graph neural network outputs a graph inference result for the element object.

[0098] In one alternative implementation, the graph reasoning result can be obtained through target classification:

[0099] in, Represents element object The results of graph reasoning; Represents element object Target classification; This represents the set of interface elements. The graph reasoning result represents the probability that an element object is predicted to be the target element of a natural language instruction.

[0100] During the training phase, if the labeled data contains real target elements, the element nodes corresponding to the real target elements can be used as supervision labels, and the cross-entropy loss function can be used to train the model.

[0101] in, This represents the true target element. By training a relational graph neural network, the correspondence between natural language instructions, interface structural relationships, and target elements can be learned. Therefore, the target score is essentially the target probability obtained through learned reasoning based on the entire interface relational graph and natural language instructions.

[0102] In this embodiment, specifically in the process of converting the interface relationship graph into a graph tensor representation for processing by a relationship graph neural network, the nodes in the interface relationship graph are first sorted according to a preset sorting rule to obtain a node sequence; the nodes include container nodes and element nodes, and the preset sorting rule is based on the depth-first traversal order of the nodes in the interface hierarchy to ensure that the node arrangement can accurately reflect the hierarchical structure characteristics of the interface.

[0103] After the nodes are sorted, the index of each element node in the node sequence is recorded based on the node sequence. This index recording process provides accurate positional reference information for subsequent node localization and feature extraction, so that the position of each element node in the overall sequence can be accurately identified and tracked.

[0104] Next, the bounding box information of each node is obtained and normalized to obtain normalized geometric features. The purpose of normalization is to eliminate the differences in scale and position of the bounding boxes of different nodes, resulting in a standardized geometric feature representation. These normalized geometric features can more accurately describe the spatial position and size information of nodes in the interface.

[0105] Finally, the node sequence, node input feature vector, bucketed relation edges, and element node indices are combined into a graph tensor representation. The node input features include at least one of geometric features, node type features, role features, type features, text features, and instruction features. The comprehensive use of these features enables the graph tensor representation to fully capture the multidimensional information of the interface relation graph, providing high-quality input data for the subsequent processing of the relation graph neural network.

[0106] In general, after obtaining visual reasoning results, structural reasoning results, and graph reasoning results respectively, the sources and functions of these different types of reasoning results differ. Visual reasoning results mainly come from interface screenshots, OCR text, candidate region images, element roles, and visual resolution confidence, used to determine whether an element object visually or semantically resembles the target element. Structural reasoning results mainly come from interface relationship graphs and structural constraints in natural language instructions, used to determine whether an element object is located in the correct region, order, hierarchy, or spatial position. Graph reasoning results mainly come from the learned reasoning output of a relationship graph neural network, used to determine whether an element object is most likely the target element from the perspective of the overall interface relationship graph. In short, visual reasoning results focus on the element object itself, structural reasoning results focus on the structural relationships of the element object in the interface, and graph reasoning results focus on the overall judgment learned by the graph neural network. These three are not redundant calculations, but rather provide visual evidence, structural evidence, and learned graph reasoning evidence respectively.

[0107] Specifically, step 104 above includes the following steps: At least two of the visual reasoning results, the structural reasoning results, and the graph reasoning results are weighted and fused to obtain a fusion score for each element object; The element with the highest fusion score is selected as the target element; the fusion weight of the weighted fusion is either a preset fixed value or adaptively adjusted according to the complexity of the natural language instruction, the confidence of visual analysis, and the reliability of structural analysis. When the natural language instruction is a simple text instruction, increase the fusion weight corresponding to the visual reasoning result; When the natural language instruction contains structural constraint information, the fusion weight corresponding to the structural reasoning result is increased; When the confidence level corresponding to the graph inference result is greater than a set value, the fusion weight corresponding to the graph inference result is increased.

[0108] In one alternative implementation, the fusion result can be obtained by weighted summation:

[0109] in, Represents element object The fusion score; Indicates the result of visual reasoning; Indicates the result of structural reasoning; Represents the result of graph reasoning; , This indicates the fusion weight.

[0110] In another alternative implementation, the fusion result can be obtained through multiplicative fusion:

[0111] Finally, the element with the highest fusion score is selected as the target element:

[0112] In practical applications, the fusion weights can be adaptively adjusted based on the characteristics of natural language instructions. For simple and direct instructions, such as clicking a clearly marked button, the weight of the visual reasoning result can be appropriately increased. For instructions with complex structural constraints, such as selecting a specific item in a list or clicking an element within a container, the weight of the structural reasoning result can be appropriately increased. This flexible weight adjustment mechanism enables the system to automatically optimize the fusion strategy according to different scenarios, further improving localization performance.

[0113] Furthermore, as a specific implementation of the above method, embodiments of this application provide a target element localization device based on interface structure awareness, such as... Figure 9 As shown, the device includes: an acquisition unit 71, a construction unit 72, a reasoning unit 73, and a fusion unit 74.

[0114] The acquisition unit 71 is used to acquire interface input data and natural language instructions in response to the positioning request of the target element. The construction unit 72 is used to construct an interface relationship diagram based on the interface input data; the interface relationship diagram is an interface diagram structure constructed based on a structured container set; the structured container set includes container objects, element objects and their relationships; the relationships include the subordinate relationship between element objects and container objects, the order relationship between element objects, and the hierarchical relationship between container objects; Inference unit 73 is used to perform multi-type inference on the element objects in the interface relationship diagram to obtain multi-type inference results of the element objects; The fusion unit 74 is used to select at least two inference results from the multiple types of inference results for fusion, and determine the location result of the target element based on the fusion result.

[0115] The target element localization device based on interface structure perception provided in this invention achieves the following beneficial effects compared with existing methods: First, by constructing an interface relationship graph, container objects, element objects, and their relationships in the graphical user interface are explicitly modeled in the form of a graph structure, transforming interface structure information from implicit features into an explicit structural representation that can be directly manipulated and reasoned about; Second, by performing multi-type reasoning on the interface relationship graph, matching and filtering can be performed directly on the graph structure, making full use of the explicit relationships between elements for accurate localization, effectively narrowing the candidate range and significantly reducing the risk of mismatch due to visually similar or semantically similar elements; In addition, by fusing multiple reasoning results, collaborative optimization based on explicit structural reasoning and multi-source information is achieved, which not only greatly improves the localization accuracy under complex interface layouts and complex natural language instructions, but also enhances the interpretability and generalization ability of the method, providing the graphical user interface agent with a more accurate target element localization capability and improving the efficiency of the agent in understanding and executing user tasks.

[0116] In specific application scenarios, the building unit is specifically used for: The interface element set is obtained based on the interface input data; when the interface input data includes an interface screenshot image, the interface screenshot image is processed by a visual model to obtain the interface element set; when the interface input data includes annotation data or interface element set data, the interface element set is obtained by reading the element information therein. Based on at least one of the spatial distribution, text information, and semantic attributes among the element objects in the interface element set, a structured container set is constructed; the element objects in the structured container set are organized into different container objects, and the area division and hierarchical relationship of the interface are expressed through the container objects; The element objects, container objects, and their relationships in the structured container set are converted into a graph structure to obtain the interface relationship graph.

[0117] In specific application scenarios, the building unit is further used for: Map the container objects in the structured container set to container nodes; Map the element objects in the structured container collection to element nodes; Relationship edges are established based on the associations between the container nodes and / or element nodes; the relationship edges include at least one of parent-child relationship edges, spatial relationship edges, containment relationship edges, order relationship edges, and same-role relationship edges; the parent-child relationship edges are used to represent the parent-child hierarchical relationship between container nodes; the spatial relationship edges are used to represent the relative spatial position between container nodes and / or element nodes; the containment relationship edges are used to represent the container node containing the element node; the order relationship edges are used to represent the visual ordering relationship between multiple container nodes or multiple element nodes with the same container node; the same-role relationship edges are used to represent the semantic association relationship between container nodes with the same role type. The structured container set is converted into a graph structure containing the container nodes, the element nodes, and the relationship edges to obtain the interface relationship graph.

[0118] In specific application scenarios, the inference unit is specifically used for: Visual reasoning is performed on the element objects in the interface relationship diagram based on the visual attribute information of the element objects to obtain the visual reasoning result of the element objects; the visual attribute information includes at least one of the element object's text content, bounding box coordinates, element type, and screenshot area image. Structural reasoning is performed on the element objects in the interface relationship graph based on the structural constraint information of the element objects to obtain the structural reasoning result of the element objects; the structural constraint information includes at least one of sequence constraints, container constraints, parent level constraints and spatial position constraints. The graph tensor representation based on the interface relationship graph performs graph reasoning on the element objects in the interface relationship graph to obtain the graph reasoning result of the element objects; the graph tensor representation is a node embedding vector obtained by encoding the interface relationship graph through a graph neural network.

[0119] In specific application scenarios, the inference unit is further used for: Obtain the bounding box of each element object in the interface relationship diagram, and locate the region image corresponding to each element object from the interface input data based on the bounding box; The region image is subjected to multi-dimensional visual analysis to obtain multi-dimensional visual attribute information of the element objects; the multi-dimensional visual attribute information includes text information, element type, element role, and analysis confidence. Based on the multi-dimensional visual attribute information of the element object, the visual matching degree between the element object and the natural language instruction in different dimensions is calculated. The visual matching degree between the element object and the natural language instruction in different dimensions is weighted and fused to obtain the visual reasoning result of the element object; the visual reasoning result represents the matching degree between the element object and the natural language instruction in the visual or semantic dimensions.

[0120] In specific application scenarios, the device further includes: The extraction unit is used to extract structural constraint information based on the natural language instructions before performing structural reasoning on the element objects in the interface relationship graph based on the structural constraint information of the element objects to obtain the structural reasoning result of the element objects. The extraction unit is specifically used for: Extract the order constraints representing the relative order of element objects within the sibling element set based on the natural language instructions; and / or Based on the natural language instructions, extract container constraints representing the container object or region to which the element object belongs; and / or Based on the natural language instructions, extract the parent hierarchy constraints representing the element object as it lies below the specified parent container object or ancestor container object; and / or Based on the natural language instructions, extract the spatial position constraints representing the relative position of element objects in the interface or container object; The reasoning unit is further used for: Obtain the structural attribute information of each element object in the interface relationship diagram; the structural attribute information includes the hierarchical path of the element object, the list of sibling elements, the parent container object, and the spatial coordinates. Based on the structural attribute information of the element object, determine the degree of matching between the element object and different constraint conditions in the structural constraint information; The degree of matching between the element object and different constraints in the structural constraint information is fused to obtain the structural reasoning result of the element object; the structural reasoning result represents the degree of matching between the element object and the structural constraint information in the natural language instruction.

[0121] In specific application scenarios, the inference unit is further used for: The interface relationship graph is converted into a graph tensor representation for processing by a relationship graph neural network; the graph tensor representation includes a node feature matrix, edge index, edge type, node type, element node index, node text features, node geometric features, and natural language instruction features; The graph tensor representation is input into the relational graph neural network for multi-round message passing inference. Different types of relational edges use different parameters or message passing methods, so that element nodes receive regional semantic information from their container nodes, receive order information from sibling element nodes, and receive spatial relation information from adjacent element nodes. Container nodes gather the information of their internal elements to form a structured representation of the local interface region. In the output stage of the relational graph neural network, the element objects are classified to obtain the graph inference result of the element objects; the graph inference result represents the probability that the element object is predicted as a target element of a natural language instruction.

[0122] In specific application scenarios, the inference unit is further used for: The nodes in the interface relationship graph are sorted according to a preset sorting rule to obtain a node sequence; the nodes include container nodes and element nodes, and the preset sorting rule is based on the depth-first traversal order of the nodes in the interface hierarchy. Based on the node sequence, record the index of each element node in the node sequence; Obtain the bounding box information of each node, and normalize the bounding box information to obtain the normalized geometric features. The node sequence, node input feature vector, bucketed relation edges, and element node indices are combined to form the graph tensor representation; the node input features include at least one of geometric features, node type features, role features, type features, text features, and instruction features.

[0123] In specific application scenarios, the fusion unit is specifically used for: At least two of the visual reasoning results, the structural reasoning results, and the graph reasoning results are weighted and fused to obtain a fusion score for each element object; The element with the highest fusion score is selected as the target element; the fusion weight of the weighted fusion is either a preset fixed value or adaptively adjusted according to the complexity of the natural language instruction, the confidence of visual analysis, and the reliability of structural analysis. When the natural language instruction is a simple text instruction, increase the fusion weight corresponding to the visual reasoning result; When the natural language instruction contains structural constraint information, the fusion weight corresponding to the structural reasoning result is increased; When the confidence level corresponding to the graph inference result is greater than a set value, the fusion weight corresponding to the graph inference result is increased.

[0124] Based on the above-described target element localization method based on interface structure awareness, this application embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described target element localization method based on interface structure awareness.

[0125] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0126] Based on the above-described target element localization method based on interface structure awareness and the corresponding virtual device embodiments, in order to achieve the above objectives, this application embodiment also provides a physical device for target element localization based on interface structure awareness. Specifically, it can be a computer, smartphone, tablet computer, smartwatch, server, or network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above-described target element localization method based on interface structure awareness.

[0127] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0128] In an exemplary embodiment, see Figure 10 The aforementioned physical device includes a communication bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the target element localization method based on interface structure awareness described in the above embodiments.

[0129] Those skilled in the art will understand that the entity device structure based on interface structure perception for target element localization provided in this embodiment does not constitute a limitation on the entity device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0130] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device based on the target element location perceived by the interface structure, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the technical solution of this application, compared with the existing methods, this application can directly match and filter on the graph structure by performing multi-type reasoning on the interface relationship graph, making full use of the explicit association relationships between elements for accurate positioning, effectively narrowing the candidate range and significantly reducing the risk of mismatch due to visually similar or semantically similar elements; in addition, by fusing multiple reasoning results, it achieves synergistic optimization based on explicit structure reasoning and multi-source information, which not only greatly improves the positioning accuracy under complex interface layouts and complex natural language instructions, but also enhances the interpretability and generalization ability of the method, providing the graphical user interface agent with a more accurate target element positioning capability, and improving the efficiency of the agent in understanding and executing user tasks.

[0132] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or they can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0133] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A target element localization method based on interface structure awareness, characterized in that, include: In response to the location request of the target element, obtain the interface input data and natural language instructions; Based on the interface input data, construct an interface relationship diagram; The interface relationship graph is an interface graph structure built based on a set of structured containers; the set of structured containers includes container objects, element objects and their relationships; the interface relationship graph is represented by a graph data structure, and each container object and element object is mapped to a node in the graph; according to the relationships defined in the set of structured containers, corresponding relationship edges are established between the nodes; Multi-type reasoning is performed on the element objects in the interface relationship graph to obtain multi-type reasoning results for the element objects. The multi-type reasoning includes at least one of visual reasoning, structural reasoning, and graph reasoning. The visual reasoning is based on the visual attribute information of the element objects for analysis. The structural reasoning is based on the structural attribute information of the element objects for analysis. The graph reasoning mechanism is based on deep learning reasoning using a relationship graph neural network. At least two inference results are selected from the multiple inference results and fused together. The location result of the target element is determined based on the fusion result.

2. The method according to claim 1, characterized in that, The step of constructing an interface relationship diagram based on the interface input data includes: The interface element set is obtained based on the interface input data; when the interface input data includes an interface screenshot image, the interface screenshot image is processed by a visual model to obtain the interface element set; when the interface input data includes annotation data or interface element set data, the interface element set is obtained by reading the element information therein. Based on at least one of the spatial distribution, text information, and semantic attributes among the element objects in the interface element set, a structured container set is constructed; the structured container set includes container objects, element objects, and their relationships; the relationships include the subordinate relationship between element objects and container objects, the order relationship between element objects, and the hierarchical relationship between container objects; the element objects in the structured container set are organized into different container objects, and the container objects express the area division and hierarchical relationship of the interface. The element objects, container objects, and their relationships in the structured container set are converted into a graph structure to obtain the interface relationship graph.

3. The method according to claim 2, characterized in that, The step of converting the element objects, container objects, and their relationships in the structured container set into a graph structure to obtain an interface relationship graph includes: Map the container objects in the structured container set to container nodes; Map the element objects in the structured container collection to element nodes; Relationship edges are established based on the associations between the container nodes and / or element nodes; the relationship edges include at least one of parent-child relationship edges, spatial relationship edges, containment relationship edges, order relationship edges, and same-role relationship edges; the parent-child relationship edges are used to represent the parent-child hierarchical relationship between container nodes; the spatial relationship edges are used to represent the relative spatial position between container nodes and / or element nodes; the containment relationship edges are used to represent the container node containing the element node; the order relationship edges are used to represent the visual ordering relationship between multiple container nodes or multiple element nodes with the same container node; the same-role relationship edges are used to represent the semantic association relationship between container nodes with the same role type. The structured container set is converted into a graph structure containing the container nodes, the element nodes, and the relationship edges to obtain the interface relationship graph.

4. The method according to claim 1, characterized in that, The step of performing multi-type reasoning on the element objects in the interface relationship graph to obtain the multi-type reasoning results of the element objects includes: Visual reasoning is performed on the element objects in the interface relationship diagram based on the visual attribute information of the element objects to obtain the visual reasoning result of the element objects; the visual attribute information includes at least one of the element object's text content, bounding box coordinates, element type, and screenshot area image. Structural reasoning is performed on the element objects in the interface relationship graph based on the structural constraint information of the element objects to obtain the structural reasoning result of the element objects; the structural constraint information includes at least one of sequence constraints, container constraints, parent level constraints and spatial position constraints. The graph tensor representation based on the interface relationship graph performs graph reasoning on the element objects in the interface relationship graph to obtain the graph reasoning result of the element objects; the graph tensor representation is a node embedding vector obtained by encoding the interface relationship graph through a graph neural network.

5. The method according to claim 4, characterized in that, The visual reasoning performed on the element objects in the interface relationship graph based on the visual attribute information of the element objects, to obtain the visual reasoning results of the element objects, includes: Obtain the bounding box of each element object in the interface relationship diagram, and locate the region image corresponding to each element object from the interface input data based on the bounding box; The region image is subjected to multi-dimensional visual analysis to obtain multi-dimensional visual attribute information of the element objects; the multi-dimensional visual attribute information includes text information, element type, element role, and analysis confidence. Based on the multi-dimensional visual attribute information of the element object, the visual matching degree between the element object and the natural language instruction in different dimensions is calculated. The visual inference result of the element object is obtained by weighted fusion of the visual matching degree between the element object and the natural language instruction in different dimensions; the visual inference result represents the matching degree between the element object and the natural language instruction in the visual or semantic dimensions.

6. The method according to claim 4, characterized in that, Before performing structural reasoning on the element objects in the interface relationship graph based on the structural constraint information of the element objects to obtain the structural reasoning result of the element objects, the method further includes: Extract the order constraints representing the relative order of element objects within the sibling element set based on the natural language instructions; and / or Based on the natural language instructions, extract container constraints representing the container object or region to which the element object belongs; and / or Based on the natural language instructions, extract the parent hierarchy constraints representing the element object as it lies below the specified parent container object or ancestor container object; and / or Based on the natural language instructions, extract the spatial position constraints representing the relative position of element objects in the interface or container object; The structural reasoning of the element objects in the interface relationship graph based on the structural constraint information of the element objects yields the structural reasoning results of the element objects, including: Obtain the structural attribute information of each element object in the interface relationship diagram; the structural attribute information includes the hierarchical path of the element object, the list of sibling elements, the parent container object, and the spatial coordinates. Based on the structural attribute information of the element object, determine the degree of matching between the element object and different constraint conditions in the structural constraint information; The degree of matching between the element object and different constraints in the structural constraint information is fused to obtain the structural reasoning result of the element object; the structural reasoning result represents the degree of matching between the element object and the structural constraint information in the natural language instruction.

7. The method according to claim 4, characterized in that, The graph tensor representation based on the interface relationship graph performs graph reasoning on the element objects in the interface relationship graph to obtain the graph reasoning result of the element objects, including: The interface relationship graph is converted into a graph tensor representation for processing by a relationship graph neural network; the graph tensor representation includes a node feature matrix, edge index, edge type, node type, element node index, node text features, node geometric features, and natural language instruction features; The graph tensor representation is input into the relational graph neural network for multi-round message passing inference. Different types of relational edges use different parameters or message passing methods, so that element nodes receive regional semantic information from their container nodes, receive order information from sibling element nodes, and receive spatial relation information from adjacent element nodes. Container nodes gather the information of their internal elements to form a structured representation of the local interface region. In the output stage of the relational graph neural network, the element objects are classified to obtain the graph inference result of the element objects; the graph inference result represents the probability that the element object is predicted as a target element of a natural language instruction.

8. The method according to claim 7, characterized in that, The step of converting the interface relationship graph into a graph tensor representation for processing by a relationship graph neural network includes: The nodes in the interface relationship graph are sorted according to a preset sorting rule to obtain a node sequence; the nodes include container nodes and element nodes, and the preset sorting rule is based on the depth-first traversal order of the nodes in the interface hierarchy. Based on the node sequence, record the index of each element node in the node sequence; Obtain the bounding box information of each node, and normalize the bounding box information to obtain the normalized geometric features. The node sequence, node input feature vector, bucketed relation edges, and element node indices are combined to form the graph tensor representation; the node input features include at least one of geometric features, node type features, role features, type features, text features, and instruction features.

9. The method according to any one of claims 1-8, characterized in that, The step of selecting at least two inference results from the multiple types of inference results for fusion, and determining the location result of the target element based on the fusion result, includes: At least two of the visual reasoning results, structural reasoning results, and graph reasoning results are weighted and fused to obtain the fusion score for each element object; The element with the highest fusion score is selected as the target element; the fusion weight of the weighted fusion is either a preset fixed value or adaptively adjusted according to the complexity of the natural language instruction, the confidence of visual analysis, and the reliability of structural analysis. When the natural language instruction is a simple text instruction, increase the fusion weight corresponding to the visual reasoning result; When the natural language instruction contains structural constraint information, the fusion weight corresponding to the structural reasoning result is increased; When the confidence level corresponding to the graph inference result is greater than a set value, the fusion weight corresponding to the graph inference result is increased.

10. A target element positioning device based on interface structure perception, characterized in that, include: The acquisition unit is used to acquire interface input data and natural language instructions in response to the location request of the target element; A construction unit is used to construct an interface relationship diagram based on the interface input data; The interface relationship graph is an interface graph structure built based on a set of structured containers; the set of structured containers includes container objects, element objects and their relationships; the interface relationship graph is represented by a graph data structure, and each container object and element object is mapped to a node in the graph; according to the relationships defined in the set of structured containers, corresponding relationship edges are established between the nodes; The reasoning unit is used to perform multi-type reasoning on the element objects in the interface relationship graph to obtain multi-type reasoning results for the element objects. The multi-type reasoning includes at least one of visual reasoning, structural reasoning, and graph reasoning. The visual reasoning is based on the visual attribute information of the element objects for analysis. The structural reasoning is based on the structural attribute information of the element objects for analysis. The graph reasoning mechanism is based on deep learning reasoning using a relationship graph neural network. The fusion unit is used to select at least two inference results from the multiple types of inference results for fusion, and determine the location result of the target element based on the fusion result.

Images