Web front-end AI agent efficient state representation method

Abstract

The application provides a webpage front-end AI agent efficient state representation method, which comprises the following steps: obtaining the global bounding box and type identification of webpage interactive elements and clustering space adjacent and non-overlapping elements into groups; generating semantic nodes for each group, which contain the minimum circumscribed rectangle coordinates covering all bounding boxes in the group and an internal mapping table; building directional edges based on the rectangular position relationship to form a semantic attribute graph; when the AI agent interacts, converting the target node rectangular coordinates to the viewport coordinate system according to the viewport parameters, and then determining the matching type of the sub-element estimated coordinates through the internal mapping table to realize accurate operation. The application solves the problem that there is no webpage front-end AI agent efficient state representation method which can guarantee the simplicity of state representation and improve the accuracy of agent interaction in the prior art.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an efficient state representation method for web front-end AI agents. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI agents are increasingly widely used in the field of web front-end automation, covering various scenarios such as automated web testing, intelligent information collection, and assisted browsing navigation. Traditional web automation solutions typically rely on pre-written fixed scripts or rules to complete interactive operations by locating specific elements in the document object model (DOM) tree, which significantly limits their applicability and generalization capabilities. In recent years, with breakthroughs in large language models and multimodal large model technologies, AI agents have begun to possess the ability to autonomously perceive, understand, and manipulate web interfaces. They can complete complex web interaction tasks without pre-set scripts based on natural language instructions or high-level task objectives. These AI agents typically need to transform the visual presentation or DOM structure of a webpage into a state representation that the model can understand and reason about, and then perform path planning and action decisions based on this state representation. Therefore, how to build efficient, accurate, and highly generalizable webpage state representations for AI agents has become one of the core issues in this technological field.

[0003] In existing webpage state representation schemes, a typical approach is to extract and serialize the interactive elements of a webpage one by one. Each element records its tag type, text content, attribute key-value pairs, and position coordinates within the document, forming an element list for AI agents to understand and manipulate. While this approach can completely preserve the individual information of each element, when a webpage contains a large number of interactive elements, the generated serialized description data is enormous. This leads to high computational overhead and long response latency for AI agents during reasoning and planning, especially in complex task scenarios requiring continuous multi-step interactions, where efficiency issues are particularly prominent. To alleviate these problems, another approach attempts to aggregate spatially adjacent interactive elements, replacing scattered individual elements with coarse-grained semantic units, thereby reducing the data scale of state representation. For example, a group of related elements in a navigation bar area can be aggregated into a whole, and multiple operation controls within a table row can be aggregated into an operation unit. Through this aggregation method, the complexity of state representation is significantly reduced, and the reasoning burden required for AI agents to perform high-level path planning is alleviated.

[0004] However, the above aggregation processing scheme faces a practical problem in real-world applications. In the process of aggregating spatially adjacent and visually non-overlapping interactive elements into coarse-grained semantic units, these units typically retain only overall spatial information such as the coordinate range of the smallest bounding rectangle covering all elements within the group. The original independent bounding box positions and individual identifiers of each sub-element are merged or discarded. When this semantic unit contains multiple sub-elements with different functional types—for example, an adjacent combination of a search input box and a search button, or a side-by-side arrangement of checkboxes and text labels in a table row—the AI ​​agent, while able to locate the overall area of ​​the semantic unit based on the aggregated state representation, cannot distinguish the specific operation location matching the desired functional type within that area. Because it only possesses the coordinates of the overall area and lacks the ability to index the positions of each sub-element within the group, the agent cannot accurately map the operational intent to the corresponding coordinates within that area when it needs to perform interactions on specific types of elements. The ambiguity in the location of the operation target within the aggregation area can easily lead to the agent triggering unexpected interactive responses, thereby interrupting the automated task process or producing incorrect operation results, thus limiting the reliability of the aggregation-based state representation method in complex web page tasks. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide an efficient state representation method for web front-end AI agents, aiming to solve the problem that there is a lack of an efficient state representation method for web front-end AI agents that can improve the accuracy of agent interaction while ensuring the simplicity of state representation.

[0006] According to an embodiment of the present invention, a method for efficient state representation of a web front-end AI agent includes: Obtain the bounding box position information and corresponding element type identifier of the interactive elements in the document global coordinate system of the target webpage, and cluster the interactive elements according to the bounding box position information to group the spatially adjacent and non-overlapping interactive elements into the same element group. A semantic node is generated for each element group. The semantic node contains the coordinate range of the minimum bounding rectangle that can cover all bounding boxes in the corresponding element group, and an internal mapping table that determines the relative positional relationship between each sub-element in the group and the minimum bounding rectangle. Based on the relative positional relationship of the minimum bounding rectangle corresponding to each semantic node in the document's global coordinate system, directed navigation edges with orientation attributes are established between semantic nodes, and a semantic attribute graph is formed by all semantic nodes and directed navigation edges. When the AI ​​agent initiates an interaction request to a specific type of element within the target semantic node based on the semantic attribute graph, it converts the coordinate range of the minimum bounding rectangle of the target semantic node into a coordinate range in the viewport coordinate system based on the current viewport parameters, and determines the estimated interaction coordinates of the type-matching sub-elements in the viewport coordinate system according to the internal mapping table of the target semantic node, so that the agent can perform precise operations.

[0007] In addition, the efficient state representation method for web front-end AI agents according to the above embodiments of the present invention may also have the following additional technical features: Furthermore, the steps for obtaining the bounding box position information and corresponding element type identifier of interactive elements in the target webpage in the document global coordinate system include: Traverse the document object model tree of the target webpage, starting from the root node and performing a depth-first traversal of each element node. Based on the tag name of the element node and the style attribute value finally calculated by the Cascading Style Sheets, determine interactivity and filter out all interactive element nodes. For each selected interactive element node, the layout application interface provided by the browser rendering engine is called to obtain the geometric bounding box object of the element node in the document global coordinate system. The coordinates of the top left and bottom right corners are read from the geometric bounding box object to form the bounding box position information of the interactive element. Obtain the tag name of each interactive element node in the document object model tree, map the tag name to a unique type in a preset element type classification system, use the mapped type as the element type identifier of the interactive element, and bind the bounding box position information with the corresponding element type identifier to form a set of elements to be processed with type identifiers.

[0008] Furthermore, the step of clustering the interactive elements according to the bounding box position information to group spatially adjacent and non-overlapping exchangeable elements into the same element group includes: For any two interactive elements in the set of elements to be processed, calculate the straight-line pixel distance between the center points of their bounding boxes and the degree of overlap between their bounding boxes; the center point of the bounding box is obtained by taking the average of the coordinates of the upper left corner and the lower right corner of the bounding box, and the degree of overlap is measured by the ratio of the area of ​​the intersection region of the two bounding boxes to the area of ​​the total region after merging. Two interactive elements whose straight-line pixel distance is less than a preset distance threshold and whose overlap is zero are grouped into the same element group. All elements in the set of elements to be processed are traversed to obtain several element groups composed of spatially adjacent and visually non-overlapping interactive elements.

[0009] Furthermore, the step of generating a semantic node for each group of elements and determining the internal mapping table of the relative positional relationship between each sub-element within the group and the minimum bounding rectangle includes: For each element group, calculate the coordinate range of the smallest bounding rectangle that can enclose all bounding boxes in the element group in the document global coordinate system, and write the coordinate range as the first attribute of the semantic node into the semantic node. For each sub-element in the element group, with the top left corner of the minimum bounding rectangle as the origin and the width and height of the minimum bounding rectangle as the normalization reference, calculate the normalized coordinate offset of the center point of the bounding box of the sub-element relative to the top left corner of the minimum bounding rectangle. Using the element type identifier of each sub-element in the element group as the key and the normalized coordinate offset corresponding to the sub-element as the value, construct the internal mapping table in the form of key-value pairs. When the same element type identifier corresponds to multiple sub-elements, store a list consisting of multiple normalized coordinate offsets under the key. The internal mapping table is bound to and stored as a second attribute of the semantic node.

[0010] Furthermore, based on the relative positional relationship of the minimum bounding rectangle corresponding to each semantic node in the document's global coordinate system, directed navigation edges with orientation attributes are established between semantic nodes. The steps to construct a semantic attribute graph from all semantic nodes and directed navigation edges include: For each pair of semantic nodes, obtain the coordinates of the center point of their smallest bounding rectangle, and calculate the horizontal and vertical offsets of the center point of the preceding semantic node pointing to the center point of the following semantic node. The absolute values ​​of the lateral offset and the longitudinal offset are compared to map the relative offset to one of the four main directions. If the absolute value of the lateral offset is greater than or equal to the absolute value of the longitudinal offset, the direction is determined according to the sign of the lateral offset: a positive lateral offset indicates the right direction, and a negative lateral offset indicates the left direction. If the absolute value of the lateral offset is less than the absolute value of the longitudinal offset, the direction is determined according to the sign of the longitudinal offset: a positive longitudinal offset indicates the down direction, and a negative longitudinal offset indicates the up direction. Based on the determined orientation, a one-way directed navigation edge is established between the two semantic nodes, pointing from the previous semantic node to the next semantic node, and the directed navigation edge is associated with and stored with the corresponding orientation attribute. All semantic nodes and all generated directed navigation edges are aggregated into the semantic attribute graph.

[0011] Furthermore, the step of determining the estimated interaction coordinates of the type-matching sub-elements in the viewport coordinate system based on the internal mapping table of the target semantic node includes: Based on the internal mapping table of the target semantic node, retrieve the normalized coordinate offset that matches the element type identifier specified in the interaction request; Calculate the width and height of the minimum bounding rectangle of the target semantic node after transformation to the viewport coordinate system; The predicted interactive coordinates are obtained by multiplying the horizontal component of the normalized coordinate offset by the width value in the viewport coordinate system, multiplying the vertical component by the height value in the viewport coordinate system, and then superimposing the horizontal and vertical coordinates of the top left corner of the smallest bounding rectangle in the viewport coordinate system.

[0012] Furthermore, it also includes a step for dynamically adjusting the granularity of semantic nodes: When the cumulative number of times the AI ​​agent continuously performs interactive operations on the same estimated interactive coordinates of the same semantic node without triggering the expected page state change reaches the first preset threshold, or when the number of child elements corresponding to the same element type identifier in the internal mapping table of a semantic node exceeds the second preset threshold, the semantic node is marked as a state to be expanded. For a semantic node marked as to be expanded, traverse each child element in its corresponding element group and generate an independent child node for each child element. Each child node carries the original bounding box position information and element type identifier of the child element. Between each child node, a directed navigation edge is established based on the relative orientation of the center point of each bounding box to form a temporary fine subgraph covering the local area. The directed navigation edge between the original semantic node and the external semantic node is transferred to the boundary child node in the temporary fine subgraph that matches the external orientation. When it is detected that the AI ​​agent has completed the interaction task of the area covered by the temporary fine subgraph or received the area leave instruction, all child nodes and corresponding directed navigation edges in the temporary fine subgraph are deleted, and the original semantic nodes and their original directed navigation edges are restored.

[0013] Another objective of this invention is to provide an efficient state representation system for web front-end AI agents, used to implement the aforementioned efficient state representation method for web front-end AI agents, the system comprising: The data acquisition and classification module is used to acquire the bounding box position information and corresponding element type identifier of interactive elements in the document global coordinate system of the target webpage, and to cluster the interactive elements according to the bounding box position information, so as to group the spatially adjacent and non-overlapping interactive elements into the same element group. A semantic node construction module is used to generate a semantic node for each of the element groups. The semantic node contains the coordinate range of the minimum bounding rectangle that can cover all bounding boxes in the corresponding element group, and an internal mapping table that determines the relative positional relationship between each child element in the group and the minimum bounding rectangle. The semantic attribute graph determination module is used to establish directed navigation edges with orientation attributes between semantic nodes based on the relative positional relationship of the minimum bounding rectangles corresponding to each semantic node in the document's global coordinate system. The semantic attribute graph is composed of all semantic nodes and directed navigation edges. The estimated interaction coordinate determination module is used to convert the coordinate range of the minimum bounding rectangle of the target semantic node into the coordinate range under the viewport coordinate system based on the current viewport parameters when the AI ​​agent initiates an interaction request to a specific type element within the target semantic node according to the semantic attribute graph, and to determine the estimated interaction coordinates of the type-matching sub-elements under the viewport coordinate system according to the internal mapping table of the target semantic node, so that the agent can perform precise operations.

[0014] Another objective of this invention is to provide a storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described efficient state representation method for webpage front-end AI agents.

[0015] Another objective of this invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described efficient state representation method for web front-end AI agents.

[0016] This invention acquires the bounding box position information and element type identifiers of interactive elements and performs clustering. While integrating scattered individual elements into coarse-grained semantic units to reduce the scale of state representation data, it constructs an internal mapping table for each semantic node, recording the normalized spatial positions of each sub-element within the group relative to its minimum bounding rectangle. This internal mapping table significantly reduces the data volume of semantic nodes compared to element-by-element serialization description schemes, thereby reducing the computational overhead and response latency in the AI ​​agent's path planning stage. Furthermore, it retains the ability to accurately index the positions of internal sub-elements after aggregation, avoiding the problem of ambiguous target positioning in the interaction stage caused by completely discarding individual position information in existing aggregation schemes. Next, directed navigation edges with orientation attributes are established based on the relative positional relationships between the minimum bounding rectangles of each semantic node, forming a semantic attribute graph. This provides the AI ​​agent with a clearly structured state space topology view, enabling efficient path planning at the coarse-grained semantic level. When the agent initiates an interaction request for a specific type of element, it transforms the coordinate range of the minimum bounding rectangle to the viewport coordinate system based on the current viewport parameters, and then calculates and estimates the interaction coordinates by combining the normalized position information of the corresponding type in the internal mapping table. This allows the high-level path planning result to be accurately mapped to the actual operation position on the screen. Therefore, this invention solves the problem in the prior art of lacking an efficient state representation method for web front-end AI agents that can improve the accuracy of agent interaction while ensuring the simplicity of state representation. Attached Figure Description

[0017] Figure 1 This is a flowchart of the efficient state representation method for webpage front-end AI agents in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the efficient state representation system for web front-end AI agents in the second embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device in the third embodiment of the present invention. Detailed Implementation

[0018] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0020] Example 1 Please see Figure 1 The figure shows the efficient state representation method for web front-end AI agent in the first embodiment of the present invention, which specifically includes S01-S04.

[0021] S01, obtain the bounding box position information and corresponding element type identifier of the interactive elements in the document global coordinate system of the target webpage, and cluster the interactive elements according to the bounding box position information to group spatially adjacent and non-overlapping exchangeable elements into the same element group.

[0022] Specifically, the document object model (DOM) tree of the target webpage is traversed, starting from the root node and performing a depth-first traversal of each element node. Interactivity is determined based on the element node's tag name and the style attribute values ​​calculated using Cascading Style Sheets (CSS), filtering out all interactive element nodes. For each filtered interactive element node, the layout application programming interface (API) provided by the browser's rendering engine is called to obtain the element node's geometric bounding box object in the document's global coordinate system. The coordinates of the top-left and bottom-right corners are read from the geometric bounding box object to form the bounding box position information of the interactive element. The tag name of each interactive element node in the DOM tree is obtained and mapped to a unique type in a preset element type classification system. The mapped type is used as the element type identifier for the interactive element, and the bounding box position information is bound to the corresponding element type identifier, forming a set of elements to be processed with type identifiers. More specifically, the interactivity determination rule is that if the element node's tag name belongs to a preset set of interactive tags, i.e. An element is considered interactive if its final style attributes include a pointer event value that is not none, a visibility value that is not hidden, and a display value that is not none. The preset set of interactive tags includes... a This includes elements such as button, input, select, textarea, and any element with an onclick event listener. The element type classification system includes six basic types: button, input box, link, selector, text area, and clickable container. The mapping rule is: elements with the tag name "button" or a role attribute value of "button" are classified as buttons; elements with the tag name "input" and a type attribute value of one of text, password, email, or search are classified as input boxes; elements with the tag name "input" and a type attribute value of "text", "password", "email", or "search" are classified as input boxes. a Elements with a non-empty href attribute are classified as links; elements with the tag name "select" are classified as selectors; elements with the tag name "textarea" are classified as text areas; elements with the tags name "div," "span," or "li" and bound to a click event listener are classified as clickable containers. If the same element matches multiple mapping rules, the unique type is determined by the priority order of button, input box, link, selector, text area, and clickable container.

[0023] In practical implementation, due to the complex DOM tree structure and dynamically changing style attributes of web pages, directly relying on the DOM hierarchy for state representation suffers from poor cross-page generalization capabilities. Therefore, by traversing the DOM tree and combining the final calculated values ​​of Cascading Style Sheets (CSS) for interactivity determination, the elements that users can actually interact with can be accurately selected, avoiding mis-extraction caused by hidden or disabled styles. Furthermore, by mapping tag names to a unified type classification system, the semantic heterogeneity of the original tags (such as different tags potentially expressing the same interactive function) is converged into standardized type identifiers, thus providing a unified basis for subsequent clustering and interaction intent matching in terms of type dimension. After this processing, individual elements scattered throughout the DOM tree are transformed into a set of elements to be processed, with bounding box coordinates and type identifiers as basic attributes.

[0024] Furthermore, for any two interactive elements in the set of elements to be processed, the linear pixel distance between the center points of their bounding boxes and the degree of overlap between their bounding boxes are calculated respectively. The center point of the bounding box is obtained by taking the average of the coordinates of the top left and bottom right corners of the bounding box, and the degree of overlap is measured by the ratio of the area of ​​the intersection region of the two bounding boxes to the area of ​​the total region after merging. Two interactive elements whose linear pixel distance is less than a preset distance threshold and whose degree of overlap is zero are grouped into the same element group. This process is repeated for all elements in the set of elements to be processed, resulting in several element groups composed of spatially adjacent and visually non-overlapping interactive elements. More specifically, the linear pixel distance between the center points of two bounding boxes and the degree of overlap are calculated in the following way: Let the i-th element in the set of elements to be processed be... The bounding box of each element is , No. The bounding box of each element is ; First, calculate the coordinates of the center points of the two bounding boxes. center point x-coordinate , y-axis , center point The linear pixel distance is obtained in the same way. Defined as the Euclidean distance between two center points, the formula is: The degree of overlap is expressed as the intersection-union ratio. The measurement and calculation formula is as follows: ; among which molecules The denominator is the area of ​​the intersection of the two bounding boxes. This represents the total area of ​​the merged region after merging the two bounding boxes. The area of ​​the intersecting region is obtained using a polygon clipping algorithm, specifically by first calculating the horizontal overlap length of the two rectangles. The overlap length in the vertical direction Then the area of ​​intersection Total area after merger .

[0025] In practice, interactive elements on a webpage are often organized according to functional modules, such as multiple buttons in a navigation bar, input boxes and submit buttons in a form area, and checkboxes and operation links in a table row. By calculating the Euclidean distance between the center points of elements, the spatial proximity of elements can be quantitatively measured; by calculating the intersection-union ratio (IUGR), it can be ensured that elements within the same group do not visually overlap, avoiding the incorrect grouping of overlapping or occluded elements of different levels into the same group. Using these two methods together ensures that the clustering results conform to human visual perception of the same functional area, while eliminating spatial confusion caused by absolute positioning or overlapping layouts. After this step, the originally numerous and scattered individual elements are divided into several element groups. The elements within each group are spatially cohesive and usually functionally related, thus providing a structured grouping basis for the subsequent generation of lightweight semantic nodes on a group-by-group basis.

[0026] Furthermore, the pre-defined distance threshold is determined using the following statistical learning method: A sample set is constructed by collecting typical web pages from multiple different domains and layout styles. For each sample web page, after extracting the bounding boxes of all interactive elements, the minimum straight-line pixel distance between each pair of non-adjacent cross-group elements is calculated, i.e., the distance between all element pairs that satisfy an intersection-union ratio of zero and do not belong to the same functional partition. These distance values ​​are then aggregated into a global distance sample set. The i-th distance threshold is calculated for this global distance sample set. Percentiles are used as candidate thresholds, where The value of is chosen such that, when clustering elements of sample web pages at this threshold, the silhouette coefficient of the clustering result is maximized; the silhouette coefficient is defined as, In the formula The total number of elements in the cluster. For the first The average distance from each element to all other elements in its group. For the first The average distance from each element to all elements in the nearest other group; traverse from For integer values ​​up to 30, calculate the corresponding threshold and silhouette coefficient, and select the value that maximizes the silhouette coefficient. The percentile corresponding to the value is used as the distance threshold; this distance threshold serves as a unified criterion for determining whether elements belong to the same element group. In practical implementation, the selection of the distance threshold directly affects the quality of clustering and the generalization ability of the method. If the threshold is too large, spatially distant and functionally unrelated elements will be incorrectly grouped into the same group, leading to a decrease in the semantic consistency of semantic nodes; if the threshold is too small, elements that originally belong to the same functional area cannot be aggregated, resulting in overly fragmented state representations and an inability to effectively reduce the data scale. Therefore, through statistical learning, the optimal threshold is calculated on sample web pages covering various layout styles, ensuring that the clustering results maintain high intra-group cohesion and inter-group separation in different web page scenarios. Once this threshold is determined, it can be applied to the clustering determination of all subsequent target web pages without needing to readjust parameters for each web page, thus ensuring universality and processing efficiency.

[0027] S02, generate a semantic node for each element group. The semantic node contains the coordinate range of the minimum bounding rectangle that can cover all bounding boxes in the corresponding element group, and determines the internal mapping table of the relative positional relationship between each child element in the group and the minimum bounding rectangle.

[0028] Specifically, for each element group, the coordinate range of the smallest bounding rectangle that can enclose all bounding boxes within the element group in the document's global coordinate system is calculated, and this coordinate range is written into the semantic node as its first attribute. For each child element within the element group, the normalized coordinate offset of the child element's bounding box center point relative to the smallest bounding rectangle's top-left corner is calculated, using the top-left corner of the smallest bounding rectangle as the origin and the width and height of the smallest bounding rectangle as normalization references. An internal mapping table in key-value pair form is constructed, using the element type identifier of each child element within the element group as the key and the corresponding normalized coordinate offset as the value. When the same element type identifier corresponds to multiple child elements, a list consisting of multiple normalized coordinate offsets is stored under that key. The internal mapping table is then bound to the semantic node as its second attribute. More specifically, when constructing the internal mapping table for a semantic node, the recorded normalized coordinate offsets are calculated as follows: For a given element group, let its smallest bounding rectangle... The coordinates of the top left corner in the document's global coordinate system are: The width of the rectangle is The height is And satisfy and For the first in this group Each child element has a bounding box center point coordinate of... The normalized coordinate offset corresponding to this sub-element Calculated as, , Use the element type of each child element as the key, and its corresponding normalized coordinate offset. As values, an internal mapping table in key-value pair form is constructed; if the same element type corresponds to multiple child elements, an offset list is maintained under the key, and each item in the list contains the normalized coordinate offset of a child element; this internal mapping table is stored as an additional attribute of the semantic node and bound to the semantic node.

[0029] In practical implementation, by using the coordinate range of the minimum bounding rectangle as the first attribute of the semantic node, the overall spatial position of the semantic node in the document's global coordinate system is provided. This allows the AI ​​agent to process only the boundary information of this overall region during path planning, without needing to consider the distribution details of individual elements within the group, thus significantly reducing the amount of data for state representation. Simultaneously, the internal mapping table, as the second attribute of the semantic node, records the spatial position of each child element within the group relative to the minimum bounding rectangle in the form of normalized coordinate offsets. Since the normalized offset is a proportional value relative to the size of the minimum bounding rectangle, its numerical range is constrained within the [0,1] interval and does not change with specific pixel values ​​in the document coordinate system, thus exhibiting good scaling invariance and cross-page universality. The existence of this internal mapping table allows the semantic node to appear as a compact overall region externally while retaining the ability to accurately index the spatial positions of each child element internally. In other words, it simultaneously achieves the simplicity of state representation and the accuracy of interactive positioning.

[0030] S03. Based on the relative positional relationship of the minimum bounding rectangle corresponding to each semantic node in the document's global coordinate system, a directed navigation edge with orientation attribute is established between the semantic nodes, and the semantic attribute graph is composed of all semantic nodes and the directed navigation edge.

[0031] Specifically, for each pair of semantic nodes, the coordinates of the center point of their smallest bounding rectangle are obtained, and the horizontal and vertical offsets from the center point of the preceding semantic node to the center point of the following semantic node are calculated. The absolute values ​​of the horizontal and vertical offsets are compared to map the relative offsets to one of the four main directions. If the absolute value of the horizontal offset is greater than or equal to the absolute value of the vertical offset, the direction is determined according to the sign of the horizontal offset: a positive horizontal offset indicates the right direction, and a negative horizontal offset indicates the left direction. If the absolute value of the horizontal offset is less than the absolute value of the vertical offset, the direction is determined according to the sign of the vertical offset: a positive vertical offset indicates the down direction, and a negative vertical offset indicates the up direction. Based on the determined direction, a unidirectional directed navigation edge is established between the two semantic nodes, pointing from the preceding semantic node to the following semantic node, and this directed navigation edge is associated with and stored with the corresponding direction attribute. All semantic nodes and all generated directed navigation edges are aggregated into a semantic attribute graph. In practice, the functional modules in a webpage typically follow certain spatial arrangement rules. For example, the top navigation area is located at the top of the page, the sidebar on the left, the main content area in the center, and the bottom information area at the bottom. Therefore, by calculating the relative offset between the center points of the minimum bounding rectangles of each semantic node and discretizing the continuous offsets into four main directions (up, down, left, and right), directed navigation edges with clear directional semantics are established between the semantic nodes. These directed navigation edges enable the AI ​​agent to perform path planning similar to reading a book. Figure 1 This allows for directional spatial reasoning between semantic nodes without relying on the hierarchical structure of the DOM tree or precise calculation of absolute coordinates. The semantic attribute graph integrates all semantic nodes and their spatial orientation relationships into a unified graph data structure. This preserves the spatial adjacency information between nodes and, through the introduction of orientation attributes, makes the path planning process closer to human spatial cognition, thereby improving the reasoning efficiency of AI agents in high-level decision-making stages.

[0032] S04. When the AI ​​agent initiates an interaction request to a specific type of element within the target semantic node based on the semantic attribute graph, it converts the coordinate range of the minimum bounding rectangle of the target semantic node into the coordinate range under the viewport coordinate system based on the current viewport parameters, and determines the estimated interaction coordinates of the type-matching sub-elements under the viewport coordinate system according to the internal mapping table of the target semantic node, so that the agent can perform precise operations.

[0033] In practice, the coordinate range of the minimum bounding rectangle is converted into the pixel coordinate range in the current viewport coordinate system. The specific process is as follows: obtain the scroll offset of the current viewport, including the horizontal scroll offset. and vertical scroll offset and the scaling ratio of the current viewport. scaleThe scaling ratio is defined by the ratio of viewport physical pixels to document global coordinate pixels; let the coordinate range of the minimum bounding rectangle in the document global coordinate system be... to Then the coordinates of the top-left corner of the smallest bounding rectangle in the transformed current viewport coordinate system are: and The coordinates of the lower right corner are and Because the global coordinate system of a webpage document originates from its absolute top-left corner, its coordinate values ​​do not change with user scrolling or window scaling; however, the coordinates required for the AI ​​agent to perform interactive operations must be located in the current screen viewport coordinate system. Therefore, by obtaining the scroll offset and scaling ratio of the current viewport, the coordinates in the document's global coordinate system are converted to coordinates in the viewport coordinate system, ensuring that the subsequently calculated interactive coordinates are locatable and executable in the current screen display state. This conversion process allows the document coordinate information stored in the semantic nodes to be correctly mapped to the actual display position on the screen under different scroll positions and scaling ratios, ensuring the availability of agent operations in various browsing states.

[0034] Specifically, based on the internal mapping table of the target semantic node, the normalized coordinate offset matching the element type identifier specified in the interaction request is retrieved; the width and height values ​​of the minimum bounding rectangle of the target semantic node after transformation to the viewport coordinate system are calculated; the horizontal component of the normalized coordinate offset is multiplied by the width value in the viewport coordinate system, and the vertical component is multiplied by the height value in the viewport coordinate system, and then the horizontal and vertical coordinates of the upper left corner of the minimum bounding rectangle in the viewport coordinate system are superimposed to obtain the estimated interaction coordinates. More specifically, when the normalized offset of the matching sub-element is retrieved from the internal mapping table, the estimated interaction coordinates are calculated by first calculating the width of the minimum bounding rectangle in the current viewport coordinate system. and high Then calculate the x-coordinate of the estimated interaction coordinates. y-axis The obtained coordinates This refers to the estimated interaction coordinates for the agent to perform precise operations. In practice, after the AI ​​agent reaches the target semantic node based on the semantic attribute graph during the path planning phase, it needs to translate the high-level intent of "initiating interaction with a specific type of element" into a specific operation position on the screen during the execution phase. This is achieved by retrieving the normalized offset matching the target type from the internal mapping table and combining it with the minimum bounding rectangle parameter already converted to the viewport coordinate system for reverse calculation. This allows for precise determination of the estimated interaction coordinates of the sub-element on the screen. Since the normalized offset is a proportional value relative to the size of the minimum bounding rectangle, multiplying it by the actual width and height in the viewport coordinate system and then adding the top-left corner coordinates restores the normalized relative position to the absolute pixel coordinates in the viewport coordinate system. This reverse calculation process has minimal computational cost and no loss of precision, enabling the AI ​​agent to obtain precise operation coordinates directly without expanding semantic nodes or re-querying the DOM. This achieves precise interaction positioning while maintaining the lightweight nature of the state representation.

[0035] Additionally, when the cumulative number of times the AI ​​agent continuously performs interactive operations on the same estimated interactive coordinates of the same semantic node without triggering the expected page state change reaches the first preset threshold, or when the number of child elements corresponding to the same element type identifier in the internal mapping table of a semantic node exceeds the second preset threshold, the semantic node is marked as pending expansion. For semantic nodes marked as pending expansion, each child element in its corresponding element group is traversed, and an independent child node is generated for each child element. Each child node carries the original bounding box position information and element type identifier of the child element. Between each child node, directed navigation edges are established based on the relative orientation of their respective bounding box center points to form a temporary fine subgraph covering the local area. The directed navigation edges between the original semantic node and the external semantic node are transferred to the boundary child node in the temporary fine subgraph that matches the external orientation. When the AI ​​agent completes the interactive task in the area covered by the temporary fine subgraph or receives the area leave instruction, all child nodes and corresponding directed navigation edges in the temporary fine subgraph are deleted, and the original semantic node and its original directed navigation edges are restored. In practical implementation, under normal circumstances, AI agents can complete accurate interactions by relying on the aggregated representation of semantic nodes and the coordinate inverse calculation of the internal mapping table. However, in some extreme scenarios, such as when high-density similar elements are arranged side by side (e.g., a large number of icons and buttons of the same type are closely arranged in a grid layout), or when the target element size is too small, resulting in insufficient accuracy of the normalized offset inverse calculation to distinguish neighboring individuals, simply relying on the internal mapping table may not meet the interaction accuracy requirements. Therefore, by setting thresholds for the number of consecutive interaction failures and the number of similar elements within a group as the conditions for triggering granularity switching, the semantic nodes can be automatically expanded into a group of child nodes carrying the original bounding box information when the conventional mechanism cannot meet the accuracy requirements. A temporary fine-grained subgraph is then constructed using consistent graph-building rules for the agent to perform finer-grained operations. After the local interaction task is completed, the child nodes are automatically recycled and the original semantic nodes are restored. This allows the semantic attribute graph to maintain the lightweight advantage of coarse granularity in most scenarios, while providing fine-grained information support as needed when high-precision operations are required in certain localities. This dynamic adjustment mechanism adapts the level of abstraction of the state representation to the dynamic needs of the actual interaction scenario, ensuring efficiency in normal scenarios while taking into account reliability in complex scenarios.

[0036] In summary, this invention acquires the bounding box position information and element type identifiers of interactive elements and performs clustering. This integrates scattered individual elements into coarse-grained semantic units, reducing the scale of state representation data. Simultaneously, it constructs an internal mapping table for each semantic node, recording the normalized spatial positions of each sub-element within its group relative to its minimum bounding rectangle. This internal mapping table significantly reduces the data volume of semantic nodes compared to element-by-element serialization description schemes, thereby reducing computational overhead and response latency in the AI ​​agent's path planning phase. Furthermore, it retains the ability to accurately index the positions of internal sub-elements after aggregation, avoiding the problem of ambiguous target positioning in the interaction phase caused by completely discarding individual position information in existing aggregation schemes. Next, directed navigation edges with orientation attributes are established based on the relative positional relationships between the minimum bounding rectangles of each semantic node, forming a semantic attribute graph. This provides the AI ​​agent with a clearly structured state space topology view, enabling efficient path planning at the coarse-grained semantic level. When the agent initiates an interaction request for a specific type of element, it transforms the coordinate range of the minimum bounding rectangle to the viewport coordinate system based on the current viewport parameters, and then calculates and estimates the interaction coordinates by combining the normalized position information of the corresponding type in the internal mapping table. This allows the high-level path planning result to be accurately mapped to the actual operation position on the screen. Therefore, this invention solves the problem in the prior art of lacking an efficient state representation method for web front-end AI agents that can improve the accuracy of agent interaction while ensuring the simplicity of state representation.

[0037] Example 2 Please see Figure 2 The diagram shown is a structural block diagram of the efficient state representation system for web front-end AI agents proposed in the second embodiment of the present invention. This efficient state representation system 200 for web front-end AI agents includes: a data acquisition and classification module 21, a semantic node construction module 22, a semantic attribute graph determination module 23, and a predicted interaction coordinate determination module 24, wherein: The data acquisition and classification module 21 is used to acquire the bounding box position information and corresponding element type identifier of the interactive elements in the document global coordinate system of the target webpage, and to cluster the interactive elements according to the bounding box position information, so as to group the spatially adjacent and non-overlapping interactive elements into the same element group. The semantic node construction module 22 is used to generate a semantic node for each element group. The semantic node contains the coordinate range of the minimum bounding rectangle that can cover all bounding boxes in the corresponding element group, and an internal mapping table that determines the relative positional relationship between each sub-element in the group and the minimum bounding rectangle. The semantic attribute graph determination module 23 is used to establish directed navigation edges with orientation attributes between semantic nodes based on the relative positional relationship of the minimum bounding rectangles corresponding to each semantic node in the document global coordinate system. The semantic attribute graph is composed of all semantic nodes and directed navigation edges. The estimated interaction coordinate determination module 24 is used to convert the coordinate range of the minimum bounding rectangle of the target semantic node into the coordinate range under the viewport coordinate system based on the current viewport parameters when the AI ​​agent initiates an interaction request to a specific type element within the target semantic node according to the semantic attribute graph, and to determine the estimated interaction coordinates of the type-matching sub-elements under the viewport coordinate system according to the internal mapping table of the target semantic node, so as to enable the agent to perform precise operations.

[0038] Example 3 In another aspect, the present invention also proposes an electronic device, please refer to [link to relevant documentation]. Figure 3 The diagram shows an electronic device according to the third embodiment of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, it implements the efficient state representation method for web front-end AI agents as described above.

[0039] In some embodiments, the processor 10 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip, used to run program code stored in memory 20 or process data, such as executing access restriction programs.

[0040] The memory 20 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 20 can include both internal and external storage units of the electronic device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or will be output.

[0041] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0042] This invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the efficient state representation method for web front-end AI agents as described above.

[0043] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0044] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0045] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0046] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0047] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A method for efficient state representation of AI agents in web front-end, characterized in that, The method includes: Obtain the bounding box position information and corresponding element type identifier of the interactive elements in the document global coordinate system of the target webpage, and cluster the interactive elements according to the bounding box position information to group the spatially adjacent and non-overlapping interactive elements into the same element group. A semantic node is generated for each element group. The semantic node contains the coordinate range of the minimum bounding rectangle that can cover all bounding boxes in the corresponding element group, and an internal mapping table that determines the relative positional relationship between each sub-element in the group and the minimum bounding rectangle. Based on the relative positional relationship of the minimum bounding rectangle corresponding to each semantic node in the document's global coordinate system, directed navigation edges with orientation attributes are established between semantic nodes, and a semantic attribute graph is formed by all semantic nodes and directed navigation edges. When the AI ​​agent initiates an interaction request to a specific type of element within the target semantic node based on the semantic attribute graph, it converts the coordinate range of the minimum bounding rectangle of the target semantic node into a coordinate range in the viewport coordinate system based on the current viewport parameters, and determines the estimated interaction coordinates of the type-matching sub-elements in the viewport coordinate system according to the internal mapping table of the target semantic node, so that the agent can perform precise operations.

2. The efficient state representation method for webpage front-end AI agents according to claim 1, characterized in that, The steps to obtain the bounding box position information and corresponding element type identifier of interactive elements in the document global coordinate system of the target webpage include: Traverse the document object model tree of the target webpage, starting from the root node and performing a depth-first traversal of each element node. Based on the tag name of the element node and the style attribute value finally calculated by the Cascading Style Sheets, determine interactivity and filter out all interactive element nodes. For each selected interactive element node, the layout application interface provided by the browser rendering engine is called to obtain the geometric bounding box object of the element node in the document global coordinate system. The coordinates of the top left and bottom right corners are read from the geometric bounding box object to form the bounding box position information of the interactive element. Obtain the tag name of each interactive element node in the document object model tree, map the tag name to a unique type in a preset element type classification system, use the mapped type as the element type identifier of the interactive element, and bind the bounding box position information with the corresponding element type identifier to form a set of elements to be processed with type identifiers.

3. The efficient state representation method for webpage front-end AI agents according to claim 2, characterized in that, The step of clustering the interactive elements according to the bounding box position information to group spatially adjacent and non-overlapping exchangeable elements into the same element group includes: For any two interactive elements in the set of elements to be processed, calculate the straight-line pixel distance between the center points of their bounding boxes and the degree of overlap between their bounding boxes; the center point of the bounding box is obtained by taking the average of the coordinates of the upper left corner and the lower right corner of the bounding box, and the degree of overlap is measured by the ratio of the area of ​​the intersection region of the two bounding boxes to the area of ​​the total region after merging. Two interactive elements whose straight-line pixel distance is less than a preset distance threshold and whose overlap is zero are grouped into the same element group. All elements in the set of elements to be processed are traversed to obtain several element groups composed of spatially adjacent and visually non-overlapping interactive elements.

4. The efficient state representation method for webpage front-end AI agents according to claim 3, characterized in that, The steps of generating a semantic node for each group of elements and determining the internal mapping table of the relative positional relationship between each child element in the group and the minimum bounding rectangle include: For each element group, calculate the coordinate range of the smallest bounding rectangle that can enclose all bounding boxes in the element group in the document global coordinate system, and write the coordinate range as the first attribute of the semantic node into the semantic node. For each sub-element in the element group, with the top left corner of the minimum bounding rectangle as the origin and the width and height of the minimum bounding rectangle as the normalization reference, calculate the normalized coordinate offset of the center point of the bounding box of the sub-element relative to the top left corner of the minimum bounding rectangle. Using the element type identifier of each sub-element in the element group as the key and the normalized coordinate offset corresponding to the sub-element as the value, construct the internal mapping table in the form of key-value pairs. When the same element type identifier corresponds to multiple sub-elements, store a list consisting of multiple normalized coordinate offsets under the key. The internal mapping table is bound to and stored as a second attribute of the semantic node.

5. The efficient state representation method for webpage front-end AI agents according to claim 1, characterized in that, Based on the relative positions of the minimum bounding rectangles corresponding to each semantic node in the document's global coordinate system, the steps for establishing directed navigation edges with orientation attributes between semantic nodes, and constructing a semantic attribute graph from all semantic nodes and directed navigation edges, include: For each pair of semantic nodes, obtain the coordinates of the center point of their smallest bounding rectangle, and calculate the horizontal and vertical offsets of the center point of the preceding semantic node pointing to the center point of the following semantic node. The absolute values ​​of the lateral offset and the longitudinal offset are compared to map the relative offset to one of the four main directions. If the absolute value of the lateral offset is greater than or equal to the absolute value of the longitudinal offset, the direction is determined according to the sign of the lateral offset: a positive lateral offset indicates the right direction, and a negative lateral offset indicates the left direction. If the absolute value of the lateral offset is less than the absolute value of the longitudinal offset, the direction is determined according to the sign of the longitudinal offset: a positive longitudinal offset indicates the down direction, and a negative longitudinal offset indicates the up direction. Based on the determined orientation, a one-way directed navigation edge is established between the two semantic nodes, pointing from the previous semantic node to the next semantic node, and the directed navigation edge is associated with and stored with the corresponding orientation attribute. All semantic nodes and all generated directed navigation edges are aggregated into the semantic attribute graph.

6. The efficient state representation method for webpage front-end AI agents according to claim 1, characterized in that, The step of determining the estimated interaction coordinates of type-matching sub-elements in the viewport coordinate system based on the internal mapping table of the target semantic node includes: Based on the internal mapping table of the target semantic node, retrieve the normalized coordinate offset that matches the element type identifier specified in the interaction request; Calculate the width and height of the minimum bounding rectangle of the target semantic node after transformation to the viewport coordinate system; The predicted interactive coordinates are obtained by multiplying the horizontal component of the normalized coordinate offset by the width value in the viewport coordinate system, multiplying the vertical component by the height value in the viewport coordinate system, and then superimposing the horizontal and vertical coordinates of the top left corner of the smallest bounding rectangle in the viewport coordinate system.

7. The efficient state representation method for webpage front-end AI agents according to claim 6, characterized in that, It also includes a step for dynamic granularity adjustment of semantic nodes: When the cumulative number of times the AI ​​agent continuously performs interactive operations on the same estimated interactive coordinates of the same semantic node without triggering the expected page state change reaches the first preset threshold, or when the number of child elements corresponding to the same element type identifier in the internal mapping table of a semantic node exceeds the second preset threshold, the semantic node is marked as a state to be expanded. For a semantic node marked as to be expanded, traverse each child element in its corresponding element group and generate an independent child node for each child element. Each child node carries the original bounding box position information and element type identifier of the child element. Between each child node, a directed navigation edge is established based on the relative orientation of the center point of each bounding box to form a temporary fine subgraph covering the local area. The directed navigation edge between the original semantic node and the external semantic node is transferred to the boundary child node in the temporary fine subgraph that matches the external orientation. When it is detected that the AI ​​agent has completed the interaction task of the area covered by the temporary fine subgraph or received the area leave instruction, all child nodes and corresponding directed navigation edges in the temporary fine subgraph are deleted, and the original semantic nodes and their original directed navigation edges are restored.

8. A high-efficiency state representation system for web front-end AI agents, characterized in that, The system for implementing the efficient state representation method for web front-end AI agents according to any one of claims 1 to 7, the system comprising: The data acquisition and classification module is used to acquire the bounding box position information and corresponding element type identifier of interactive elements in the document global coordinate system of the target webpage, and to cluster the interactive elements according to the bounding box position information, so as to group the spatially adjacent and non-overlapping interactive elements into the same element group. A semantic node construction module is used to generate a semantic node for each of the element groups. The semantic node contains the coordinate range of the minimum bounding rectangle that can cover all bounding boxes in the corresponding element group, and an internal mapping table that determines the relative positional relationship between each child element in the group and the minimum bounding rectangle. The semantic attribute graph determination module is used to establish directed navigation edges with orientation attributes between semantic nodes based on the relative positional relationship of the minimum bounding rectangles corresponding to each semantic node in the document's global coordinate system. The semantic attribute graph is composed of all semantic nodes and directed navigation edges. The estimated interaction coordinate determination module is used to convert the coordinate range of the minimum bounding rectangle of the target semantic node into the coordinate range under the viewport coordinate system based on the current viewport parameters when the AI ​​agent initiates an interaction request to a specific type element within the target semantic node according to the semantic attribute graph, and to determine the estimated interaction coordinates of the type-matching sub-elements under the viewport coordinate system according to the internal mapping table of the target semantic node, so that the agent can perform precise operations.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the efficient state representation method for web front-end AI agents as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the efficient state representation method for web front-end AI agents as described in any one of claims 1-7.

Images