A multi-level hybrid execution method and system for Web Agents
Patent Information
- Application Number
- CN202610940036.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-26
- Publication Date
- 2026-10-20
AI Technical Summary
然而，在面对仅由矢量图形组态且无稳定文本绑定的动态界面时，上述现有技术仍会因坐标封装混乱、结构无规律漂移等因素，导致定位失效并引发任务中断
本发明通过多层控制架构，将常规操作维持在低开销层级，并能在动态界面元素定位失效时，借助第三层级获取的全局DOM快照与结构残差，结合基于空间距离的图匹配计算，恢复对目标元素的定位，减少了因动态渲染或坐标封装导致的任务中断情况。该方法能够适应老旧系统或非标准前端框架下的自动化场景，有助于提高任务执行的完成率与连续性。共享的任务状态机使定位修复结果可跨层级复用，避免了重复试错，减少了上下文丢失风险。整体上，在兼顾执行效率的同时，增强了Web Agent对复杂界面环境的适应能力。
Smart Images
Figure FT_1 
Figure FT_2 
Figure SMS_8
Abstract
Description
Technical Field
[0001] This invention belongs to the field of power automation technology, and in particular relates to a multi-level hybrid execution method and system for Web Agent. Background Technology
[0002] A Web Agent, or web page automation agent, is a software entity capable of autonomously understanding task objectives and driving the browser to complete a sequence of web page operations. During its operation, the DOM (Document Object Model) forms the core interface for programmatic access to the web page structure. It organizes web page elements into a tree structure of nodes, and the agent performs interactions such as clicks and input by locating specific nodes. An element locator is a path expression used to locate a target node in the DOM tree; common forms include XPath and CSS selectors. In practical web page automation technology, the Chrome DevTools Protocol (CDP) is a widely used low-level debugging protocol. Mainstream automation frameworks such as Puppeteer and Selenium can use this protocol to obtain snapshots of the page's DOM structure, execute scripts, or simulate interactive operations.
[0003] In enterprise-oriented automated inspection and process management scenarios, many long-established business systems, such as dispatching or monitoring systems in the power and railway sectors, still heavily rely on dynamic SVG (Scalable Vector Graphics) or rendering methods based on early graphics libraries for their front-end interfaces. The wiring diagrams, single-line diagrams, and other visual components in these interfaces are entirely generated in real-time on the client-side using JavaScript. Their DOM structures lack fixed IDs or text attributes that can be directly referenced, and the hierarchical relationships and attribute values of nodes change with each page load or interaction.
[0004] Currently, the element location methods widely used in web automation mainly rely on toolchains based on static selectors. The working mechanism of these existing technologies is as follows: operators pre-write or manually write XPath or CSS selector expressions pointing to the target element in the task script. After the page loads, the automation execution engine traverses the DOM tree according to this expression and returns the matching node to complete the operation. When the selector fails due to dynamic changes in node attributes, conventional solutions include using fuzzy matching with text content or relative position, or dynamically searching for the target node at runtime by injecting JavaScript. However, when facing dynamic interfaces configured only with vector graphics and without stable text bindings, the above-mentioned existing technologies still suffer from location failures and task interruptions due to factors such as chaotic coordinate encapsulation and irregular structural drift. Therefore, the following solutions are proposed to address these problems. Summary of the Invention
[0005] The purpose of this invention is to provide a multi-level hybrid execution method and system for Web Agents. By using a graph matching location recovery mechanism based on the residual of failed instructions and the spatial constraints of DOM snapshots, when the location at the first and second levels fails, graph matching calculations are performed using the global DOM snapshot and the text and spatial clues left over from the failure to re-lock the target element. The location results are reused across levels, which solves the problem of location failure and task interruption caused by missing element identifiers and structural drift in existing web automation solutions when facing dynamic SVG or graphical interfaces.
[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution: This invention relates to a multi-level hybrid execution method for Web Agents, comprising the following steps: At the first level, a task is received and a high-order semantic instruction stream is generated, which contains a sequence of atomic operations and a description of the target element. The high-order semantic instruction stream is then executed by the browser executor within the first level. When it is detected that the execution of the instruction targeting the target element in the higher-order semantic instruction stream is blocked, the system adaptively routes to the second level, dynamically constructs and injects a customized script, attempts to locate the target element by traversing the document object model and returns the result; When the result returned by the second level is determined to be unusable for generating a valid operation, the system triggers the privilege escalation mechanism to enter the third level, obtains a snapshot of the global document object model of the current page through the browser debugging protocol, and constructs a structural residual by combining the invalid result returned by the second level. In the third level, a spatiotemporal constraint graph matching calculation is performed based on the structural residual and the global document object model snapshot to redetermine a target node that matches the operation intention in the snapshot and generate a corresponding precise locator. The precise locator is reconstructed into a high-order semantic instruction of the first level, and the execution control is degraded back to the first level to complete the location and operation of the target element.
[0007] Furthermore, in the third level, a spatiotemporal constraint graph matching calculation is performed based on the structural residual and the global document object model snapshot, specifically including: By utilizing the text content features of the target elements contained in the structural residual, a set of candidate nodes is selected from the global document object model snapshot; For each candidate node in the candidate node set, its precise coordinates in the viewport coordinate system are obtained through the browser debugging protocol; Define a matching cost function to quantify the degree of matching of each candidate node with respect to the operation intention, wherein the matching cost function integrates at least a spatial distance metric and a saliency metric of the candidate node in the population; Calculate the matching cost for each candidate node, and select the candidate node with the lowest matching cost as the target node that matches the operation intention.
[0008] Furthermore, defining the matching cost function and calculating the matching cost for each candidate node further includes: Obtain the reference coordinates left on the viewport by the historical interaction operations as spatial anchor points; The spatial distance metric is determined by calculating the Euclidean distance between the precise coordinates of the candidate node and the spatial anchor point.
[0009] Furthermore, the salience measure of the candidate node in the population is determined by calculating the average distance between the target candidate node and all other candidate nodes in the candidate node set.
[0010] Furthermore, the matching cost function is defined as: ; In the formula, Candidate nodes The matching cost function value; Candidate nodes; As a balance factor; This refers to the node deviation calculated based on spatial anchor points. This is the node isolation degree calculated based on the candidate node set.
[0011] Furthermore, the step of determining that the result returned by the second level cannot be used to generate a valid operation specifically includes: The coordinates of the target element returned by the customized script are determined to be such that, due to coordinate transformation or offset, they cannot be directly mapped into a valid area within the browser viewport where a clickable operation can be performed.
[0012] A multi-level hybrid execution system for Web Agents, comprising: The first-level execution module is used to receive tasks and generate a high-order semantic instruction stream, which contains atomic operation sequences and target element descriptions, and drives the browser executor to execute the high-order semantic instruction stream. The second-level execution module is used to dynamically construct and inject a customized script when it is detected that the execution of the instruction for the target element in the higher-order semantic instruction stream is blocked, and to attempt to locate the target element by traversing the document object model and return its result. The third-level execution module is used to obtain a snapshot of the global document object model of the current page through the browser debugging protocol when the result returned by the second level is determined to be unusable for generating a valid operation. It then constructs a structural residual based on the invalid result returned by the second level, and performs a spatiotemporal constraint graph matching calculation based on the structural residual and the global document object model snapshot. This is to re-determine a target node that matches the operation intention in the snapshot and generate a corresponding precise locator. The hierarchical scheduling and recovery module is used to reconstruct the precise locator into high-order semantic instructions of the first level, and to degrade the execution control from the third-level execution module back to the first-level execution module.
[0013] Furthermore, the third-level execution module includes: The candidate node filtering unit is used to filter out a set of candidate nodes in the global document object model snapshot by utilizing the text content features of the target elements contained in the structural residual. The node coordinate acquisition unit is used to acquire the precise coordinates of each candidate node in the candidate node set in the viewport coordinate system through the browser debugging protocol. The matching calculation unit is used to define a matching cost function, quantify the degree of matching of each candidate node with the operation intention, calculate the matching cost of each candidate node, and select the candidate node with the minimum matching cost as the target node matching the operation intention. The matching cost function incorporates at least a spatial distance metric and a salience metric of candidate nodes within the population.
[0014] Furthermore, the matching calculation unit is further used for: The reference coordinates left on the viewport by the historical interaction operations are obtained as spatial anchors, and the spatial distance metric is determined by calculating the Euclidean distance between the precise coordinates of the candidate node and the spatial anchor. The salience measure of a candidate node in the population is determined by calculating the average distance between the target candidate node and all other candidate nodes in the candidate node set.
[0015] Furthermore, the execution system also includes a unified task state machine for sharing and broadcasting session identifiers, authentication tokens, and document object model mapping tables among the first-level execution module, the second-level execution module, and the third-level execution module, so as to maintain the continuity of the task context during execution level transitions.
[0016] The present invention has the following beneficial effects: This invention employs a multi-layered control architecture to maintain routine operations at a low-overhead level. Even when the location of dynamic UI elements fails, it can recover the location of the target element by leveraging a global DOM snapshot and structural residual obtained from the third layer, combined with graph matching calculations based on spatial distance. This reduces task interruptions caused by dynamic rendering or coordinate encapsulation. This method is adaptable to automation scenarios in legacy systems or non-standard front-end frameworks, contributing to improved task completion rates and continuity. The shared task state machine allows for the reuse of location repair results across layers, avoiding repeated trial and error and reducing the risk of context loss. Overall, while maintaining execution efficiency, it enhances the Web Agent's adaptability to complex UI environments.
[0017] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating a multi-level hybrid execution method for Web Agents according to the present invention. Figure 2 This is a schematic diagram of the structure of a multi-level hybrid execution system for Web Agents according to the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Please see Figure 1 As shown, this invention is a multi-level hybrid execution method for Web Agents, and the execution method includes the following steps: At the first level, the task is received and a high-order semantic instruction stream is generated. The high-order semantic instruction stream contains a sequence of atomic operations and a description of the target element. The high-order semantic instruction stream is executed by the browser executor within the first level. When the execution of instructions targeting a target element is blocked in the higher-order semantic instruction stream, the system adaptively routes to the second level, dynamically constructs and injects a customized script, attempts to locate the target element by traversing the document object model and returns the result; When the result returned by the second level is determined to be unusable for generating a valid operation, the system triggers the privilege escalation mechanism to enter the third level, obtains a snapshot of the global document object model of the current page through the browser debugging protocol, and constructs a structural residual by combining the invalid result returned by the second level. In the third level, a spatiotemporal constraint graph matching calculation is performed based on the structural residual and the global document object model snapshot to redetermine a target node that matches the operation intention in the snapshot and generate a corresponding precise locator. The precise locator is reconstructed into a high-level semantic instruction at the first level, and execution control is degraded back to the first level to complete the location and operation of the target element.
[0022] In the third level, a spatiotemporal constraint graph matching calculation is performed based on the structural residuals and the global document object model snapshot, specifically including: By utilizing the text content features of the target elements contained in the structural residuals, a set of candidate nodes is selected from the global document object model snapshot; For each candidate node in the candidate node set, obtain its precise coordinates in the viewport coordinate system through the browser debugging protocol; Define a matching cost function to quantify the degree of matching of each candidate node with respect to the operation intention, wherein the matching cost function integrates at least the spatial distance metric and the salience metric of the candidate node in the population; Calculate the matching cost for each candidate node, and select the candidate node with the lowest matching cost as the target node that matches the operation intention.
[0023] Define a matching cost function and calculate the matching cost for each candidate node, further including: Obtain the reference coordinates left on the viewport by the historical interaction operations as spatial anchor points; Spatial distance metrics are determined by calculating the Euclidean distance between the precise coordinates of the candidate node and the spatial anchor point.
[0024] The salience of a candidate node in the population is measured by calculating the average distance between the target candidate node and all other candidate nodes in the candidate node set.
[0025] The matching cost function is defined as:
[0026] In the formula, Candidate nodes The matching cost function value; Candidate nodes; As a balance factor; This refers to the node deviation calculated based on spatial anchor points. This is the node isolation degree calculated based on the candidate node set.
[0027] When the result returned by the second level is determined to be unusable for generating a valid operation, specifically including: The coordinates of the target element returned by the customized script are determined. Due to coordinate transformation or offset, it cannot be directly mapped into a valid area for clickable operations within the browser viewport.
[0028] Please see Figure 2 As shown, this invention is a multi-level hybrid execution system for Web Agents, comprising: The first-level execution module is used to receive tasks and generate a high-order semantic instruction stream. The high-order semantic instruction stream contains atomic operation sequences and target element descriptions, and drives the browser executor to execute the high-order semantic instruction stream. The second-level execution module is used to dynamically construct and inject a customized script when it is detected that the execution of the instruction for the target element in the higher-order semantic instruction stream is blocked. The script attempts to locate the target element by traversing the document object model and returns the result. The third-level execution module is used to obtain a snapshot of the global document object model of the current page through the browser debugging protocol when the result returned by the second level is determined to be unusable for generating a valid operation. It then constructs a structural residual based on the invalid result returned by the second level, and performs a spatiotemporal constraint graph matching calculation based on the structural residual and the global document object model snapshot. This allows it to redetermine a target node that matches the operation intent in the snapshot and generate a corresponding precise locator. The hierarchical scheduling and recovery module is used to reconstruct the precise locator into high-order semantic instructions at the first level and to degrade execution control from the third-level execution module back to the first-level execution module.
[0029] The third-level execution module includes: The candidate node filtering unit is used to filter out a set of candidate nodes in the global document object model snapshot by utilizing the text content features of the target elements contained in the structural residual. The node coordinate acquisition unit is used to obtain the precise coordinates of each candidate node in the candidate node set in the viewport coordinate system through the browser debugging protocol. The matching calculation unit is used to define a matching cost function, quantify the degree of matching of each candidate node with the operation intention, calculate the matching cost of each candidate node, and select the candidate node with the minimum matching cost as the target node matching the operation intention. Among them, the matching cost function integrates at least the spatial distance metric and the salience metric of the candidate node in the population.
[0030] The matching calculation unit is further used for: The reference coordinates left on the viewport by historical interaction operations are obtained as spatial anchors, and the spatial distance metric is determined by calculating the Euclidean distance between the precise coordinates of the candidate node and the spatial anchor. The significance of a candidate node in the population is determined by calculating the average distance between the target candidate node and all other candidate nodes in the candidate node set.
[0031] The execution system also includes a unified task state machine, which is used to share and broadcast session identifiers, authentication tokens, and document object model mapping tables among the first-level execution modules, the second-level execution modules, and the third-level execution modules, in order to maintain the continuity of the task context when transitioning between execution levels.
[0032] The specific application of this embodiment is as follows: Step S1: Task initialization and first-level instruction stream generation
[0033] The system receives a high-level task objective, such as "Inspect the single-line diagram of the 110kV outgoing line bay of the ×× substation, record the switch status, and save the screenshot." The core decision module (large model) of the Agent first decomposes the task into a standard, indivisible sequence of atomic operations, generating a first-level high-order semantic instruction stream. The instruction set is defined as follows: click [target element description], fill [target element description] "input content", screenshot, wait [milliseconds]. Among them, [target element description] is initially generated by the large model based on the general knowledge of the task context and the target system, such as click "110kV outgoing line bay". The system executes the instruction stream by default using the first-level scheduling strategy. The instruction stream is serialized into lightweight text and sent to the browser's automated executor. This process has extremely low inference cost and a response speed of milliseconds.
[0034] Step S2: First-level execution and element location failure detection
[0035] The browser executor parses the instruction stream; when it executes the click "110kV outgoing line interval", the executor's internal positioning module attempts to locate the element in the current DOM tree; the positioning strategies are, in order: exact text matching, fuzzy text matching, and CSS selector matching; in the old OMS SVG single-line graph embodiment, the target text "110kV outgoing line interval" is not an independent DOM node, but rather as a certain <text>The dynamic properties of an element are written by a script, and are updated every time the page renders its parent element. <g>The transform property value of an element is generated randomly; therefore, after traversing the DOM tree, the positioning module reports a positioning failure; the system detects that the execution of the first-level instruction is blocked and triggers the inter-level adaptive routing mechanism.
[0036] Step S3: Second-level tentative localization and script injection failure
[0037] The system automatically elevates the task to the second level: the page-level dynamic script layer. Based on the current failure context, the agent's core decision-making module dynamically constructs a customized JavaScript script, intending to find the target through more complex DOM traversal logic. For example, the script logic might be: traverse all... <text>The node checks if its textContent property contains "110kV outgoing line interval" and returns the bounding box of its parent component; the script evaluates the channel injection and executes it at browser runtime. However, because the OMS system's single-line graph uses a highly encapsulated SVG graphics library, which internally maintains a virtual DOM and data binding, after the injected external script executes, although it finds the text node, the returned parent component bounding box coordinates are the relative coordinates of that component within the SVG's internal coordinate system. Furthermore, due to multiple layers... <g>Due to the nesting of tags and complex matrix transformations, there is an offset between these coordinates and the actual click coordinates in the browser viewport that cannot be directly calculated; the script returns a coordinate region that theoretically exists but cannot be physically hit by an external click. The system determined that the second-level execution result was invalid and could not generate a reliable click operation, thus the execution was blocked again.
[0038] Step S4: Graph matching localization and recovery based on residual and snapshot spatiotemporal constraints
[0039] At this point, the system determines that it has encountered an element positioning deadlock caused by dynamic rendering and coordinate encapsulation, triggering the privilege escalation mechanism and entering the third level: the system-level external collaboration layer.
[0040] Step S41: Obtain the global DOM snapshot and structural residual: The system calls the DOM.getDocument command via the CDP protocol to obtain a complete, frozen DOM tree snapshot of the current page. This snapshot is a structured JSON object containing the type, attributes, unique backend node IDs, and parent-child relationships of all nodes, reflecting the absolute true structure of the page at that moment. Simultaneously, the system defines the invalid "result" returned by the second-level script in step S3 as a structural residual. In this case, the residual R contains: the text content of the target element. "110kV outgoing line bay", the label name of this text node. <text>and meaningless relative coordinates returned by the script. .
[0041] Step S42: Construct a spatial candidate node set: In the obtained CDP DOM snapshot, the system performs a breadth-first or depth-first traversal to filter out all nodes of type [missing information]. <text>And its node value (accessed via DOM.getOuterHTML or by directly accessing node properties in the snapshot) contains the target text. The nodes constitute the candidate text node set. This step utilizes the strong constraint of text content to quickly narrow down the search scope.
[0042] Step S43: Perform graph matching calculation based on spatial distance: for the set Each candidate node in The system uses the DOM.getBoxModel command of CDP, specifying its backend node ID, to obtain the precise, clickable center coordinates of the BoxModel in the browser viewport coordinate system. Subsequently, the system needs to calculate a decision score to select the real target operation node from multiple text nodes with the same name. Due to the failure of relative coordinates with viewport coordinates Although they are not in the same coordinate system and cannot be directly compared, this embodiment utilizes the following spatial constraints: In visualization interfaces such as single-line graphs, the graphic elements (composed of shapes and lines) representing the same device (such as outgoing line bays) and their text labels ("110kV outgoing line bays") are spatially tightly coupled and adjacent; although the absolute position of the text nodes is unknown, this embodiment can find a reference anchor point; the system traces back to the moment the first-level failed instruction click "110kV outgoing line bay" was executed and finds that the Agent attempted to click at a certain old coordinate or default area, but the click did not produce the expected page jump; this embodiment records the coordinates of this click on the viewport as... Under certain conditions, this embodiment can also use the estimated region within a viewport defined by the invalid bounding box returned by the second-level script. Define the matching cost function Used to measure candidate nodes The degree of deviation from the user's intent; this embodiment uses a metric that combines spatial distance and visual salience: ; In the formula, Candidate nodes The matching cost function value, which comprehensively reflects The degree of deviation from the expected operational objectives in terms of spatial location and candidate population; For the first candidate text node in the set Each node represents a potential target element to be evaluated. The balancing factor is a preset constant with a value range of [value range missing]. This is used to adjust the weight between the historical click spatial location constraint and the node spatial uniqueness constraint. In this embodiment, The value is 0.8; Candidate nodes The coordinates of the center point in the browser viewport coordinate system are a two-dimensional coordinate vector. This is obtained by calling the DOM.getBoxModel command in CDP, representing the precise clickable position of the node on the screen; The viewport click coordinates recorded at the moment of first-level execution failure are a two-dimensional coordinate vector. , serving as a spatial historical clue to operational intent; For vectors The norm, or Euclidean distance between two coordinate points, is used to measure the difference in spatial location. For the set of candidate text nodes Except for the current evaluation node Other candidate nodes The coordinates of the center point are also two-dimensional coordinate vectors; The set of candidate text nodes consists of all nodes of type in the DOM snapshot. <text>And the node value contains the target text. The nodes constitute; For the set of candidate text nodes The cardinality, that is, the total number of candidate nodes in the set; This is an index variable used in the summation process to iterate through the candidate node set, excluding... Other nodes besides; In this embodiment, due to erroneous click points It is a reliable spatial clue left over from system operation. This embodiment will... Set to a relatively high value of 0.8; the first part of the formula calculates the spatial proximity of the candidate node to the historical operation focus, and the second part calculates the average distance between the candidate node and all other candidate nodes, representing its significance as a unique target rather than group background noise; nodes that are far away and not isolated are costly.
[0043] Step S44, Target Localization and Command Reconstruction: The system traverses and calculates all... Based on the Cost value, select the candidate node with the lowest cost. The target element that is successfully matched; the calculation formula is: ; In the formula, The optimal candidate node selected by the algorithm, i.e. the node with the lowest cost, is ultimately determined to be the real target element that matches the operation intention. The mathematical symbol represents taking the value of the independent variable that minimizes the objective function; in this case, it represents the cost function. Minimum candidate node ; As a constraint, it indicates Taken from the candidate text node set ; Once found The system obtains its unique and stable XPath path or CSS selector chain in the DOM through CDP, for example... At this point, the location recovery is complete; the system sends this precise selector back to the first-level instruction generator to reconstruct the original instruction as click [xpath='...'].
[0044] Step S5: Degradation Execution and Status Broadcast Update
[0045] After successful location recovery, the system performs a fallback operation, returning control from the third level to the first level. The browser executor receives and executes the reconstructed first-level instruction `click [xpath='...']`, successfully activating the corresponding switch element on the single-line graph and returning a task status of "success". The system's internal task state machine then broadcasts an update: the execution record of this instruction from failure to recovery, the reconstructed XPath locator, and the hash value of the DOM snapshot are atomically updated to the unified task context session storage. This ensures that subsequent instructions can directly reuse this corrected and accurate location, without repeating the failure process.
[0046] Step S6: Complete subsequent tasks and exit the closed loop.
[0047] Based on the accurate context updated in the state machine, the Agent smoothly executes the remaining atomic operations: executing the screenshot instruction to capture a viewport screenshot of the currently focused switch state and encode it as a Base64 stream for storage in short-term memory; subsequently, the large model decision center analyzes the screenshot, identifies the "open / closed" state of the switch, and generates structured data; after all task sequences are completed, the Agent cleans up the injected scripts, releases the specific handle occupied by the browser through CDP, and reports the task success to the host system; throughout the process, routine operations are efficiently completed in the first and second layers, and for the dynamic SVG positioning problem of the old system, autonomous penetration is achieved through the positioning recovery algorithm in the third layer, ensuring the end-to-end success rate of the inspection task.
[0048] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0049] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.< / text> < / text> < / text> < / g> < / text> < / g> < / text>
Claims
1. A multi-level hybrid execution method for Web Agents, characterized in that, The execution method includes the following steps: At the first level, a task is received and a high-order semantic instruction stream is generated, which contains a sequence of atomic operations and a description of the target element. The high-order semantic instruction stream is then executed by the browser executor within the first level. When it is detected that the execution of the instruction targeting the target element in the higher-order semantic instruction stream is blocked, the system adaptively routes to the second level, dynamically constructs and injects a customized script, attempts to locate the target element by traversing the document object model and returns the result; When the result returned by the second level is determined to be unusable for generating a valid operation, the system triggers the privilege escalation mechanism to enter the third level, obtains a snapshot of the global document object model of the current page through the browser debugging protocol, and constructs a structural residual by combining the invalid result returned by the second level. In the third level, a spatiotemporal constraint graph matching calculation is performed based on the structural residual and the global document object model snapshot to redetermine a target node that matches the operation intention in the snapshot and generate a corresponding precise locator. The precise locator is reconstructed into a high-order semantic instruction of the first level, and the execution control is degraded back to the first level to complete the location and operation of the target element.
2. The multi-level hybrid execution method for Web Agents according to claim 1, characterized in that, In the third level, a spatiotemporal constraint graph matching calculation is performed based on the structural residual and the global document object model snapshot, specifically including: By utilizing the text content features of the target elements contained in the structural residual, a set of candidate nodes is selected from the global document object model snapshot; For each candidate node in the candidate node set, its precise coordinates in the viewport coordinate system are obtained through the browser debugging protocol; Define a matching cost function to quantify the degree of matching of each candidate node with respect to the operation intention, wherein the matching cost function integrates at least a spatial distance metric and a saliency metric of the candidate node in the population; Calculate the matching cost for each candidate node, and select the candidate node with the lowest matching cost as the target node that matches the operation intention.
3. The multi-level hybrid execution method for Web Agents according to claim 2, characterized in that, The definition of the matching cost function and the calculation of the matching cost for each candidate node further include: Obtain the reference coordinates left on the viewport by the historical interaction operations as spatial anchor points; The spatial distance metric is determined by calculating the Euclidean distance between the precise coordinates of the candidate node and the spatial anchor point.
4. A multi-level hybrid execution method for Web Agents according to claim 2, characterized in that, The salience of a candidate node in the population is determined by calculating the average distance between the target candidate node and all other candidate nodes in the candidate node set.
5. A multi-level hybrid execution method for Web Agents according to claim 2, characterized in that, The matching cost function is defined as follows: In the formula, Candidate nodes The matching cost function value; Candidate nodes; As a balance factor; This refers to the node deviation calculated based on spatial anchor points. This is the node isolation degree calculated based on the candidate node set.
6. The multi-level hybrid execution method for Web Agents according to claim 1, characterized in that, When the result returned by the second level is determined to be unusable for generating a valid operation, the specific steps include: The coordinates of the target element returned by the customized script are determined to be such that, due to coordinate transformation or offset, they cannot be directly mapped into a valid area within the browser viewport where clickable operations can be performed.
7. A multi-level hybrid execution system for Web Agents, characterized in that, include: The first-level execution module is used to receive tasks and generate a high-order semantic instruction stream, which contains atomic operation sequences and target element descriptions, and drives the browser executor to execute the high-order semantic instruction stream. The second-level execution module is used to dynamically construct and inject a customized script when it is detected that the execution of the instruction for the target element in the higher-order semantic instruction stream is blocked, and to attempt to locate the target element by traversing the document object model and return its result. The third-level execution module is used to obtain a snapshot of the global document object model of the current page through the browser debugging protocol when the result returned by the second level is determined to be unusable for generating a valid operation. It then constructs a structural residual based on the invalid result returned by the second level, and performs a spatiotemporal constraint graph matching calculation based on the structural residual and the global document object model snapshot. This is to re-determine a target node that matches the operation intention in the snapshot and generate a corresponding precise locator. The hierarchical scheduling and recovery module is used to reconstruct the precise locator into high-order semantic instructions of the first level, and to degrade the execution control from the third-level execution module back to the first-level execution module.
8. A multi-level hybrid execution system for Web Agents according to claim 7, characterized in that, The third-level execution module includes: The candidate node filtering unit is used to filter out a set of candidate nodes in the global document object model snapshot by utilizing the text content features of the target elements contained in the structural residual. The node coordinate acquisition unit is used to acquire the precise coordinates of each candidate node in the candidate node set in the viewport coordinate system through the browser debugging protocol. The matching calculation unit is used to define a matching cost function, quantify the degree of matching of each candidate node with the operation intention, calculate the matching cost of each candidate node, and select the candidate node with the minimum matching cost as the target node matching the operation intention. The matching cost function incorporates at least a spatial distance metric and a salience metric of candidate nodes within the population.
9. A multi-level hybrid execution system for Web Agents according to claim 8, characterized in that, The matching calculation unit is further used for: The reference coordinates left on the viewport by the historical interaction operations are obtained as spatial anchors, and the spatial distance metric is determined by calculating the Euclidean distance between the precise coordinates of the candidate node and the spatial anchor. The salience measure of a candidate node in the population is determined by calculating the average distance between the target candidate node and all other candidate nodes in the candidate node set.
10. A multi-level hybrid execution system for Web Agents according to claim 7, characterized in that, The execution system also includes a unified task state machine, which is used to share and broadcast session identifiers, authentication tokens, and document object model mapping tables among the first-level execution modules, the second-level execution modules, and the third-level execution modules, so as to maintain the continuity of the task context when the execution level jumps.