Webpage element processing method and device, equipment and storage medium

Abstract

The application relates to the technical field of webpage automation, and discloses a webpage element processing method, device, equipment and storage medium, which comprises the following steps: acquiring identification information of a target element in a webpage; analyzing the target element based on a backend node identification, and generating a sampling point set according to an analysis result; performing a hit test based on the sampling point set, and determining an action point according to a hit test result; matching a debugging session according to a frame identification, and dispatching a trusted input event at the action point through the debugging session to process the target element. The operation position cannot be hijacked by an upper layer shielding element, the dispatched event can be normally responded by a listener that depends on real interaction of a page, deviation of a coordinate space and a session under a cross-source subframe scene is eliminated, accurate access of webpage automation operation under a complex structure is realized, and the application is especially suitable for a business page automation scene with a floating component and multi-frame nesting.

Description

Technical Field

[0001] This application relates to the field of web page automation technology, and in particular to a method, apparatus, device and storage medium for processing web page elements. Background Technology

[0002] In web page automation scenarios, existing solutions typically obtain the geometric position of the target element directly and dispatch it by constructing a composite event through the script in the page, or uniformly issue coordinate instructions through the main page session. However, when the target element is obscured by floating layers, sticky components, etc., the fixed selected geometric position actually corresponds to the upper covering element. Elements in cross-source subframes will fail due to the mismatch between the operation session and the coordinate space, ultimately causing the automated operation to fail to accurately apply to the target element, and the failure state is difficult to perceive. Summary of the Invention

[0003] The main objective of this application is to provide a web page element processing method, apparatus, device, and storage medium, aiming to solve the technical problem that automated agents cannot reliably complete click / input operations on target elements that can be recognized by the real event listener of the page on real web pages with overlays and multiple frames (including cross-source subframes).

[0004] To achieve the above objectives, this application proposes a method for processing web page elements, the method comprising: Obtain the identification information of the target element in the webpage, the identification information including the backend node identifier and the frame identifier; The target element is parsed based on the backend node identifier, and a set of sampling points is generated based on the parsing result; A hit test is performed based on the set of sampling points, and the action point is determined based on the hit test results; The debugging session is matched according to the frame identifier, and a trusted input event is dispatched at the action point through the debugging session to process the target element.

[0005] Optionally, the step of parsing the target element based on the backend node identifier and generating a sampling point set based on the parsing result includes: The target element is parsed based on the backend node identifier, and at least one bounding rectangle of the target element within the viewport is determined based on the parsing result. Multiple sampling points are generated within the circumscribed rectangle to obtain a sampling point set.

[0006] Optionally, generating multiple sampling points within the circumscribed rectangle to obtain a sampling point set includes: Obtain the geometric center point of the circumscribed rectangle; The vertices of the circumscribed rectangle are translated into the interior of the circumscribed rectangle to obtain multiple indentation points; Based on the midpoints of each side of the circumscribed rectangle, multiple boundary sampling points are determined; The geometric center point, the plurality of recessed corner points, and the plurality of boundary sampling points are combined to generate a sampling point set.

[0007] Optionally, the step of performing a hit test based on the sample point set and determining the action point based on the hit test result includes: The sampling points in the sampling point set are traversed sequentially according to the preset traversal order. For the currently traversed sampling point, perform a hit test on the root node corresponding to the target element to obtain the hit element at the sampling point position; Determine whether the hit element is the target element or a descendant node of the target element; If so, the first hit element will be designated as the action point.

[0008] Optionally, the identification information further includes path information, and after performing the hit test based on the sampling point set, it further includes: If the hit test result is that the hit element of the sampling point set is not the target element or a descendant node of the target element, then the target element is determined to be in an occluded state. In response to the target element being obscured, the number of candidate frames is obtained based on the path information in multiple frames of the webpage, and the candidate frames are used to parse the target element; If the number of candidate frames is one, then the processing operation on the target element is performed within the candidate frame; If there are multiple candidate frames, the processing operation is terminated.

[0009] Optionally, the step of matching a debugging session based on the frame identifier and dispatching a trusted input event at the action point through the debugging session to process the target element includes: The debugging session is matched based on the frame identifier, and the operation type of the operation to be performed on the target element is determined. If the operation type is an input operation, a trusted focus event is dispatched at the action point through the debugging session to focus the target element, and a trusted input event is dispatched after focusing to input content into the target element; If the operation type is a click operation, then when the target element is an editable element, a focus operation is performed on the target element through the debugging session, instead of dispatching a trusted click event at the action point through the debugging session.

[0010] Optionally, after determining the operation type to be performed on the target element, the method further includes: If the operation type is a visual coordinate operation, obtain the normalized coordinate components; Map the normalized coordinate components to viewport pixel coordinates; Trusted input events are dispatched at the viewport pixel coordinates through a debug session corresponding to the main frame to perform viewport-level operations.

[0011] Furthermore, to achieve the above objectives, this application also proposes a webpage element processing apparatus, the webpage element processing apparatus comprising: The acquisition module is used to acquire the identification information of the target element in the webpage, the identification information including the backend node identifier and the frame identifier; The parsing module is used to parse the target element based on the backend node identifier and generate a set of sampling points based on the parsing results; An execution module is used to perform a hit test based on the set of sampling points and determine the action point based on the hit test results; The processing module is configured to match a debugging session based on the frame identifier and dispatch a trusted input event at the action point through the debugging session to process the target element.

[0012] In addition, to achieve the above objectives, this application also proposes a web page element processing device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the web page element processing method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the web page element processing method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the web page element processing method described above.

[0015] This application obtains identification information to determine the action point corresponding to the actual visible area of ​​the target element based on sampling point hit test. Then, it dispatches browser native trusted input events through a dedicated debugging session that matches the frame identifier. This ensures that the operation position is not hijacked by upper-layer occluding elements, and the dispatched events can be normally responded to by the page's listeners that rely on real interactions. This eliminates the deviation between coordinate space and session mismatch in cross-source subframe scenarios, and realizes accurate access to automated web page operations in complex DOM (Original Object Model) structures. It is especially suitable for business page automation scenarios with floating components and multiple nested frames. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the webpage element processing method of this application; Figure 2 This is a schematic diagram of the hit backtesting sampling for occlusion perception in this application. Figure 3 This is a flowchart illustrating the second embodiment of the webpage element processing method of this application; Figure 4 This is a schematic diagram illustrating the frame selection for the location-free rollback method in this application. Figure 5 This is a flowchart illustrating the third embodiment of the webpage element processing method of this application; Figure 6 This is a simplified flowchart of a webpage element processing method; Figure 7 This is a schematic diagram of the module structure of the web page element processing device according to an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the web page element processing method in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] This application provides a method for processing web page elements, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the webpage element processing method of this application.

[0023] In this embodiment, the webpage element processing method includes: Step S10: Obtain the identification information of the target element in the webpage, the identification information including the backend node identifier and the frame identifier.

[0024] It should be noted that the execution entity in this embodiment is an automated proxy running in a browser. This automated proxy can be an intelligent proxy driven by a large language model, capable of interacting with the browser kernel, calling browser debugging protocol interfaces, and executing script injection. The browser runs on a general-purpose computer or server hardware device, exposing interfaces for page document manipulation, event dispatching, and status monitoring to the outside world through debugging protocols (such as the Chrome DevTools Protocol). The following description uses an automated proxy running in a browser as an example to illustrate this embodiment and the subsequent embodiments.

[0025] It should be noted that the target element can be an interactive node in the Document Object Model (DOM) tree, such as a button, input box, link, dropdown menu, checkbox, rich text editor, or custom interactive component. The target element can be the object to be operated on by the automation agent in the webpage, and the operation includes, but is not limited to, click, input, hover, or drag operations. The target element can be obtained by the automation agent based on the received operation instructions. For example, if the operation instruction is "click the login button," then the target element is the DOM node corresponding to the "login button."

[0026] It is understood that identification information is a set of data used to uniquely identify a target element within the document structure of a webpage and indicate its context. This identification information includes at least backend node identifiers and frame identifiers. Specifically, the backend node identifier is a globally unique identifier assigned by the browser kernel to each node in the Document Object Model (DOM) tree, used to uniquely represent that DOM node during a debugging session; the frame identifier is a unique identifier assigned by the browser to each browsing context (including the main frame and all child frames) within the page, used to distinguish between different frame environments.

[0027] Optionally, the identification information further includes path information, which is used to locate the target element in the document tree of the frame to which the target element belongs. The specific form of the path information may be an XPath expression or a CSS selector. The path information serves as a backup positioning method for the backend node identifier, providing auxiliary positioning capabilities when the backend node identifier is unavailable or the target element has not yet finished rendering.

[0028] Specifically, as an optional implementation, the automation agent injects a document traversal script into each frame of the webpage via the browser debugging protocol. It traverses all interactive DOM element nodes (such as buttons, input boxes, links, dropdown menus, etc.) within the current frame and assigns a unique element index within the current frame to each traversed element. Subsequently, the automation agent obtains the backend node identifier corresponding to each element with an assigned index via the debugging protocol, and simultaneously obtains the frame identifier and frame address of the frame to which the element belongs, along with the path information used to parse the target element within the frame. The backend node identifier is only meaningful to the debugging session that owns its frame.

[0029] In one example, the automated agent does not directly obtain the target element through the backend node identifier, but instead locates the target element in multiple frames of the webpage through semantic clues, as follows:

[0030] in, For the k-th candidate element, For anchor elements, For the least common ancestor, For hierarchy depth, The selected candidate number.

[0031] Among all candidate elements, the one with the smallest least common ancestor (LCA) depth with the anchor element is selected as the target element. A smaller LCA depth indicates that the common ancestor of the candidate element and the anchor element is closer to the root node in the DOM tree, meaning a weaker semantic connection. Conversely, a larger LCA depth indicates that the common ancestor of the candidate element and the anchor element is closer to themselves in the DOM tree, meaning a closer position and stronger semantic connection. Therefore, the target element is selected... The smallest candidate element is actually the element that is closest to the anchor element in the DOM tree and is most semantically relevant among all candidates.

[0032] Step S20: Parse the target element based on the backend node identifier, and generate a set of sampling points based on the parsing result.

[0033] It should be noted that the sampling point set is a collection of multiple two-dimensional coordinate points distributed within the visible area of ​​the target element. This set can be used to subsequently determine valid action points that can be manipulated through hit testing. The coordinate points are represented in viewport coordinates. The generation of the sampling point set follows a multi-point coverage principle, that is, sampling points are selected from multiple directions within the display area of ​​the target element to address scenarios where the target element is partially occluded by upper-layer elements.

[0034] Specifically, the automated proxy uses the obtained backend node identifier as a query parameter to initiate a DOM node query request to the browser kernel via the debugging protocol. In a debugging session containing the frame to which the backend node identifier belongs, the browser kernel searches for the corresponding DOM node object in the DOM node index based on the backend node identifier and returns a reference to that node object to the automated proxy. If the query is successful, the automated proxy obtains a node reference to the target element in memory, and can then read the node's layout information, style attributes, geometric position, and other rendering data.

[0035] After successfully parsing the target element and obtaining its node reference, the automated agent reads the layout information of the target element through the browser debugging protocol and uses this layout information as the parsing result to generate a set of sampling points. For example, the set of sampling points can be generated using the bounding rectangle of the target element.

[0036] In the specific implementation process, the bounding rectangle serves as the precise projection area of ​​the target element in the viewport, accurately reflecting the visible range of the target element. Therefore, generating a sampling point set based on the bounding rectangle of the target element limits the sampling range to the actual display area of ​​the target element, avoiding invalid sampling across the entire page and reducing computational overhead. At this point, step S20 may include: The target element is parsed based on the backend node identifier, and at least one bounding rectangle of the target element within the viewport is determined based on the parsing result. Multiple sampling points are generated within the circumscribed rectangle to obtain a sampling point set.

[0037] It should be noted that the outer rectangle is the smallest axis-aligned rectangle that contains all the rendered content of the target element. The outer rectangle is represented in viewport coordinates, and its data includes the x-coordinate and y-coordinate of the top-left corner, the width, and the height of the rectangle. For target elements with complex layouts (such as inline elements spanning multiple lines or elements with complex CSS shapes), the outer rectangle can be the union of multiple rectangular regions. For example, when the target element is a text node that wraps to the next line, each line of text corresponds to an independent outer rectangle.

[0038] Specifically, after successfully parsing the target element and obtaining its node reference, the automated proxy calls the browser kernel's layout query interface via the browser debugging protocol to read one or more bounding rectangles of the target element in the viewport. For example, the layout query interface returns the axis-aligned bounding box data of the target element in the viewport coordinate system. This axis-aligned bounding box data is provided in the form of a DOMRect object, including the rectangle's x-coordinate, y-coordinate, width, and height. The read bounding rectangles serve as the geometric boundaries for subsequently generating the sampling point set.

[0039] In one alternative implementation, if the target element is not currently within the viewport (e.g., the target element is outside the scrolling area), the automation agent first dispatches a scrolling event through the debugging protocol to scroll the target element into the viewport, and then reads its bounding rectangle to ensure that the obtained bounding rectangle coordinates are within the valid range of the current viewport coordinate system.

[0040] After obtaining the circumscribed rectangle, the automated agent generates multiple sampling points within the circumscribed rectangle. The sampling point generation logic involves selecting representative coordinate positions in different regions of the circumscribed rectangle, ensuring that sampling points are distributed in the central, corner, and boundary regions of the rectangle.

[0041] In practical implementation, a multi-point sampling strategy can be used to ensure that even when a target element is partially occluded by upper-layer elements (e.g., the occlusion only covers the central area or a certain edge area), effective action points can still be found at unoccluded corners or boundaries, significantly improving the fault tolerance and success rate of target element localization in partially occluded scenarios. In this case, generating multiple sampling points within the bounding rectangle to obtain a sampling point set can include: Obtain the geometric center point of the circumscribed rectangle; The vertices of the circumscribed rectangle are translated into the interior of the circumscribed rectangle to obtain multiple indentation points; Based on the midpoints of each side of the circumscribed rectangle, multiple boundary sampling points are determined; The geometric center point, the plurality of recessed corner points, and the plurality of boundary sampling points are combined to generate a sampling point set.

[0042] As an optional implementation, the sampling point set includes the following types of coordinate points: geometric center point, indentation corner point, and boundary sampling point.

[0043] The geometric center point refers to the intersection of the two diagonals of the circumscribed rectangle. As the most typical representative point of the target element, the geometric center point is the preferred operation position in the absence of obstruction.

[0044] The indentation corner points are four points obtained by translating the four vertices of the circumscribed rectangle inwards by a predetermined pixel distance. The translation direction is from the vertex to the center of the rectangle, and the predetermined distance is a preset fixed pixel value, such as 3 pixels. When the upper overlay only covers the edge area of ​​the target element's circumscribed rectangle, the indentation corner points may still be located in the visible part of the target element, thus avoiding the direct use of the covered vertex positions.

[0045] The boundary sampling points are determined based on the division points of each side of the circumscribed rectangle. For example, each side is divided into several equal parts, and the division points are taken. Specifically, for each of the four sides of the circumscribed rectangle, its geometric midpoint is taken as the boundary sampling point, resulting in four boundary sampling points. In an optional implementation, each side can also be divided into four equal parts, and each of the four division points can be taken as the boundary sampling point, resulting in twelve boundary sampling points. The purpose of the boundary sampling points is that even when the central area of ​​the rectangle is obscured by an object, it is still possible to find effective action points near the boundary.

[0046] By merging the geometric center point, multiple indentation points, and multiple boundary sampling points, and removing any potentially overlapping coordinate points, we obtain the sampling point set corresponding to the circumscribed rectangle.

[0047] Step S30: Perform a hit test based on the set of sampling points, and determine the action point based on the hit test results.

[0048] It should be noted that the hit test refers to the browser determining the topmost rendered DOM element based on the viewport coordinates.

[0049] In a browser, hit testing can be performed by calling... The method implementation takes viewport coordinates as a parameter and returns the topmost interactive element at that coordinate location (i.e., the topmost element along the z-axis). The hit test results directly reflect the target element to which the browser will dispatch events when the user actually clicks or touches.

[0050] It is understood that the action point is a valid sampling point selected from the sampling point set and confirmed by a hit test to be able to hit the target element or any descendant element of the target element.

[0051] In one example, reference Figure 2 , Figure 2This is a schematic diagram of the hit backtesting sampling for occlusion awareness in this application. Based on the backend node identifier, the target element is parsed, and one or more bounding rectangles of the target element are determined. A set of sampling points is generated on these bounding rectangles. A preferred configuration of the sampling point set is the center point of the bounding rectangle, the four corner points indented by a predetermined number of pixels (e.g., 3 pixels) from the four corners inwards from the four corners, and the quarter points. Multi-point sampling ensures that the target may still be hit even when partially occluded. A hit test is performed on the root node of the target element for each sampling point within the viewport. The first hit result, where the target element itself or its descendant elements are selected as the action point. Hit testing on the root node ensures that the target can still be hit even when it is located in the ShadowDOM. If no hit occurs, occlusion is determined, and a location-free backtest is triggered.

[0052] Step S40: Match the debugging session according to the frame identifier, and dispatch a trusted input event at the action point through the debugging session to process the target element.

[0053] It's important to note that a debug session is a communication channel established between the automation agent and the browser kernel via the browser debugging protocol. Each debug session is bound to a specific frame (browser context) and is used to perform operations such as DOM queries, event dispatching, and property reading / writing within that frame's context. Because each frame has its own independent JavaScript execution environment and DOM tree, DOM node identifiers (such as backend node identifiers) between different frames are only valid within the debug session of their respective frames.

[0054] The trusted input event is a native input event directly generated and dispatched by the browser kernel, with its `isTrusted` property set to `true`. Unlike synthetic events constructed and dispatched by scripts using the `dispatchEvent` method, trusted input events are generated by the browser kernel when it receives actual user input or simulates input through a debugging protocol. These events can be correctly identified as genuine user actions by event listeners on the page that rely on the `isTrusted` property for security verification. The processing operations corresponding to the trusted input event include, but are not limited to, click operations, input operations, hover operations, or drag operations.

[0055] Specifically, the automated proxy matches the target element's frame identifier against an established pool of debug sessions that contain the frame to which the target element belongs. Upon finding a correct debug session, it dispatches a trusted input event at the action point within that session; this dispatching process is handled by the browser kernel. After receiving the instruction, the browser kernel generates the corresponding trusted input event within the target frame's context and executes the browser's standard hit test procedure based on the action point coordinates, distributing the event to the top-level element at that coordinate location. Because the event is directly generated by the browser kernel and processed through the standard hit test and dispatch procedure, its behavior is completely consistent with real user actions.

[0056] In this embodiment, by acquiring identification information, the action point corresponding to the actual visible area of ​​the target element is determined based on the sampling point hit test. Then, the browser's native trusted input event is dispatched through a dedicated debugging session that matches the frame identifier. This ensures that the operation position is not hijacked by upper-layer occluding elements, and the dispatched event can be normally responded to by the page's listener that relies on real interaction. This eliminates the deviation between the coordinate space and the session in cross-source subframe scenarios, and realizes accurate reach of automated web page operations in complex DOM structures. It is especially suitable for business page automation scenarios with floating components and multiple nested frames.

[0057] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the webpage element processing method of this application. Based on the first embodiment described above, a second embodiment of the webpage element processing method of this application is proposed.

[0058] In the second embodiment, step S30 includes: Step S301: Traverse each sampling point in the sampling point set in a preset traversal order.

[0059] It should be noted that the preset traversal order refers to the order in which each sampling point in the sampling point set is visited one by one. When there are multiple sampling points, sorting the sampling points according to the established priority rules can ensure that the execution of the hit test has a definite sequential logic, and prioritize the sampling point with better geometric location as the potential action point.

[0060] It is understood that the sampling point set contains multiple sampling points, each represented in viewport coordinates and distributed within the bounding rectangle of the target element. The preset traversal order of the sampling points can be traversed sequentially according to the geometric center point, indentation corner point, and boundary sampling point.

[0061] Step S302: For the currently traversed sampling point, perform a hit test on the root node corresponding to the target element to obtain the hit element at the sampling point position.

[0062] It should be noted that the root node is the top-level node of the document tree or shadow document tree containing the target element. Specifically, if the target element is located in the regular Document Object Model (DOM) tree, the root node is the document object of the frame to which the target element belongs; if the target element is located inside the Shadow DOM, the root node is the shadow root node that contains the shadow document tree.

[0063] It should be understood that when the target element is inside the Shadow DOM, the regular elementFromPoint method executed on the document object cannot penetrate the Shadow boundary to detect elements inside the Shadow DOM, while executing elementFromPoint (i.e., shadowRoot.elementFromPoint) with the shadow root node as the context can correctly detect elements inside the Shadow DOM.

[0064] It should be noted that the hit element is the top-level rendered element located at the sampling point coordinates, determined by the browser's hit test method. In a browser, hit testing is typically performed by calling... The method takes viewport coordinates as a parameter and returns the DOM element at the top of the z-axis at that coordinate position. If there is no element at that coordinate position (e.g., the click position is outside the viewport), it returns null.

[0065] Step S303: Determine whether the hit element is the target element or a descendant node of the target element.

[0066] Step S304: If so, the first hit element is determined as the action point.

[0067] It should be noted that descendant nodes refer to all child nodes, grandchild nodes, and nodes at deeper levels below the target element in the DOM tree. When the hit element is any descendant node of the target element, the coordinates of the sampling point still visually fall within the rendering area of ​​the target element, because the rendering area of ​​the target element is composed of the rendering areas of all its descendant nodes.

[0068] Specifically, the automated proxy compares the obtained hit element with the target element or the target element's descendant nodes, comparing whether the node reference of the hit element and the node reference of the target element point to the same DOM node.

[0069] If the node reference of the hit element and the node reference of the target element point to the same DOM node, then the hit element is determined to be the target element itself.

[0070] If the node reference of the matched element differs from that of the target element, the automatic proxy traces back through the DOM tree, starting from the matched element node and moving upwards along its parent node chain, checking if the node references of its ancestor nodes at each level are identical to those of the target element. If a node identical to the target element is found during this backtracking process, the matched element is determined to be a descendant node of the target element.

[0071] If the hit element has the same node reference as the target element, or if the hit element is a descendant node of the target element, then the current sampling point is determined to be a valid sampling point, and the sampling point is identified as the action point. At the same time, the traversal of the remaining sampling points in the sampling point set is stopped.

[0072] If the hit element is neither the target element itself nor a descendant node of the target element, the current sampling point is determined to be an invalid sampling point, and the process continues to traverse the next sampling point in the sampling point set.

[0073] Understandably, during the traversal of the sample point set, the first sample point deemed valid is identified as the action point. Through the first-hit strategy, the automated agent prioritizes sample points that rank higher in the traversal order (usually the geometric center point or indentation point) while ensuring the validity of the action point. This ensures that the determination of the action point satisfies both the validity constraint and the rationality of the operation position.

[0074] In one example, the action point for occlusion detection is selected as follows:

[0075] in, For the action point, Its sampling point number, For the i-th sampling point, For viewport, For hit testing on the target root node, For target element, To collect its descendants.

[0076] In the specific implementation process, to ensure that the target operation can still be completed through path information in scenarios where the backend node identifier cannot be obtained, such as cross-source sub-frames and controlled inputs, read-only probing can be performed on each frame using path information. Operations are only executed when a unique parseable frame is detected, and the process is actively stopped when multiple parseable frames are detected. This avoids the problem of repeated actions across multiple frames (such as duplicate submissions), improving the robustness of the solution in extreme scenarios. At this point, after step S20, the following is also included: If the hit test result is that the hit element of the sampling point set is not the target element or a descendant node of the target element, then the target element is determined to be in an occluded state. In response to the target element being obscured, the number of candidate frames is obtained based on the path information in multiple frames of the webpage, and the candidate frames are used to parse the target element; If the number of candidate frames is one, then the processing operation on the target element is performed within the candidate frame; If there are multiple candidate frames, the processing operation is terminated.

[0077] It should be noted that occlusion refers to a state where the target element is completely covered by other upper-level elements (such as floating layers, sticky navigation bars, floating buttons, or masking layers) in the viewport, making it impossible for the target element itself or any descendant elements to be detected through hit testing. In this occlusion state, the automated agent cannot find a valid action point through hit testing.

[0078] Understandably, path information is one of the acquired identifiers used to locate the target element within the document tree of the frame to which it belongs. This path information can take the form of an XPath expression or a CSS selector. When the target element is obscured or the backend node identifier cannot be obtained, path information serves as a backup location method, used to detect the parsability of the target element in each frame.

[0079] Understandably, a candidate frame refers to a frame among multiple frames of a webpage (including the main frame and all child frames) where the target element can be successfully parsed using path information. Since a webpage may contain multiple child frames from the same or different origins, the same path information may be able to parse the corresponding element node in multiple frames. Therefore, it is necessary to obtain the number of candidate frames to determine the subsequent operation strategy.

[0080] Specifically, if the automated agent fails to find any valid action points after traversing all sampling points in the sampling point set, it determines that the target element is completely occluded by the upper-layer element. In response to this occlusion state, the automated agent initiates a location-free rollback process.

[0081] In the location-free rollback process, the automated agent parses target elements in multiple frames of the webpage using the acquired path information in a read-only manner. This read-only mode means that when the automated agent performs document query operations in each frame, it only obtains the parsing results (i.e., whether the element node was successfully located) and does not perform any actual operations (such as clicking, assigning values, etc.) to avoid unexpected changes to the page state during the probing phase.

[0082] Specifically, the automated agent traverses all frames of the webpage. For each frame, it performs a node query on the frame's document tree using the path information. If the query returns a valid DOM node, the frame is marked as a candidate frame; if the query returns a null value or throws an exception (e.g., the path does not match any node in the frame's document tree), the frame is marked as a non-candidate frame. After traversal, the automated agent counts the total number of candidate frames.

[0083] If there is only one candidate frame, it means that the path information uniquely points to the target element in a single frame within the entire webpage's frame tree, and there is no path ambiguity. The automated agent performs processing operations on the target element within this candidate frame. These processing operations include the automated agent injecting a script into the frame's context through the corresponding debugging session, constructing a synthetic event dispatch through the script, or directly assigning values ​​to the target element's attributes to achieve click or input operations. Alternatively, when performing processing operations on the target element within this candidate frame, the automated agent can also directly return to the step of matching the debugging session based on the frame identifier and dispatching a trusted input event at the action point through the debugging session to process the target element.

[0084] If there are multiple candidate frames, it indicates that the same path information can be resolved to element nodes in multiple frames, resulting in path ambiguity. In this case, since it is impossible to determine which frame the user intends to target, the automated agent suspends the current processing operation and does not dispatch events or perform assignments to any frame to avoid repeatedly performing the same operation in multiple frames, such as repeatedly submitting forms or clicking buttons.

[0085] If the number of candidate frames is zero, it means that the path information cannot be resolved to any element node in any frame of the webpage, and the target element may have been removed or the page structure has changed. The automated agent outputs an error status indicating that the target is unreachable.

[0086] In one example, reference Figure 4 , Figure 4 This is a schematic diagram of the frame selection for the location-free fallback method in this application. When the target element is determined to be occluded or the automated agent cannot obtain a valid backend node identifier for the target element, the automated agent initiates the location-free fallback process and obtains the pre-stored path information. Next, it enters the first stage, the read-only detection stage. The automated agent uses the path information to detect frames in multiple frames of the webpage that can resolve to the target element in read-only mode. If the current operation to be performed is an input operation, a fillable constraint is added during the detection process; that is, the frame is only counted as a valid detection result if the element resolved by the path information in the frame is a text input box, text field, or editable element.

[0087] After completing the read-only probing of all frames, the automated agent counts the number of candidate frames that can be parsed for the target element and determines the subsequent operation strategy based on the number of candidate frames.

[0088] If there is only one candidate frame, it means that the path information uniquely points to the target element in a frame within the webpage's frame tree. At this point, the second stage, the conditional execution stage, begins. Within this unique candidate frame, the automated agent performs processing operations on the target element through synthetic events or native assignments.

[0089] If there are multiple candidate frames, it means that the same path information can be resolved to element nodes in multiple frames, indicating path ambiguity. In this case, the automated agent suspends the current processing operation to avoid repeatedly executing the same action in multiple frames (such as repeatedly submitting forms or clicking buttons).

[0090] If the number of candidate frames is zero, it means that the path information cannot be parsed to any element node in all frames of the webpage, and the automated agent outputs an error status indicating that the target is unreachable.

[0091] In this embodiment, by traversing the sampling points in the sampling point set according to a preset traversal order and performing a hit test with the root node of the target element as the context, it can be ensured that when the target element is located inside the Shadow DOM, the hit test can still penetrate the Shadow boundary and correctly detect the target element or its child nodes. Simultaneously, the first sampling point whose hit result is the target element or its descendant is determined as the action point, which not only ensures the validity of the action point but also improves the localization efficiency through the first-hit strategy.

[0092] Reference Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the webpage element processing method of this application. Based on the above embodiments, the third embodiment of the webpage element processing method of this application is proposed.

[0093] In the third embodiment, step S40 includes: Step S401: Match the debugging session according to the frame identifier and determine the operation type of the operation to be performed on the target element.

[0094] It's important to note that a debugging session is a communication channel established between the automation proxy and the browser kernel via the browser debugging protocol. Each debugging session is bound to a specific frame (browser context) and is used to perform operations such as DOM queries, event dispatching, and property reading / writing within that frame's context. When the automation proxy establishes a connection with the browser, it creates an independent debugging session for each frame on the page (including the main frame and all child frames), storing and managing them using the frame identifier as an index.

[0095] It is understood that operation types include, but are not limited to, click operations, input operations, hover operations, drag operations, and visual coordinate operations.

[0096] Specifically, the automated agent matches the debug session containing the frame to which the target element belongs in the established debug session pool based on the frame identifier of the acquired target element. The matching process can involve using the frame identifier as the query key to search for the corresponding debug session object in the debug session pool. If a match is successful, the automated agent obtains a handle or reference to the debug session for that frame, which is used in subsequent steps to dispatch events in the correct frame context.

[0097] If a match fails, meaning there is no debug session in the debug session pool that corresponds to the target element frame identifier (e.g., the target frame has been destroyed, the page has been navigated to a new address, or the frame identifier has expired), the automated agent will output an error status indicating that the session is unavailable and stop subsequent operations.

[0098] During or after the matching debugging session, the automation agent parses the received operation instructions to determine the operation type to be performed on the target element.

[0099] In the specific implementation process, when the operation type is a visual coordinate operation, normalized coordinate components can be obtained and mapped to viewport pixel coordinates. Then, trusted input events can be dispatched through the debugging session corresponding to the main frame. This allows the automated proxy to perform click or input operations based on visual information such as webpage screenshots, without relying on the DOM identifier information of the target element. At this point, after step S401, the following may be included: If the operation type is a visual coordinate operation, obtain the normalized coordinate components; Map the normalized coordinate components to viewport pixel coordinates; Trusted input events are dispatched at the viewport pixel coordinates through a debug session corresponding to the main frame to perform viewport-level operations.

[0100] It should be noted that visual coordinate operation is an operation mode that does not rely on the DOM identification information of the target element. The automated proxy determines the operation position based on the webpage screenshot and represents the operation target point through normalized coordinates. It is suitable for scenarios where the target element cannot be located by DOM or the proxy generates the operation intent based on visual information.

[0101] It is understood that normalized coordinate components are coordinate values ​​represented by a predetermined normalization interval, which is an integer range from 0 to 1000, representing the relative position in the viewport width and height directions, respectively. Viewport pixel coordinates are coordinate values ​​in physical pixels with the top-left corner of the viewport as the origin. The horizontal axis represents the pixel distance from the left edge of the viewport, and the vertical axis represents the pixel distance from the top edge of the viewport.

[0102] It should be understood that the main frame is the top-level browsing context of a webpage, that is, the root frame containing all child frames. The debug session corresponding to the main frame is the debug communication channel allocated to the main frame when the automation proxy establishes a connection with the browser. Since visual coordinate operations do not depend on the frame identifier and DOM identifier of the target element, but are performed directly based on viewport coordinates, there is no need to match the debug session of the frame to which the target element belongs; instead, event dispatch is uniformly completed through the debug session corresponding to the main frame. The viewport-level operation refers to the operation performed directly on a specific pixel position in the viewport, using the viewport coordinate system as a reference. This operation does not depend on the DOM context of any frame, but directly acts on the rendered content at the viewport coordinate position.

[0103] Specifically, if the operation type is a visual coordinate operation, the automated agent receives the normalized coordinate components carried in the operation instruction, which can be generated by a large language model. After analyzing the webpage screenshot, the large language model outputs the relative position coordinates of the target in the screenshot as a normalized interval, and passes these coordinates as parameters of the operation instruction to the automated agent. The automated agent parses and extracts the normalized horizontal coordinate component and the normalized vertical coordinate component from the operation instruction.

[0104] After obtaining the normalized coordinate components, the automated proxy uses the browser's debugging protocol to acquire the actual size information of the current viewport and converts the normalized coordinate components into viewport pixel coordinates according to the mapping formula. Since the normalized coordinate components are independent of the actual viewport size and instead represent position as a relative proportion, the mapped viewport pixel coordinates can automatically adapt to browser environments with different resolutions and window sizes. Furthermore, because event dispatch is based on viewport pixel coordinates rather than element DOM identifiers, visual coordinate operations are immune to browser scaling (such as page scaling) and device pixel ratios. That is, even under page scaling or high-DPI screens, the pixel coordinates obtained from the normalized coordinate mapping still correspond to the same visual position in the screenshot.

[0105] In one example, the viewport pixel coordinate normalization mapping is as follows:

[0106] in, For viewport pixel coordinates, For normalized coordinate components, The dimensions within the viewport are used to constrain the coordinates within a legal range using the clamp function, eliminating the risk of exceeding the limits.

[0107] After determining the viewport pixel coordinates, the automation agent dispatches trusted input events at the viewport pixel coordinates through the debug session corresponding to the main frame. For click operations, the automation agent sends a trusted mouse event dispatch instruction to the browser kernel via the debug protocol. The instruction contains the click event type and coordinate parameters. After receiving the instruction, the browser kernel performs a standard hit test and event dispatch process at the viewport coordinates. For input operations, the automation agent first dispatches a trusted focus event to the viewport coordinates, and then dispatches a trusted input event.

[0108] Understandably, since the visual coordinate operation mode does not depend on the frame identifier of the target element, when matching the debugging session, even if the target element is located within a cross-source sub-frame, it is not necessary to match the debugging session of that sub-frame. Instead, the debugging session of the main frame is used to complete the event dispatch, thus avoiding the problem that the operation cannot be executed due to the failure of cross-source sub-frame debugging session matching.

[0109] Step S402: If the operation type is an input operation, a trusted focus event is dispatched at the action point through the debugging session to focus the target element, and a trusted input event is dispatched after focusing to input content into the target element.

[0110] It should be noted that input operations refer to the actions of automated agents writing or inserting text content into target elements, which are commonly seen in scenarios such as form filling, search box input, and text editing.

[0111] Understandably, the trusted focus event is a focus event directly generated and dispatched by the browser engine, and its isTrusted property is true. The focus event is used to set keyboard input focus on the target element, allowing subsequent keyboard input events to be received by the target element. For text input boxes, text fields, and editable elements, focus is a prerequisite for performing input operations. The trusted input event is used to write text content to the focused element, and its behavior is completely consistent with a real user typing character by character on the keyboard.

[0112] Specifically, the automated agent dispatches a trusted focus event at a determined action point based on the matched debugging session. The dispatch method involves sending an event dispatch instruction to the browser kernel via the debugging protocol. This instruction includes the event type "focus," event coordinates (action point coordinates), and other standard event parameters. Upon receiving the instruction, the browser kernel generates a trusted focus event in the context of the target frame and dispatches it to the topmost element at the action point coordinates according to the browser's standard event dispatch process. If this element is a focusable element (such as an input box, text field, editable element, or element with a tabindex set), the element gains input focus.

[0113] After the target element gains focus, the automation agent dispatches a trusted input event through the same debugging session to input content into the target element. The source of the input content is text data carried in the operation instructions received by the automation agent.

[0114] Step S403: If the operation type is a click operation, then when the target element is an editable element, a focus operation is performed on the target element through the debugging session, instead of dispatching a trusted click event at the action point through the debugging session.

[0115] It should be noted that when the target element is an editable element, the focus operation is used instead of the click operation. Editable elements are DOM elements whose content can be directly edited by the user, including but not limited to text input boxes, text fields, and elements set to editable state. These elements typically have an independent internal document structure, and handling click events may result in cursor position movement, changes in internal node structure, or changes in focus state.

[0116] As you can understand, focus operation refers to the action of programmatically setting keyboard input focus on a target element, with the same effect as a user clicking the element or using the Tab key to navigate to the element. Focus operation does not change the element's internal document structure; it only activates the element's input state, enabling the element to receive subsequent keyboard events.

[0117] Specifically, after determining that the operation type is a click operation, the automated agent further determines whether the target element is an editable element. If the target element is determined to be an editable element, the automated agent performs a focus operation on the target element through the matched debug session, instead of dispatching a trusted click event at the action point.

[0118] It should be understood that by replacing the click operation with the focus operation, the following problems caused by the click operation can be avoided: (1) When clicking an editable element, the browser will position the cursor at the click position (i.e., the text index position corresponding to the action point), which may change the document structure or cursor state inside the element; (2) For rich text editors, the click operation may cause the editor to generate new nodes or modify the attributes of existing nodes, resulting in the failure of subsequent operation positioning based on the element index; (3) The click operation may trigger business logic related to the click event in the page (such as expand, collapse, pop-up, etc.), while the focus operation is more "lightweight" and only activates the input state of the element.

[0119] If the target element is determined not to be an editable element, the automated agent will execute the normal click operation process, that is, dispatch a trusted click event at the action point through the matched debug session to perform a click operation on the target element.

[0120] In this embodiment, when the operation type is an input operation, the target element is first given input focus through a focus event, and then the input event is dispatched to write the content. This ensures that the input operation can be correctly captured and responded to by the front-end framework's state management mechanism, avoiding the problem of interface and data being out of sync due to directly assigning values ​​to the controlled component. When the operation type is a click operation, a focus operation is used instead of a click operation for editable elements. This avoids the click action from damaging the internal document structure of the rich text editor, thereby preventing subsequent input positioning based on element index from failing and ensuring the reliability of operations in rich text editing scenarios.

[0121] For example, to help understand the implementation flow of the webpage element processing method obtained by combining this embodiment with the above embodiment one, please refer to... Figure 6 , Figure 6 This is a simplified flowchart of a webpage element processing method. Specifically: First, step S1 is executed to obtain the target element identifier, which includes the backend node identifier, frame identifier, and path information. Next, step S2 is executed to generate a set of sampling points on the bounding rectangle of the target element and perform a hit test point by point, taking the first sampling point that hits the target element as the action point. Then, step S3 is executed to determine whether there is a sampling point that hits the target element. If not, it is determined that the target is occluded and a backtrack is triggered. If it exists, step S4 is executed to dispatch a trusted input event at the action point through the debugging session that owns the frame to which the target belongs. When step S3 triggers a backtrack, step S5 is executed to enter the location-free backtracking process, which performs multi-frame detection in read-only mode. If a unique hit is found, the action is executed in the corresponding frame. If multiple frames are hit, the operation is stopped to avoid duplication. Finally, step S6 is executed to output the operation result, which can be recognized by the webpage event listener as a click or input to the target element.

[0122] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the web page element processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0123] This application also provides a web page element processing device, please refer to... Figure 7 The webpage element processing device includes: The acquisition module 10 is used to acquire the identification information of the target element in the webpage, the identification information including the backend node identifier and the frame identifier; The parsing module 20 is used to parse the target element based on the backend node identifier and generate a set of sampling points based on the parsing result; The execution module 30 is used to perform a hit test based on the set of sampling points and determine the action point based on the hit test result; The processing module 40 is used to match a debugging session according to the frame identifier, and dispatch a trusted input event at the action point through the debugging session to process the target element.

[0124] The webpage element processing apparatus provided in this application, employing the webpage element processing method in the above embodiments, can solve the technical problem that automated proxies cannot reliably complete click / input operations on target elements that can be recognized by the page's real event listener on real webpages with overlay obstruction and multiple frames (including cross-source subframes). Compared with the prior art, the beneficial effects of the webpage element processing apparatus provided in this application are the same as those of the webpage element processing method provided in the above embodiments, and other technical features in the webpage element processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0125] This application provides a web page element processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the web page element processing method in the first embodiment described above.

[0126] The following is for reference. Figure 8 This document illustrates a structural diagram of a web page element processing device suitable for implementing embodiments of this application. The web page element processing device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The web page element processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0127] like Figure 8As shown, the web page element processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the web page element processing device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the web element processing device to communicate wirelessly or wiredly with other devices to exchange data. Although web element processing devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0128] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0129] The webpage element processing device provided in this application, employing the webpage element processing method in the above embodiments, can solve the technical problem that automated agents cannot reliably complete click / input operations on target elements that can be recognized by the page's real event listener on real webpages with overlay obstruction and multiple frames (including cross-source subframes). Compared with the prior art, the beneficial effects of the webpage element processing device provided in this application are the same as those of the webpage element processing method provided in the above embodiments, and other technical features in this webpage element processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0130] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0131] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0132] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the web page element processing method in the above embodiments.

[0133] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0134] The aforementioned computer-readable storage medium may be included in a web page element processing device; or it may exist independently and not be assembled into a web page element processing device.

[0135] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the web page element processing device, cause the web page element processing device to perform the web page element processing method described above.

[0136] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0139] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described webpage element processing method. This solves the technical problem that on real webpages with overlays and multiple frames (including cross-origin subframes), automated agents cannot reliably perform click / input operations on target elements that can be recognized by the page's real event listener. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the webpage element processing method provided in the above embodiments, and will not be repeated here.

[0140] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the web page element processing method described above.

[0141] The computer program product provided in this application can solve the technical problem that automated operations cannot be accurately applied to the target element when the target element is obscured by an overlay on the page. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the web page element processing method provided in the above embodiments, and will not be repeated here.

[0142] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

[0143] It should be noted that the data collection, tag management, rule setting, and push decision-making processes involved in this application are designed to work with other technical features to solve technical problems. They do not involve or support any illegal activities. Any data processing that may violate laws and regulations (such as unauthorized collection of privacy data, generation of discriminatory tags, setting unfair rules, or pushing illegal information) is not within the scope of protection of this application's technical solution. Of course, the user data in this application will be encrypted, anonymized, or de-identified before storage to ensure user data security.

Claims

1. A method for processing web page elements, characterized in that, The webpage element processing method includes: Obtain the identification information of the target element in the webpage, the identification information including the backend node identifier and the frame identifier; The target element is parsed based on the backend node identifier, and a set of sampling points is generated based on the parsing result; A hit test is performed based on the set of sampling points, and the action point is determined based on the hit test results; The debugging session is matched according to the frame identifier, and a trusted input event is dispatched at the action point through the debugging session to process the target element.

2. The webpage element processing method as described in claim 1, characterized in that, The step of parsing the target element based on the backend node identifier and generating a sampling point set based on the parsing result includes: The target element is parsed based on the backend node identifier, and at least one bounding rectangle of the target element within the viewport is determined based on the parsing result. Multiple sampling points are generated within the circumscribed rectangle to obtain a sampling point set.

3. The webpage element processing method as described in claim 2, characterized in that, The process of generating multiple sampling points within the circumscribed rectangle to obtain a sampling point set includes: Obtain the geometric center point of the circumscribed rectangle; The vertices of the circumscribed rectangle are translated into the interior of the circumscribed rectangle to obtain multiple indentation points; Based on the midpoints of each side of the circumscribed rectangle, multiple boundary sampling points are determined; The geometric center point, the plurality of recessed corner points, and the plurality of boundary sampling points are combined to generate a sampling point set.

4. The webpage element processing method as described in claim 1, characterized in that, The step of performing a hit test based on the sample point set and determining the action point based on the hit test result includes: The sampling points in the sampling point set are traversed sequentially according to the preset traversal order. For the currently traversed sampling point, perform a hit test on the root node corresponding to the target element to obtain the hit element at the sampling point position; Determine whether the hit element is the target element or a descendant node of the target element; If so, the first hit element will be designated as the action point.

5. The webpage element processing method as described in claim 1, characterized in that, The identification information also includes path information, and after performing the hit test based on the sampling point set, it further includes: If the hit test result is that the hit element of the sampling point set is not the target element or a descendant node of the target element, then the target element is determined to be in an occluded state. In response to the target element being obscured, the number of candidate frames is obtained based on the path information in multiple frames of the webpage, and the candidate frames are used to parse the target element; If the number of candidate frames is one, then the processing operation on the target element is performed within the candidate frame; If there are multiple candidate frames, the processing operation is terminated.

6. The webpage element processing method as described in claim 1, characterized in that, The step of matching a debugging session based on the frame identifier and dispatching a trusted input event at the action point through the debugging session to process the target element includes: The debugging session is matched based on the frame identifier, and the operation type of the operation to be performed on the target element is determined. If the operation type is an input operation, a trusted focus event is dispatched at the action point through the debugging session to focus the target element, and a trusted input event is dispatched after focusing to input content into the target element; If the operation type is a click operation, then when the target element is an editable element, a focus operation is performed on the target element through the debugging session, instead of dispatching a trusted click event at the action point through the debugging session.

7. The webpage element processing method as described in claim 6, characterized in that, After determining the operation type to be performed on the target element, the method further includes: If the operation type is a visual coordinate operation, obtain the normalized coordinate components; Map the normalized coordinate components to viewport pixel coordinates; Trusted input events are dispatched at the viewport pixel coordinates through a debug session corresponding to the main frame to perform viewport-level operations.

8. A webpage element processing device, characterized in that, The device includes: The acquisition module is used to acquire the identification information of the target element in the webpage, the identification information including the backend node identifier and the frame identifier; The parsing module is used to parse the target element based on the backend node identifier and generate a set of sampling points based on the parsing results; An execution module is used to perform a hit test based on the set of sampling points and determine the action point based on the hit test results; The processing module is configured to match a debugging session based on the frame identifier and dispatch a trusted input event at the action point through the debugging session to process the target element.

9. A webpage element processing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the web page element processing method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the web page element processing method as described in any one of claims 1 to 7.

Images