Webpage data processing method, apparatus, device, and medium

Abstract

The present disclosure provides a webpage data processing method and device, equipment and medium, relates to the technical field of artificial intelligence, in particular to the technical field of webpage development and deep learning. The method comprises: determining an interactive operation for a target webpage; obtaining a screenshot of the target webpage and attributes of a plurality of webpage elements in the target webpage, the attributes indicating functions possessed by the webpage elements; based on the attributes of the plurality of webpage elements, screening a plurality of candidate webpage elements related to the interactive operation from the plurality of webpage elements; determining the positions and sizes of the plurality of candidate webpage elements, and cutting the screenshot to obtain visual segments of the plurality of candidate webpage elements; determining the intention matching degrees of the attributes of the plurality of candidate webpage elements and the interactive operation; determining the visual matching degrees of the visual segments of the plurality of candidate webpage elements and the interactive operation; and based on the intention matching degrees and the visual matching degrees, determining a target webpage element from the plurality of candidate webpage elements and executing the interactive operation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of web development and deep learning, specifically to a web data processing method, a web data processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include natural language processing, computer vision, speech recognition, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a web page data processing method, a web page data processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a webpage data processing method is provided, comprising: determining an interactive operation for a target webpage; acquiring a screenshot of the target webpage and attributes of multiple webpage elements in the target webpage, wherein the attributes indicate the functions possessed by the corresponding webpage elements; filtering multiple candidate webpage elements related to the interactive operation from the multiple webpage elements based on the attributes of the multiple webpage elements; determining the position and size of the multiple candidate webpage elements and cropping visual fragments of the multiple candidate webpage elements from the screenshot; determining the degree of matching between the attributes of the multiple candidate webpage elements and the intent of the interactive operation; determining the degree of visual matching between the visual fragments of the multiple candidate webpage elements and the interactive operation; and determining the target webpage element from the multiple candidate webpage elements based on the degree of matching and the degree of visual matching, and performing an interactive operation on the target webpage element.

[0006] According to another aspect of this disclosure, a web page data processing apparatus is provided, comprising: a first determining unit configured to determine an interactive operation for a target web page; an acquiring unit configured to acquire a screenshot of the target web page and attributes of multiple web page elements in the target web page, wherein the attributes indicate the functions possessed by the corresponding web page elements; a filtering unit configured to filter multiple candidate web page elements related to the interactive operation from the multiple web page elements based on the attributes of the multiple web page elements; a cropping unit configured to determine the position and size of the multiple candidate web page elements and crop visual fragments of the multiple candidate web page elements from the screenshot; a second determining unit configured to determine the degree of matching between the attributes of the multiple candidate web page elements and the intent of the interactive operation; a third determining unit configured to determine the degree of visual matching between the visual fragments of the multiple candidate web page elements and the interactive operation; and a fourth determining unit configured to determine the target web page element from the multiple candidate web page elements based on the degree of matching and the degree of visual matching, and perform an interactive operation on the target web page element.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0010] According to one or more embodiments of this disclosure, the disclosure first obtains interactive operations targeting a target webpage, and then parses the target webpage in both visual and functional dimensions. Next, it first filters candidate webpage elements using attributes, then performs dual-path matching between the candidate webpage elements and the interactive operations at both the intent and visual levels, and finally determines the target webpage element. This method improves the accuracy of identifying the target webpage element targeted by the interactive operation, ensuring that the interactive operation can be executed accurately according to its intent.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0014] Figure 2 A flowchart of a web page data processing method according to an embodiment of the present disclosure is shown; Figure 3 A flowchart illustrating the acquisition of a screenshot of a target webpage and the attributes of multiple webpage elements in the target webpage according to an embodiment of the present disclosure is shown. Figure 4 A flowchart of a web page data processing method according to an embodiment of the present disclosure is shown; Figure 5 A flowchart of a web page data processing method according to an embodiment of the present disclosure is shown; Figure 6 A structural block diagram of a web page data processing apparatus according to an embodiment of the present disclosure is shown; Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0017] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0018] In related technologies, existing methods for performing interactive operations on web pages have a low accuracy rate in identifying the web page elements targeted by the interactive operations, resulting in the inability to accurately complete the interactive operations.

[0019] To address the aforementioned issues, this disclosure first acquires the interactive operations targeting a target webpage and analyzes the target webpage from both visual and functional dimensions. Then, it first filters candidate webpage elements using attributes, and then performs dual-path matching between the candidate webpage elements and the interactive operations at both the intent and visual levels to ultimately determine the target webpage element. This approach improves the accuracy of identifying the target webpage element targeted by the interactive operation, ensuring that the interactive operation is executed accurately according to its intent.

[0020] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0022] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of the methods of this disclosure.

[0023] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0024] exist Figure 1In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0025] Users can use client devices 101, 102, 103, 104, 105, and / or 106 for human-computer interaction. The client devices provide interfaces that enable users to interact with them. The client devices can also output information to the user through these interfaces. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0026] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0027] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0028] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0029] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0030] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0031] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0032] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0033] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0034] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0035] According to one aspect of this disclosure, a web page data processing method is provided. Figure 2 This is a flowchart illustrating a web page processing method 200 according to an exemplary embodiment.

[0036] refer to Figure 2 In step S201, the interactive operation for the target webpage is determined.

[0037] In some embodiments, an interactive operation (also known as an atomic operation) refers to the smallest executable unit obtained after performing intent recognition and task decomposition on a natural language instruction from a user or intelligent agent. An interactive operation represents a single interactive action or behavior in a web page environment. In an exemplary embodiment, when the user instruction is to book a flight, the decomposed interactive operation could be entering the destination city name in the destination input box, or clicking the search button on the web page.

[0038] In step S202, a screenshot of the target webpage and the attributes of multiple webpage elements in the target webpage are obtained.

[0039] In some embodiments, a screenshot can refer to image data captured from a rendered visual image of a target webpage. Webpage elements can be relatively independent logical units within a webpage, such as a button or a CAPTCHA input box. The attributes of a webpage element indicate its functionality, and may include its function category, descriptive information, and displayed text. This information can be obtained directly through the browser's developer interface or render tree parsing tools, or it can be obtained by processing the raw webpage data.

[0040] In step S203, based on the attributes of multiple web page elements, multiple candidate web page elements related to interactive operations are filtered from the multiple web page elements.

[0041] In some embodiments, the name, category, or other descriptive information of the interactive operation can be filtered based on keywords and the attributes of web page elements. Alternatively, filtering can be performed using preset rules, such as removing static text elements or decorative background elements that lack interactive functionality. This filtering process aims to filter out irrelevant and distracting items from the full web page data, thereby reducing the scale of subsequent calculations. In an exemplary embodiment, if the interactive operation is clicking "Login," the system will iterate through the attributes of web page elements, extract all web page elements categorized as buttons or links, and whose display text contains the words "Login," "Submit," or "Login," marking them as candidate web page elements. It is understood that other filtering methods can also be used, and are not limited here.

[0042] In step S204, the positions and sizes of multiple candidate web page elements are determined, and visual fragments of multiple candidate web page elements are obtained by cropping from the screenshot.

[0043] In some embodiments, the position and size of a candidate webpage element define the spatial extent of that element in the webpage coordinate system. For example, the position and size of these candidate webpage elements can be determined by analyzing the raw data of the target webpage.

[0044] In some embodiments, a visual fragment can be a local image region in a screenshot corresponding to a specific candidate webpage element. By cropping, the webpage screenshot can be transformed into local image features targeting a specific object. In an exemplary embodiment, based on the coordinate range of each candidate webpage element obtained in step S202, the corresponding image block is extracted from the original screenshot as a visual fragment. These visual fragments preserve the appearance of the webpage element and its surrounding visual context.

[0045] In step S205, the matching degree between the attributes of multiple candidate web page elements and the intent of the interactive operation is determined.

[0046] In some embodiments, intent matching can be used to measure the degree to which candidate web page elements match the operational intent at the logical level. The system can derive a quantitative score by analyzing the correlation between the web page attributes of the elements and the operation instructions. In an exemplary embodiment, a deep learning model can be used to calculate the similarity score between the name or text description of the interactive operation and the attributes of the candidate web page elements. The higher the score, the more the element's functional positioning matches the intent of the interactive operation.

[0047] It should be noted that the screening process in step S203 is a preliminary filtering stage, which can quickly retrieve candidate objects from the full amount of web page data based on preset rules or keyword matching, thereby reducing the processing scale of subsequent complex calculations. Step S205, on the other hand, is a fine-grained quantification stage, which can use a deep learning model to perform fine-grained intent matching on the selected candidate web page elements. This coarse-to-fine processing mechanism ensures recognition accuracy while reducing the system resource consumption when processing highly complex web pages.

[0048] In step S206, the visual matching degree between visual fragments of multiple candidate web page elements and interactive operations is determined.

[0049] In some embodiments, visual matching can be used to verify whether candidate web page elements meet operational expectations at the image feature level, and can identify visual features that cannot be fully described by the attributes of web page elements. In an exemplary embodiment, a lightweight image classification model can be used to identify visual segments to confirm whether the segment has the visual content of buttons, input boxes or specific icons required for interactive operations, thereby determining whether the element supports the corresponding interactive operations in terms of visual presentation.

[0050] In step S207, based on intent matching degree and visual matching degree, the target web page element is determined from multiple candidate web page elements, and interactive operations are performed on the target web page element.

[0051] This step achieves a fusion decision based on intent matching and visual matching. By comprehensively utilizing the matching results of both paths, the system can accurately locate the operation target among candidate web page objects. In an exemplary embodiment, the system comprehensively evaluates the intent matching degree and visual matching degree, selects the candidate web page element with the highest matching degree as the final operation object, and sends an instruction to the browser to perform an interactive operation at the coordinate position corresponding to the target web page element.

[0052] Therefore, this disclosure first obtains the interactive operations targeting the target webpage, and then analyzes the target webpage in both visual and functional dimensions. Next, it filters candidate webpage elements using attributes, and then performs dual-path matching between the candidate webpage elements and the interactive operations at both the intent and visual levels, ultimately determining the target webpage element. This method improves the accuracy of identifying the target webpage element targeted by the interactive operation, ensuring that the interactive operation can be executed accurately according to its intent.

[0053] According to some embodiments, such as Figure 3 As shown, step S202, obtaining a screenshot of the target webpage and the attributes of multiple webpage elements in the target webpage, may include: step S301, obtaining the document object model tree of the target webpage and identifying multiple interactive nodes in the document object model tree; step S302, extracting the interactive node information of multiple interactive nodes and the parent container node information of each parent container node of multiple interactive nodes from the document object model tree, wherein the interactive node information and the parent container node information each include at least one of the node name, node value, and node type of the corresponding node; and step S303, constructing multiple webpage elements based on multiple interactive nodes, wherein the attributes of each webpage element include the interactive node information of the corresponding interactive node and the parent container node information of the parent container node of the interactive node.

[0054] Therefore, by acquiring the node information of interactive nodes and the node information of parent container nodes, and constructing web page elements accordingly, it is possible to reduce the amount of web page structure data while fully preserving the functional characteristics of web page elements. This processing method avoids the waste of computational resources caused by directly processing the entire deeply nested document object model tree, reduces the computational overhead during subsequent intent matching, improves data processing efficiency, and enhances the accuracy of target web page element identification.

[0055] In some embodiments, the Document Object Model (DOM) tree can be a structured representation of a webpage, consisting of multiple nested nodes. Interactive nodes refer to interactive nodes in the DOM tree that can respond to user actions, typically including buttons, input boxes, hyperlinks, or components with specific event listeners. Parent container nodes refer to nodes that are above the interactive nodes in the DOM tree hierarchy; these nodes often carry titles, labels, or descriptive text describing the functionality of the interactive nodes.

[0056] In some embodiments, node information can refer to data used to describe the characteristics of a specific node in a Document Object Model (DOM) tree. Node information may include at least one of node name, node value, and node type. The node name may include the node's tag name, such as button, div, etc.; the node value may include the text content, attribute value, or input value carried by the node; and the node type may include the node's category identifier or functional definition. Interactive node information refers to the aforementioned data possessed by the interactive node itself. Parent container node information refers to the aforementioned data extracted from the parent container node, which may include, for example, its corresponding tag text or title. This data typically provides supplementary descriptions of the interactive node.

[0057] In step S301, the system can obtain the complete document object model tree of the current page through the interface provided by the browser engine. Subsequently, the system can identify nodes with interactive functions based on preset filtering rules. In an exemplary embodiment, the system traverses the document object model tree, identifies all nodes labeled as button, input, and select as interactive nodes, and can remove invisible or disabled nodes to reduce interference from invalid data.

[0058] In step S302, since complex web pages often place descriptive text within the parent container of interactive nodes, directly extracting the interactive node may result in the loss of crucial context. In some embodiments, the system can extract the node information of the interactive node itself and the node information of its corresponding parent container node from the Document Object Model tree, thereby obtaining more comprehensive information about the interactive function. In an exemplary embodiment, for an input box node, the system can extract its own type attribute and automatically search for the associated label text in its parent div or section tag as the parent container node information.

[0059] In step S303, by reconstructing the extracted node information, the system can generate a flattened webpage data representation. The generated webpage elements no longer retain the redundant nested levels of the original document object model tree, but instead are centered around interactive nodes. In some embodiments, each constructed webpage element can be encapsulated as a structured object, whose attribute set includes interactive node information and parent container node information. In this way, the originally deeply nested webpage structure can be transformed into a list of parallel webpage elements, facilitating rapid intent matching calculation in subsequent steps.

[0060] According to some embodiments, step S204, determining the position and size of multiple candidate web page elements and cropping the visual fragments of multiple candidate web page elements from the screenshot, may include (not shown in the figure): step S2041, determining the coordinate information of the interactive nodes corresponding to each of the multiple candidate web page elements in the target web page based on the document object model tree; and step S2042, determining the position and size of multiple candidate web page elements based on the coordinate information of the corresponding interactive nodes.

[0061] Therefore, by extracting and calculating the precise coordinates of interactive nodes from the document object model tree, an accurate mapping between web page elements and visual spatial layout can be established. This provides a basis for accurately capturing visual fragments in subsequent screenshots, reduces recognition errors caused by coordinate alignment deviations, and improves the accuracy of web page element positioning.

[0062] In some embodiments, coordinate information may refer to the geometric attributes of candidate interactive nodes in the webpage rendering result, and may be described using a Cartesian coordinate system. In an exemplary embodiment, coordinate information may include the horizontal and vertical offsets of the top-left corner of the node relative to the page origin or the current viewport origin, as well as the length of the node in the horizontal and vertical directions.

[0063] In some embodiments, in step S2041, the system can call the layout calculation interface provided by the browser engine to obtain the rendering box attributes of the interactive node. Specifically, the system can call the interface to obtain the original coordinates of the node relative to the current viewport, and can perform a linear conversion by combining the scroll bar offset value of the current page and the page scaling ratio, thereby obtaining the absolute coordinates of the interactive node in the global coordinate system of the target webpage. In this way, the system can transform the nodes in the abstract document object model tree into geometric regions with clear spatial boundaries.

[0064] In some embodiments, in step S2042, the system can determine the sampling area of ​​the corresponding candidate webpage element in the screenshot based on the calculated absolute coordinates of the interactive node. Since the candidate webpage element is constructed based on the interactive node, the system can establish a correspondence between coordinate information and candidate webpage element identifiers. Before performing the cropping operation, the system can extract the boundary range of the interactive node associated with a specific candidate webpage element, and can determine this range as the position and size of the candidate webpage element, thereby ensuring that the cropped visual fragment can accurately contain the interactive node.

[0065] According to some embodiments, such as Figure 4As shown, the webpage data processing method 400 may include: step S403, segmenting the screenshot of the target webpage into regions to obtain multiple region images; step S404, determining the importance of each of the multiple region images; step S405, determining the compression ratio of each region image based on its importance; and step S406, compressing the multiple region images based on their respective compression ratios. It is understood that the operations and effects of steps S401-S402 and steps S407-S411 in method 400 can be referred to the description of steps S201-S207 in method 200 above, and will not be repeated here.

[0066] Therefore, by applying non-uniform regional compression to webpage screenshots, the system can significantly reduce visual data transmission bandwidth and storage overhead while selectively preserving the visual features of key areas. This processing method avoids the waste of computational resources associated with full-resolution screenshots, improves the efficiency of visual processing, and ensures the recognition accuracy of core areas related to interactive operations.

[0067] In some embodiments, region segmentation can refer to dividing the entire screenshot into multiple blocks with independent attributes using image processing algorithms or prior knowledge of webpage structure. Importance can refer to the potential value of each block for completing the current interactive operation. Compression ratio can refer to the proportion of lossy or lossless processing of image data, used to control the balance between image quality and data volume.

[0068] In step S403, the system can use a lightweight superpixel segmentation algorithm to initially divide the screenshot. Furthermore, the system can also combine prior knowledge of the DOM structure during webpage rendering to divide the screenshot into different functional blocks such as areas with dense interactive controls, text content areas, and background decoration areas, thereby obtaining multiple independent region images.

[0069] In step S404, the system can perform analysis based on the task objective of the current interactive operation. If a certain region of the image contains candidate web page elements related to the intent or function of the interactive operation, the system can identify that region as a high-importance region. Conversely, for blocks that only contain large background images or decorative elements, the system can identify them as low-importance regions.

[0070] In step S405, the system can establish a mapping relationship between importance and compression parameters. For highly important regions, the system can assign a lower compression rate to preserve high-fidelity visual details. For less important regions, the system can assign a higher compression rate, thereby minimizing data volume without affecting core tasks.

[0071] In step S406, the system can compress the images of each region according to a preset compression rate. The compressed images of multiple regions can be recombined or transmitted independently to the subsequent visual recognition module.

[0072] According to some embodiments, step S204, determining the position and size of multiple candidate web page elements and cropping the screenshot to obtain visual fragments of multiple candidate web page elements, may include: for each candidate web page element, determining a corresponding region image in multiple region images based on the position of the candidate web page element; and cropping the corresponding region image based on the size of the candidate web page element to obtain a visual fragment corresponding to the candidate web page element.

[0073] Therefore, by determining the corresponding region image based on the position of the candidate web page element and cropping it, it can be ensured that the extracted visual fragment accurately contains the corresponding candidate web page element.

[0074] In some embodiments, the system can utilize the location information of candidate web page elements to locate the specific region image to which they belong. Specifically, the system can determine the spatial correspondence between the bounding box range of the candidate web page element and multiple region images based on the bounding box range of the target web page's global coordinate system. Furthermore, the system can identify the compressed region image containing the bounding box range and perform a cropping operation on the identified region image based on the size of the candidate web page element, thereby extracting visual segments for subsequent visual matching degree calculation.

[0075] According to some embodiments, such as Figure 5 As shown, the webpage processing method 500 may further include: step S507, determining the task type of the interactive operation; and step S508, determining the intent weight for intent matching degree and the visual weight for visual matching degree based on the task type. It is understood that the operations and effects of steps S501-S506 and S509 in method 500 can be referred to the description of steps S201-S207 in method 200 above, and will not be repeated here.

[0076] In some embodiments, step S509, determining the target webpage element from multiple candidate webpage elements based on intent matching degree and visual matching degree, and performing interactive operations on the target webpage element, may include (not shown in the figure): step S5091, performing a weighted summation of intent matching degree and visual matching degree based on intent weight and visual weight to obtain the comprehensive matching degree of each of the multiple candidate webpage elements; and step S5092, determining the target webpage element based on the comprehensive matching degree of each of the multiple candidate webpage elements.

[0077] Therefore, by determining the task type of the interactive operation and dynamically adjusting the weight allocation of intent and visual perception, automatic weight adjustment of intent matching and visual matching can be achieved. This approach ensures that the system maintains high recognition accuracy in different types of web page interactive tasks, thereby improving the accuracy of the operation.

[0078] In some embodiments, task type can refer to the classification of operations based on the functional attributes and expected goals of the interactive operation. Intent weight and visual weight can refer to the proportional coefficients assigned to the intent matching result and visual matching result, respectively, during the comprehensive judgment process. The comprehensive matching degree can refer to the final confidence index used to measure the candidate web page element as the target of the operation, obtained through weighted calculation.

[0079] In step S507, the system can classify tasks into form-interactive, information extraction, or navigation / browsing types based on the action instructions and context of the interactive operation. For example, when the interactive operation involves text input, the system can identify it as a form-interactive task.

[0080] In step S508, the system can pre-maintain a mapping table between task types and weights. For example, for information extraction tasks, the system can assign higher intent weights and lower visual weights to make full use of the functional attribute information of web page elements; for tasks involving complex graphical interactions, the system can correspondingly increase the visual weights and decrease the intent weights.

[0081] In step S5091, the system can multiply the intent matching degree of each candidate web page element by its intent weight, and multiply its visual matching degree by its visual weight. Finally, the system can add the products of the two to obtain the comprehensive matching degree.

[0082] In step S5092, the system can sort all candidate web page elements according to the overall matching degree, and select the element that best meets the requirements as the target web page element according to the preset judgment rules.

[0083] According to some embodiments, step S5092, determining the target webpage element based on the comprehensive matching degree of each of the multiple candidate webpage elements, may include: in response to determining that the highest comprehensive matching degree among the multiple candidate webpage elements is greater than a first threshold, determining the corresponding candidate webpage element as the target webpage element.

[0084] Therefore, by setting a first threshold for direct judgment, rapid decision-making can be achieved in high-confidence scenarios, reducing the system's decision-making delay.

[0085] In some embodiments, the first threshold may refer to a preset confidence threshold used to determine that the operation target is unique and has high credibility. When the comprehensive matching score of the candidate webpage element ranked first exceeds this threshold, the system may no longer perform subsequent fine-grained analysis steps, but may directly output it as the target webpage element, thereby improving the response speed of the processing flow.

[0086] According to some embodiments, step S5092, determining the target webpage element based on the comprehensive matching degree of each of the multiple candidate webpage elements, may include: in response to determining that the comprehensive matching degree of each of the multiple candidate webpage elements is less than a second threshold, or that the comprehensive matching degree of each of the multiple candidate webpage elements is greater than the second threshold and the difference is within a preset range, cropping extended visual fragments of each of the multiple candidate webpage elements from the screenshot, wherein each extended visual fragment of the candidate webpage element includes the visual fragment of the webpage candidate element and its surrounding area; constructing question text for the extended visual fragment based on interactive operations; and inputting the extended visual fragment and the question text into a multimodal model, and determining the target webpage element from the multiple candidate webpage elements based on the output of the multimodal model.

[0087] Therefore, by introducing a refined analysis sub-process based on a multimodal model, situations where there is conflict or insufficient differentiation between intent and visual information can be effectively resolved. This conflict resolution mechanism is triggered only in low-confidence scenarios, ensuring the system's operational accuracy in complex environments while avoiding the cost issues caused by frequent calls to high-overhead models.

[0088] In some embodiments, the second threshold refers to a preset confidence limit for identifying low-confidence or ambiguous scenarios. An extended visual segment refers to an image region that extends outward by a certain number of pixels from the original visual segment to include more surrounding contextual information. Question text refers to a natural language description generated based on the current operational intent and the features of candidate elements, used to guide the multimodal model in making judgments.

[0089] In some embodiments, when the system cannot determine candidate elements with a significant advantage through weighted calculation, it can automatically expand the cropping range to obtain an extended visual fragment containing adjacent elements. The system can further convert the current interactive operation command into a specific question-answer pair and simultaneously submit the image data and question text to a multimodal model with visual understanding capabilities. In some embodiments, the system can re-lock onto the target among candidate elements based on the logical reasoning results fed back by the multimodal model, thereby overcoming the recognition bottleneck caused by simply relying on basic feature matching.

[0090] According to some embodiments, the web page processing method may further include: performing visibility verification on the visual segments of each of a plurality of candidate web page elements, and removing candidate web page elements that fail the visibility verification.

[0091] Therefore, by performing visibility checks on visual fragments of candidate web page elements and removing invisible elements, invalid nodes that exist in the document object model tree but are not actually displayed on the screen (such as elements that are occluded, have zero transparency, or are located outside the viewport) can be effectively eliminated. This pre-filtering mechanism reduces the computational load of subsequent matching calculations, improves the accuracy of target web page element identification, and saves computational overhead caused by invalid operations.

[0092] In some embodiments, visibility verification refers to the process of verifying whether a webpage element actually exists and can be perceived in a screenshot based on visual features or rendering attributes. Failing visibility verification usually means that the candidate webpage element cannot be observed by the user or agent in the current visual context, and therefore lacks interactivity.

[0093] In some embodiments, visibility verification of visual segments of multiple candidate web page elements may include: the system analyzing the pixel distribution characteristics of each visual segment. Specifically, the system may check whether the visual segment is mostly composed of blank pixels or a single background color, or it may calculate the color histogram of the visual segment to verify whether it has obvious boundary or contrast features.

[0094] In some embodiments, the system can also perform validation by combining the style attributes of the webpage. For example, the system can obtain the transparency attribute or display status attribute of the candidate webpage element. If the transparency is lower than a preset value or the display status is marked as hidden, the system can determine that the element has failed the visibility validation.

[0095] In some embodiments, the system can dynamically adjust the candidate list based on the verification results. For candidate webpage elements determined to be invisible, the system can remove them from the current candidate list, thereby ensuring that subsequent intent matching and visual matching are performed only on elements with real visibility. In this way, the system can avoid sending operation instructions to unclickable or invalid coordinate areas, improving the success rate of interactive operations.

[0096] According to another aspect of this disclosure, a web page data processing apparatus is provided. For example... Figure 6As shown, the device 600 includes: a first determining unit 610 configured to determine an interactive operation for a target webpage; an acquiring unit 620 configured to acquire a screenshot of the target webpage and attributes of multiple webpage elements in the target webpage, the attributes indicating the functions of the corresponding webpage elements; a filtering unit 630 configured to filter multiple candidate webpage elements related to the interactive operation from the multiple webpage elements based on the attributes of the multiple webpage elements; a cropping unit 640 configured to determine the position and size of the multiple candidate webpage elements and crop visual fragments of the multiple candidate webpage elements from the screenshot; a second determining unit 650 configured to determine the degree of matching between the attributes of the multiple candidate webpage elements and the intent of the interactive operation; a third determining unit 660 configured to determine the degree of visual matching between the visual fragments of the multiple candidate webpage elements and the interactive operation; and a fourth determining unit 670 configured to determine the target webpage element from the multiple candidate webpage elements based on the degree of matching and the degree of visual matching, and perform an interactive operation on the target webpage element.

[0097] It is understood that the operation and effects of units 610-670 in device 600 can be referred to the description of steps S201-S207 above, and will not be repeated here.

[0098] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0099] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0100] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0101] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0102] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0103] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods, processes, and / or processes described above. For example, in some embodiments, these methods, processes, and / or processes may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods, processes, and / or processes described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform these methods, processes, and / or processes by any other suitable means (e.g., by means of firmware).

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0109] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0110] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0111] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A webpage data processing method, comprising: Determine the interactive actions for the target webpage; Obtain a screenshot of the target webpage and the attributes of multiple webpage elements in the target webpage, wherein the attributes indicate the functions of the corresponding webpage elements; Based on the attributes of the multiple web page elements, multiple candidate web page elements related to the interactive operation are filtered from the multiple web page elements. Determine the position and size of the multiple candidate web page elements, and crop the visual fragments of the multiple candidate web page elements from the screenshot; Determine the degree of matching between the attributes of the multiple candidate web page elements and the intent of the interactive operation; Determine the visual matching degree between the visual fragments of the multiple candidate web page elements and the interactive operation; as well as Based on the intent matching degree and the visual matching degree, the target webpage element is determined from the plurality of candidate webpage elements, and the interactive operation is performed on the target webpage element.

2. The method according to claim 1, wherein, Obtaining a screenshot of the target webpage and the attributes of multiple webpage elements within the target webpage includes: Obtain the document object model tree of the target webpage and identify multiple interactive nodes in the document object model tree that can be interacted with. Extract the interaction node information of the plurality of interactive nodes from the document object model tree, and the parent container node information of the parent container node of each of the plurality of interactive nodes, wherein the interaction node information and the parent container node information each include at least one of the node name, node value, and node type of the corresponding node; and Based on the multiple interactive nodes, the multiple web page elements are constructed, wherein the attributes of each web page element include the interactive node information of the corresponding interactive node and the parent container node information of the parent container node of the interactive node.

3. The method according to claim 2, wherein, Determining the position and size of the plurality of candidate web page elements, and cropping visual fragments of the plurality of candidate web page elements from the screenshot, includes: Based on the document object model tree, determine the coordinate information of the interactive nodes corresponding to each of the multiple candidate webpage elements in the target webpage; and Based on the coordinate information of the corresponding interactive nodes, the position and size of the multiple candidate web page elements are determined.

4. The method according to claim 3, further comprising: The screenshot of the target webpage is segmented into regions to obtain multiple region images; Determine the importance of each of the multiple image regions; Based on the aforementioned importance, the compression ratio of each region's image is determined; as well as The multiple region images are compressed based on their respective compression ratios.

5. The method according to claim 4, wherein, Determining the position and size of the plurality of candidate web page elements, and cropping visual fragments of the plurality of candidate web page elements from the screenshot, includes: For each candidate webpage element, based on the position of the candidate webpage element, determine the corresponding region image among the plurality of region images; and Based on the size of the candidate webpage element, the corresponding region image is cropped to obtain a visual fragment corresponding to the candidate webpage element.

6. The method according to any one of claims 1-3, further comprising: Determine the task type of the interactive operation; as well as Based on the task type, determine the intent weight for the intent matching degree and the visual weight for the visual matching degree. Specifically, determining the target webpage element from the plurality of candidate webpage elements based on the intent matching degree and the visual matching degree, and performing the interactive operation on the target webpage element includes: The intent matching degree and the visual matching degree are weighted and summed based on the intent weight and the visual weight to obtain the comprehensive matching degree of each of the multiple candidate webpage elements; and The target webpage element is determined based on the overall matching degree of each of the multiple candidate webpage elements.

7. The method according to claim 6, wherein, Based on the overall matching degree of each of the multiple candidate webpage elements, the target webpage element is determined to include: In response to determining that the highest overall matching degree among the plurality of candidate web page elements is greater than a first threshold, the corresponding candidate web page element is determined as the target web page element.

8. The method according to claim 6, wherein, Based on the overall matching degree of each of the multiple candidate webpage elements, the target webpage element is determined to include: In response to determining that the overall matching degree of the plurality of candidate web page elements is less than a second threshold, or that the overall matching degree of the plurality of candidate web page elements is greater than the second threshold and the difference is within a preset range, an extended visual fragment of each of the plurality of candidate web page elements is cropped from the screenshot, and the extended visual fragment of each candidate web page element includes the visual fragment of the web page candidate element and its surrounding area. Based on the interactive operation, construct question text for the extended visual fragment; and The extended visual fragment and the question text are input into a multimodal model, and the target webpage element is determined from the multiple candidate webpage elements based on the output of the multimodal model.

9. The method according to any one of claims 1-3, further comprising: The visibility of each visual segment of the multiple candidate web page elements is verified, and the candidate web page elements that fail the visibility verification are removed.

10. A web page data processing apparatus, comprising: The first determining unit is configured to determine the interactive operation for the target webpage; The acquisition unit is configured to acquire a screenshot of the target webpage and attributes of multiple webpage elements in the target webpage, wherein the attributes indicate the functions of the corresponding webpage elements; The filtering unit is configured to filter multiple candidate web page elements related to the interactive operation from among the multiple web page elements based on the attributes of the multiple web page elements. The cropping unit is configured to determine the position and size of the plurality of candidate web page elements and crop visual fragments of the plurality of candidate web page elements from the screenshot; The second determining unit is configured to determine the degree of matching between the attributes of the plurality of candidate web page elements and the intent of the interactive operation; The third determining unit is configured to determine the visual matching degree between the visual fragments of the plurality of candidate web page elements and the interactive operation; as well as The fourth determining unit is configured to determine a target webpage element from the plurality of candidate webpage elements based on the intent matching degree and the visual matching degree, and to perform the interactive operation on the target webpage element.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

13. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-9.

Images