Web element web parsing method based on large model data enhancement

Abstract

The present application relates to the field of web interface analysis, and relates to a web element analysis method based on text data enhancement. In order to improve the performance of the existing web intelligent agent on the real world webpage, a web element analysis method based on large model data enhancement is proposed. The method extracts and integrates elements of the running state webpage through virtual DOM, uses the BERT model to filter out the corresponding elements that match the input semantics, and finally uses the large language model to reason the filtered elements, so as to realize more accurate target element selection.

Description

Technical Field

[0001] The present invention relates to the field of web interface parsing, and in particular to a web element parsing method based on text data enhancement. Background Art

[0002] As modern websites provide more and more functions, their complexity and learning difficulty are also increasing, which makes the Internet more difficult to use for users. On the other hand, the emerging research on web agents is trying to turn the entire Internet into an unprecedented powerful and versatile tool. These web assistants that can understand natural language instructions and retrieve relevant information from the web user interface (UI) can significantly improve user efficiency. With the development of large language models (LLMs), they have considerable potential in understanding HTML. Past research has mainly focused on web agents driven by LLMs. Some web agents, such as Mind2Web[1], can even navigate and complete some complex tasks autonomously. It obtains a large amount of data sets by crawling real web pages, and annotates the data according to user intentions and behaviors. It is trained by combining the simple content of DOM nodes obtained from the data sets, and then completes the screening of a large amount of web page data. The first five data are then handed over to the large model for user behavior decision-making, so that the large language model can independently complete a series of complex web page operations through method calls. In addition, other current research such as [2,3,4] is committed to using multimodal methods of image combination to combine page images and web page codes. However, Mind2Web's filtering model achieves less than 20% accuracy on real-world websites, and its top-5 recall is less than 50%, resulting in a decreased success rate for subsequent tasks. Given its limited performance in filtering page elements, these web agents have yet to become practical solutions for everyday use. Finding the best way to filter web elements from complex, real-world pages is becoming a significant challenge. While some annotated datasets that more closely resemble real-world websites have emerged, processing these massive datasets to fit within the model's smaller input window remains a significant challenge.

[0003] [1]Deng X, Gu Y, Zheng B, et al. Mind2web: Towards a generalist agent for the web[J]. Advances in Neural Information Processing Systems, 2024, 36.

[0004] [2]Qinghong Lin K, Li L, Gao D, et al.ShowUI: One Vision-Language-ActionModel for GUI Visual Agent[J].arXiv e-prints,2024:arXiv:2411.17465.

[0005] [3]Verma G, Kaur R, Srishankar N, et al. AdaptAgent: Adapting MultimodalWeb Agents with Few-Shot Learning from Human Demonstrations[J]. arXiv preprintarXiv:2411.13451,2024.

[0006] [4]Liu J,Ou T,Song Y,et al.Harnessing Webpage UIs for Text-RichVisual Understanding[J].arXiv preprint arXiv:2410.13824,2024. Summary of the Invention

[0007] This paper aims to improve the performance of existing web agents on real-world web pages by proposing a web element parsing method based on large-scale model data augmentation. This method extracts and integrates elements from running web pages using a virtual DOM. It then uses the BERT model to filter out elements that match the input semantics. Finally, a large language model is used to infer these filtered elements, achieving more accurate target element selection.

[0008] The web element parsing method based on large model data enhancement includes the following steps:

[0009] S1 obtains a web page and a user task, wherein the web page includes a web element to be parsed;

[0010] generating a virtual DOM tree composed of virtual DOM nodes based on the web page, generating a corresponding selector for each virtual DOM node, wherein the selector records a complete path from the root node of the virtual DOM tree to the corresponding virtual DOM node; and using all virtual DOM nodes constituting the virtual DOM tree and the corresponding selectors as simplified web page information;

[0011] The traditional web element parsing method needs to analyze the HTML original file to obtain the web page element. The text length of the HTML original file is generally much larger than the input parameter limit of the large language model. Through the construction mode of virtual DOM and selector, the information of the web page element can be further preserved without occupying too much text length.

[0012] S2 constructs a BERT model, wherein the input of the BERT model is the simplified web page information and the user task; the output of the BERT model is a plurality of virtual DOM nodes with the highest relevance to the user task; the BERT model is supervised trained to obtain a trained BERT model;

[0013] The current BERT does not have a pre-training model corresponding to the user intent to the WEB element. Therefore, the corresponding labeled data generated by S1 needs to be fine-tuned on the basis of the pre-training BERT model.

[0014] S3 inputs the simplified web page information and the user task in step S1 into the trained BERT model to obtain a plurality of virtual DOM nodes most relevant to the user task, and converts the virtual DOM nodes into candidate web elements respectively;

[0015] Through the pre-training BERT model, the number of optional nodes of the web page can be reduced to tens of nodes or so. Due to the limitation of the current data set size and coverage range, the model may have poor generalization reasoning ability in the more open real world web page. Therefore, the large language model needs to be further screened according to the semantics.

[0016] S4 constructs a prompt word template, wherein the prompt word template includes an example part and a question part;

[0017] The example part includes: a user task example, a plurality of candidate WEB element examples, and a large language model answer example;

[0018] The large language model answer example is specifically: a WEB element example selected from the plurality of candidate WEB element examples and most relevant to the user task example;

[0019] The question part includes: the user task obtained in S1 and the candidate WEB element obtained in S3;

[0020] According to the prompt word template, the large language model is questioned, and the target web element is selected from the candidate web element by the large language model.

[0021] Current state-of-the-art large language models often have strong in-context learning (ICL) capabilities, enabling them to make better inferences given specific templates and examples. In the S1 method, web elements not only retain their internal content through the virtual DOM, but also often include developer-readable textual information in selectors, such as id and class attributes. Therefore, through ICL, large language models can often find the target element among many others.

[0022] Preferably, in S1, generating a virtual DOM tree composed of virtual DOM nodes based on the web page specifically includes the following steps: rendering the web page using a declarative web framework to obtain a virtual DOM tree composed of virtual DOM nodes.

[0023] Preferably, in S1, generating a selector corresponding to each virtual DOM node specifically includes:

[0024] (1) Based on the constructed DOM tree, find the current DOM node and construct the selector string of the current DOM node according to its node attributes such as class and id;

[0025] (2) Get the parent node of the current DOM node, take the parent node as the current node, and construct a selector string based on its node attributes such as class and id;

[0026] (3) Repeat step (2) recursively until the root node is reached;

[0027] (4) All the selector strings obtained in the above steps are concatenated to obtain the selector corresponding to the current DOM node in step (1).

[0028] Preferably, in S2, the BERT model is subjected to supervised training, specifically comprising: obtaining a training set simulating real-world web pages, wherein the data in the training set includes web pages and user tasks; obtaining simplified web page information using the method described in S1; inputting the simplified web page information and user tasks into a Cross-Encoder model, and calculating the similarity between each virtual DOM node in the simplified web page information and the user task using the method of calculating similarity scores using the Cross-Encoder model, thereby obtaining a correspondence between the virtual DOM nodes and the user tasks, which is used for supervised training of the BERT model.

[0029] Preferably, the declarative web framework includes any one of React, Vue.js and Elm.

[0030] Preferably, the training set simulating real-world web pages is any one of the Mind2Web dataset and the RUSS dataset.

[0031] Preferably, in S2, the BERT model includes: any one of the RoBERTa model, the DistilBERT model, the ALBERT model, and the ELECTRA model.

[0032] Preferably, in S4, the large language model includes any one of the ChatGPT model, the Wenxin Yiyan model, the Tongyi Qianwen model, and the LLama model.

[0033] The present invention has the following beneficial effects:

[0034] (1) A semantic expression method for web elements is proposed. Through the virtual DOM, more detailed web element information is provided from the running state of the web page. Combined with the selector, the specific location information of the web elements is provided for subsequent model training.

[0035] (2) A prediction model for user intent to web page elements was trained. The model can focus on the content and location of web elements. Through this more detailed information, it can provide more connections between user intent text and integrated web element text, thereby increasing the probability of hitting the target web element.

[0036] (3) Through the contextual learning ability of the large language model, we can break through the relatively limited application scenario coverage and relatively limited data size of the dataset, and provide higher accuracy for the application of the model in real-world web pages. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0039] The present invention proposes a web element parsing method based on large model data enhancement, the process of which includes S1, S2, S3, S4 related processes, see Figure 1 shown.

[0040] S1 is a semantic expression method for web elements. It runs the page in a simulated environment, obtains virtual DOM elements, and simplifies them.

[0041] The Document Object Model (DOM) is a cross-platform, language-independent interface that views HTML or XML documents as a tree structure, where each node is an object representing a part of the document. The DOM represents a document as a logical tree. Each branch of the tree ends in a node, and each node contains an object. DOM methods allow programmatic access to the tree; using them, you can change the document's structure, style, or content. Nodes can have event handlers (also called event listeners) attached to them. When an event is triggered, the event handler is executed.

[0042] Virtual DOM is a lightweight JavaScript representation of the Document Object Model used in declarative web frameworks such as React, Vue.js, and Elm. Because generating a virtual DOM is relatively fast, any given framework can re-render the virtual DOM as many times as needed at a relatively low cost. The framework can then find the difference between the previous virtual DOM and the current virtual DOM and make only the necessary changes to the actual DOM. Compared to HTML source code, the virtual Dom can better represent the state and hierarchy of page elements. The following is a simplified virtual Dom node:

[0043] {

[0044] tagName:"SELECT",

[0045] id:"reservations-city-search-type",

[0046] innerText:"In-house dining\nTakeout\nDelivery\nEvents\nWinery\nAll",

[0047] 0:option,

[0048] 1:option,

[0049] 2:option,

[0050] 3:option,

[0051] 4:option,

[0052] 5:option

[0053] …

[0054] }

[0055] Selectors are tools used to render HTML elements in Cascading Style Sheets (CSS). They uniquely target one or a group of HTML elements. Each element has its own unique selector, which provides information about the element's location in the DOM tree. Through recursive operations, the complete selector for the element, starting from the root node, can be obtained, displayed as a chain:

[0056] document>html>body>main#maincontent>div.pagecontent>...div.muiinputbase-

[0057] root>select#reservations-city-search-type

[0058] By combining virtual DOM nodes and selectors, we can obtain comprehensive information about the location and content of page elements after simplification. The following is an example:

[0059] {'selector: / html / main#maincontent / div.pagecontent / select#reservations-city-search-type,node:{select{option reservations trueIn-house dining}{option pickup takeout}{option

[0060] delivery}{option events}{option wineries}{option all}}}

[0061] Previous web agents often only split the original HTML file to obtain web page elements, ignoring the role of virtual DOM and selectors. However, through the construction method of this patent, the information of web page elements can be further preserved without occupying too much text length.

[0062] S2 training from user intent to web page elements

[0063] Due to the complexity of web pages, it is impossible to input all page elements into a large language model. The input parameters of existing large language models are generally 8-128k, which is far lower than the hundreds of thousands or even millions of texts on real web pages. Therefore, a smaller model is needed for pre-screening.

[0064] BERT is a language model. It uses self-supervised learning to represent text as a sequence of vectors. It uses an encoder-only Transformer architecture. BERT has multiple variants, including RoBERTa, DistilBERT, ALBERT, ELECTRA, etc., which are not explicitly limited in this patent. BERT is trained through masked token prediction and next sentence prediction. Through this training process, BERT can learn the potential representation of tokens in their context. However, the current BERT does not have a pre-trained model that corresponds to user intent to web elements. To this end, fine-tuning training is required on the corresponding labeled data generated by S1 on the basic pre-trained BERT model.

[0065] There are many training sets that simulate real-world web pages, such as Mind2Web and RUSS, and this patent is not limited to them. The data in the training set includes web pages and user tasks; the simplified web page information is obtained using the method described in S1;

[0066] The simplified web page information and user tasks are input into the Cross-Encoder model. The Cross-Encoder model is used to calculate the similarity between each virtual DOM node in the simplified web page information and the user task. The correspondence between the virtual DOM node and the user task is obtained, and the labeled data is obtained. The labeled data is shown below:

[0067] {

[0068] label:1,

[0069] selector: / html / main#maincontent / div.pagecontent / select#reservations-city-search-type,

[0070] node:{select{option reservations true internal dining}{option pickup takeout}{option

[0071] delivery}{option events}{option wineries}{option all}},task:

[0072] Please help me choose a restaurant in Beijing

[0073] Previous actions:

[0074] }

[0075] S3 filters candidate web elements

[0076] The simplified web page information and user tasks described in step S1 are input into the trained BERT model to obtain several virtual DOM nodes most relevant to the user task, and each of them is converted into a candidate web element. Due to the limitations of the current dataset size and coverage, the model may have poor generalization reasoning capabilities on more open real-world web pages. Therefore, it needs to be fed into a large language model for further semantic screening.

[0077] We define an example as the set "{{ω,q},e}", where ω, q, and e represent the HTML (web page) of the current viewport, the user query reflecting the real user task, and the real UI element related to the real user task (candidate web element). The BERT model Ψ outputs the UI element related to the real user task as follows:

[0078]

[0079] in, is the user interface element output by the BERT model (as a candidate web element), which is compared with e to evaluate the performance of the BERT model.

[0080] S4 Generalization of contextual learning based on large language models

[0081] Candidate web elements screened by S3 are subjected to contextual learning (ICL) of a large language model using a prompt word template. ICL refers to a specific prompt engineering method in which the model receives several examples of the task as part of the prompt. This patent does not focus on a specific large language model and does not limit the specific model type. Any of the ChatGPT model, Wenxin Yiyan model, Tongyi Qianwen model, and LLama model can be used.

[0082] Here is an example prompt template for contextual learning:

[0083] You are a web designer and need to help users identify possible web elements. Please select the top n most relevant web elements from the provided web elements. In this embodiment, n=1.

[0084] Example section:

[0085] User Task: Please order food for me

[0086] Current web element: [{…},…,{…}]

[0087] Your answer: Element 3.

[0088] Question:

[0089] User task: Please help me choose a restaurant in Beijing

[0090] Current web element: [{'selector: / html / main#maincontent / div.pagecontent /

[0091] select#reservations-city-search-type,node:{select…}},…,{}]

[0092] Your answer:

[0093] The above description is only part of the specific implementation methods of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person familiar with the art within the technical scope disclosed in the present invention should be covered by the protection scope of the present invention.

Claims

1. A web element parsing method based on large model data enhancement, characterized by: The following steps are involved: S1 obtains a web page and a user task, wherein the web page includes a web element to be parsed; generating a virtual DOM tree composed of virtual DOM nodes based on the web page, generating a corresponding selector for each virtual DOM node, wherein the selector records a complete path from the root node of the virtual DOM tree to the corresponding virtual DOM node; and using all virtual DOM nodes constituting the virtual DOM tree and the corresponding selectors as simplified web page information; S2 builds a BERT model, wherein the input of the BERT model is the simplified web page information and the user task; the output of the BERT model is a number of virtual DOM nodes that are most relevant to the user task; supervised training is performed on the BERT model to obtain a trained BERT model; S3 inputs the simplified web page information and user tasks described in step S1 into the trained BERT model, obtains several virtual DOM nodes most relevant to the user tasks, and converts them into candidate web elements respectively; S4 constructs a prompt word template, wherein the prompt word template includes an example part and a question part; The examples section includes: user task examples, several candidate WEB element examples, and large language model answer examples; The large language model answer example is specifically: a WEB element example that is most relevant to the user task example selected from a number of candidate WEB element examples; The question part includes: the user tasks obtained in S1 and the candidate WEB elements obtained in S3; Questions are asked to the large language model based on the prompt word template, and the target web element is selected from the candidate web elements through the large language model.

2. The web element parsing method based on large model data enhancement according to claim 1 is characterized in that: In S1, a virtual DOM tree consisting of virtual DOM nodes is generated based on the web page, which specifically includes the following steps: using a declarative web framework to render the web page to obtain a virtual DOM tree consisting of virtual DOM nodes.

3. The web element parsing method based on large model data enhancement according to claim 1 is characterized in that: In S1, generating a corresponding selector for each virtual DOM node specifically includes: (1) Based on the constructed DOM tree, find the current DOM node and construct the selector string of the current DOM node according to its node attributes; (2) Get the parent node of the current DOM node, take the parent node as the current node, and construct a selector string based on its node attributes; (3) Repeat step (2) recursively until the root node is reached; (4) All the selector strings obtained in the above steps are concatenated to obtain the selector corresponding to the current DOM node in step (1).

4. The web element parsing method based on large model data enhancement according to claim 1 is characterized in that: In S2, supervised training of the BERT model is performed, specifically including: Obtain a training set simulating real-world web pages, wherein the data in the training set includes web pages and user tasks; obtain simplified web page information using the method described in S1; The simplified web page information and user tasks are input into the Cross-Encoder model. The Cross-Encoder model is used to calculate the similarity between each virtual DOM node in the simplified web page information and the user task. The correspondence between the virtual DOM node and the user task is obtained, which is used for supervised training of the BERT model.

5. The web element parsing method based on large model data enhancement as claimed in claim 2 is characterized in that: The declarative web framework includes any one of React, Vue.js and Elm.

6. The web element parsing method based on large model data enhancement as claimed in claim 4 is characterized in that: The training set simulating real-world web pages is any one of the Mind2Web dataset and the RUSS dataset.

7. The web element parsing method based on large model data enhancement according to claim 1 is characterized in that: In S2, the BERT model includes: any one of the RoBERTa model, the DistilBERT model, the ALBERT model, and the ELECTRA model.

8. The web element parsing method based on large model data enhancement according to claim 1 is characterized in that: In S4, the large language model includes any one of the ChatGPT model, the Wenxin Yiyan model, the Tongyi Qianwen model, and the LLama model.

Images