Web element web analysis method based on large model data enhancement

Abstract

The invention relates to the field of web interface analysis, in particular to a web element analysis method based on text data enhancement. In order to improve the performance insufficiency of an existing web agent on a real world webpage, the web element web analysis method based on large model data enhancement is provided, element extraction and integration are carried out on a running state webpage through a virtual DOM, corresponding elements fitting input semantics are screened out through a BERT model, and the elements are extracted and integrated. And finally, reasoning the screened elements by adopting a large language model to realize more accurate target element selection.

Description

Technical Field

[0001] The present invention relates to the field of web interface analysis, and in particular to a web element analysis method based on text data enhancement. Background Art

[0002] As modern websites provide more and more features, their complexity and learning difficulty are also increasing, which makes the Internet more difficult for users to use. On the other hand, the emerging research on web agents is trying to turn the entire Internet into an unprecedented powerful and versatile tool. These web assistants that can understand natural language instructions and retrieve relevant information from the web user interface (UI) can significantly improve user efficiency. With the development of large language models (LLMs), they have considerable potential in understanding HTML. Previous research has mainly focused on web agents driven by LLMs. Some web agents, such as Mind2Web[1], can even navigate autonomously to complete some complex tasks. It obtains a large amount of data sets by crawling real web pages, annotates the data according to user intentions and behaviors, and trains it with simple content of DOM nodes obtained from the data sets. It then completes the screening of a large amount of web page data, and then gives the first five screened data to the large model for user behavior decision-making, so that the large language model can independently complete a series of complex web page operations through method calls. In addition, other current research such as [2,3,4] is committed to the direction of combining page images and web page codes using multimodal methods combined with images. However, the accuracy of Mind2Web's filtering model on real-world websites is less than 20%, and the recall rate of the top 5 is less than 50%, resulting in a decrease in the success rate of subsequent tasks. Given the limited performance in page element filtering, these web agents have not yet become a practical solution for daily use. How to better filter web elements from complex real-world pages is becoming a huge challenge. Although some annotated datasets that are closer to real-world websites have appeared, how to better process these huge datasets to fit the smaller input window of the model has become a real problem.

[0003] [1]Deng X, Gu Y, Zheng B, et al. Mind2web: Towards a generalist agent for the web[J]. Advances in Neural Information Processing Systems, 2024, 36.

[0004] [2]Qinghong Lin K, Li L, Gao D, et al.ShowUI:One Vision-Language-ActionModel for GUI Visual Agent[J].arXiv e-prints,2024:arXiv:2411.17465.

[0005] [3]Verma G, Kaur R, Srishankar N, et al. AdaptAgent: Adapting MultimodalWeb Agents with Few-Shot Learning from Human Demonstrations[J]. arXiv preprintarXiv:2411.13451,2024.

[0006] [4]Liu J,Ou T,Song Y,et al.Harnessing Webpage UIs for Text-RichVisual Understanding[J].arXiv preprint arXiv:2410.13824,2024. Summary of the invention

[0007] The present invention aims to improve the performance deficiencies of existing web agents on real-world web pages, and proposes a web element parsing method based on large model data enhancement. This method extracts and integrates elements of running web pages through virtual DOM, uses the BERT model to filter out corresponding elements that fit the input semantics, and finally uses a large language model to infer the filtered elements to achieve more accurate target element selection.

[0008] The web element parsing method based on large model data enhancement includes the following steps:

[0009] S1 obtains a web page and a user task, wherein the web page includes a web element to be parsed;

[0010] Generate a virtual DOM tree composed of virtual DOM nodes based on the web page, generate a selector corresponding to each virtual DOM node, wherein the selector records a complete path from the root node of the virtual DOM tree to the corresponding virtual DOM node; and use all virtual DOM nodes constituting the virtual DOM tree and the selectors corresponding thereto as simplified web page information;

[0011] Traditional web element parsing methods often need to analyze the original HTML file to obtain web page elements. The text length of the original HTML file is generally much longer than the input parameter limit of the large language model. However, through the construction of virtual DOM and selectors, the information of web page elements can be further retained without occupying too much text length.

[0012] S2 constructs a BERT model, wherein the input of the BERT model is simplified web page information and user tasks; the output of the BERT model is a number of virtual DOM nodes that are most relevant to the user tasks; supervised training is performed on the BERT model to obtain a trained BERT model;

[0013] However, the current BERT does not have a pre-trained model that corresponds to user intent to WEB elements. Therefore, it is necessary to perform fine-tuning training on the corresponding annotated data generated by S1 on the basic pre-trained BERT model.

[0014] S3 inputs the simplified web page information and user tasks described in step S1 into the trained BERT model, obtains several virtual DOM nodes most relevant to the user tasks, and converts them into candidate web elements respectively;

[0015] By pre-training the BERT model, the number of optional nodes on a web page can be reduced to about dozens. Due to the limitations of the current dataset size and coverage, the model may have poor generalization reasoning capabilities on more open real-world web pages, so it needs to be fed into a large language model for further screening based on semantics.

[0016] S4 constructs a prompt word template, wherein the prompt word template includes an example part and a question part;

[0017] The example section includes: user task examples, several candidate WEB element examples, and large language model answer examples;

[0018] The large language model answer example is specifically: a WEB element example most relevant to the user task example selected from a number of candidate WEB element examples;

[0019] The question part includes: the user tasks obtained in S1 and the candidate WEB elements obtained in S3;

[0020] Questions are asked to the large language model based on the prompt word template, and the target web element is selected from the candidate web elements through the large language model.

[0021] The current advanced large language models often have strong in-context learning (ICL) capabilities, and can make better inferences given specified templates and examples. In the S1 method, the WEB elements not only retain their internal specific content through the virtual DOM, but also the attributes such as id and class in the selector often carry text information that is easy for developers to read. Therefore, through ICL, the large language model can often find the target element among many elements.

[0022] Preferably, in S1, generating a virtual DOM tree composed of virtual DOM nodes based on the web page specifically includes the following steps: rendering the web page using a declarative Web framework to obtain a virtual DOM tree composed of virtual DOM nodes.

[0023] Preferably, in S1, generating a selector corresponding to each virtual DOM node specifically includes:

[0024] (1) Based on the constructed DOM tree, find the current DOM node and construct the selector string of the current DOM node according to its node attributes such as class and id;

[0025] (2) Get the parent node of the current DOM node, take the parent node as the current node, and construct a selector string based on its node attributes such as class and id;

[0026] (3) Repeat step (2) recursively until the root node is reached;

[0027] (4) Concatenate all the selector strings obtained in the above steps to obtain the selector corresponding to the current DOM node in step (1).

[0028] Preferably, in S2, the BERT model is subjected to supervised training, specifically comprising: obtaining a training set simulating real-world web pages, wherein the data in the training set includes web pages and user tasks; obtaining simplified web page information using the method described in S1; inputting the simplified web page information and user tasks into a Cross-Encoder model, and calculating the similarity between each virtual DOM node in the simplified web page information and the user task using the method of calculating the similarity score using the Cross-Encoder model, thereby obtaining the corresponding relationship between the virtual DOM node and the user task, which is used for the supervised training of the BERT model.

[0029] Preferably, the declarative web framework includes any one of React, Vue.js and Elm.

[0030] Preferably, the training set simulating real-world web pages is any one of the Mind2Web dataset and the RUSS dataset.

[0031] Preferably, in S2, the BERT model includes: any one of a RoBERTa model, a DistilBERT model, an ALBERT model, and an ELECTRA model.

[0032] Preferably, in S4, the large language model includes: any one of the ChatGPT model, the Wenxinyiyan model, the Tongyiqianwen model, and the LLama model.

[0033] The present invention has the following beneficial effects:

[0034] (1) A semantic expression method for web elements is proposed. Through the virtual DOM, more detailed web element information is provided from the running state of the web page. Combined with the selector, the specific location information of the web element is provided for subsequent model training.

[0035] (2) A prediction model for user intent to web page elements is trained. The model can focus on the content and location of web elements. This more detailed information can provide more connections between user intent text and integrated web element text, thereby increasing the hit probability of the target web element.

[0036] (3) Through the contextual learning ability of the large language model, we can break through the relatively limited application scenario coverage and relatively limited data size of the dataset, and provide higher accuracy for the application of the model in real-world web pages. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The present invention proposes a web element parsing method based on large model data enhancement, and its process includes S1, S2, S3, S4 related processes, see Figure 1 shown.

[0040] S1 The semantic expression method of web elements runs the page in a simulated environment, obtains virtual DOM elements, and simplifies them

[0041] The Document Object Model (DOM) is a cross-platform, language-independent interface that treats HTML or XML documents as a tree structure where each node is an object that represents a part of the document. The DOM represents a document as a logical tree. Each branch of the tree ends with a node, and each node contains objects. And DOM methods allow programmatic access to the tree; using them, you can change the structure, style, or content of the document. Nodes can have event handlers (also called event listeners) attached to them. Once an event is triggered, the event handler is executed.

[0042] Virtual DOM is a lightweight JavaScript representation of the Document Object Model used in declarative web frameworks such as React, Vue.js, and Elm. Because generating a virtual DOM is relatively fast, any given framework can re-render the virtual DOM as many times as needed at a relatively low cost. The framework can then find the difference between the previous virtual DOM and the current virtual DOM and make only the necessary changes to the actual DOM. Compared to HTML source code, the virtual Dom can better represent the state and hierarchy of page elements. The following is a simplified virtual Dom node:

[0043] {

[0044] tagName:"SELECT",

[0045] id:"reservations-city-search-type",

[0046] innerText:"In-house dining\nTakeout\nDelivery\nEvents\nWinery\nAll",

[0047] 0:option,

[0048] 1:option,

[0049] 2:option,

[0050] 3:option,

[0051] 4:option,

[0052] 5:option

[0053] …

[0054] }

[0055] Selector: This is a tool used by Cascading Style Sheets (CSS) to render HTML elements. It can uniquely point to one or a group of HTML elements. Each element has its own unique selector, so the selector can provide the element with its location information in the DOM tree. Through recursive operations, the complete selector of the element starting from the root node can be obtained, and displayed in a chain style:

[0056] document>html>body>main#maincontent>div.pagecontent>...div.muiinputbase-

[0057] root>select#reservations-city-search-type

[0058] By combining the virtual DOM node and the selector, after simplification, comprehensive information about the location and content of page elements can be obtained. The following is an example:

[0059] {'selector: / html / main#maincontent / div.pagecontent / select#reservations-city-search-type,node:{select{option reservations trueIn-house dining}{option pickup takeout}{option

[0060] delivery}{option events}{option wineries}{option all}}}

[0061] In the past, web agents often only split the original HTML file to obtain web page elements, ignoring the role of virtual DOM and selectors. However, through the construction method of this patent, the information of web page elements can be further retained without occupying too much text length.

[0062] S2 training from user intent to web page elements

[0063] Due to the complexity of web pages, it is impossible to input all page elements into a large language model. The input parameters of existing large language models are generally 8-128k, which is far lower than the hundreds of thousands or even millions of texts on real web pages. Therefore, a smaller model is needed for pre-screening.

[0064] BERT is a language model. It uses self-supervised learning to represent text as a sequence of vectors. It uses an encoder-only Transformer architecture. BERT has multiple variants including RoBERTa, DistilBERT, ALBERT, ELECTRA, etc., which are not explicitly limited in this patent. BERT is trained by masked tag prediction and next sentence prediction. Through this training process, BERT can learn the potential representation of tags in their context. However, the current BERT does not have a pre-trained model that corresponds to user intent to web elements. For this reason, it is necessary to fine-tune the training on the corresponding labeled data generated by S1 on the basic pre-trained BERT model.

[0065] There are many current training sets that simulate real-world web pages, such as Mind2Web, RUSS, etc., which are not limited by this patent. The data in the training set includes web pages and user tasks; simplified web page information is obtained using the method described in S1;

[0066] The simplified web page information and user tasks are input into the Cross-Encoder model, and the similarity between each virtual DOM node in the simplified web page information and the user task is calculated by using the Cross-Encoder model to calculate the similarity score, and the corresponding relationship between the virtual DOM node and the user task is obtained, thereby obtaining the labeled data. The labeled data is as follows:

[0067] {

[0068] label:1,

[0069] selector: / html / main#maincontent / div.pagecontent / select#reservations-city-search-type,

[0070] node:{select{option reservations trueIn-house dining}{option pickup}{option

[0071] delivery}{option events}{option wineries}{option all}},task:

[0072] Please help me choose a restaurant in Beijing

[0073] Previous actions:

[0074] }

[0075] S3 filters candidate web elements

[0076] The simplified web page information and user tasks described in step S1 are input into the trained BERT model to obtain several virtual DOM nodes most relevant to the user tasks, and are converted into candidate web elements respectively; limited by the current data set size and coverage, the model may produce poor generalization reasoning ability on more open real-world web pages, so it needs to be sent to a large language model for further screening based on semantics.

[0077] We define an example as the set "{{ω,q},e}", where ω, q, and e represent the HTML (web page) of the current viewport, the user query reflecting the real user task, and the real UI element (candidate web element) related to the real user task. The BERT model Ψ outputs the UI element related to the real user task as follows:

[0078]

[0079] in, is the user interface element output by the BERT model (as a candidate web element), which is compared with e to evaluate the performance of the BERT model.

[0080] S4 Generalization of contextual learning based on large language models

[0081] The candidate web elements screened by S3 are subjected to contextual learning (ICL) of a large language model through a prompt word template. ICL refers to a specific prompt engineering method in which the model receives several examples of the task as part of the prompt. This patent does not focus on a specific large language model and does not limit the specific model type. Any of the ChatGPT model, Wenxin Yiyan model, Tongyi Qianwen model, and LLama model can be used.

[0082] Here is an example prompt template for contextual learning:

[0083] You are a web designer and need to help users identify possible web elements. Please select the top n most relevant ones from the provided web elements. In this embodiment, n=1.

[0084] Example section:

[0085] User task: Please order food for me

[0086] Current web element: [{…},…,{…}]

[0087] Your answer: Element 3.

[0088] Question:

[0089] User task: Please help me choose a restaurant in Beijing

[0090] Current web element: [{'selector: / html / main#maincontent / div.pagecontent /

[0091] select#reservations-city-search-type,node:{select…}},…,{}]

[0092] Your answer:

[0093] The above descriptions are only some specific implementation methods of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person familiar with the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A web element parsing method based on large model data enhancement, characterized in that: The following steps are involved: S1 obtains a web page and a user task, wherein the web page includes a web element to be parsed; Generate a virtual DOM tree composed of virtual DOM nodes based on the web page, generate a selector corresponding to each virtual DOM node, wherein the selector records a complete path from the root node of the virtual DOM tree to the corresponding virtual DOM node; and use all virtual DOM nodes constituting the virtual DOM tree and the selectors corresponding thereto as simplified web page information; S2 constructs a BERT model, wherein the input of the BERT model is simplified web page information and user tasks; the output of the BERT model is a number of virtual DOM nodes that are most relevant to the user tasks; supervised training is performed on the BERT model to obtain a trained BERT model; S3 inputs the simplified web page information and user tasks described in step S1 into the trained BERT model, obtains several virtual DOM nodes most relevant to the user tasks, and converts them into candidate web elements respectively; S4 constructs a prompt word template, wherein the prompt word template includes an example part and a question part; The example section includes: user task examples, several candidate WEB element examples, and large language model answer examples; The large language model answer example is specifically: a WEB element example most relevant to the user task example selected from a number of candidate WEB element examples; The question part includes: the user tasks obtained in S1 and the candidate WEB elements obtained in S3; Questions are asked to the large language model based on the prompt word template, and the target web element is selected from the candidate web elements through the large language model.

2. The web element parsing method based on large model data enhancement as claimed in claim 1, characterized in that: In S1, a virtual DOM tree consisting of virtual DOM nodes is generated based on the web page, which specifically includes the following steps: using a declarative Web framework to render the web page to obtain a virtual DOM tree consisting of virtual DOM nodes.

3. The web element parsing method based on large model data enhancement as claimed in claim 1, characterized in that: In S1, generating a selector corresponding to each virtual DOM node specifically includes: (1) Based on the constructed DOM tree, find the current DOM node and construct the selector string of the current DOM node according to its node attributes; (2) Get the parent node of the current DOM node, take the parent node as the current node, and construct a selector string based on its node attributes; (3) Repeat step (2) recursively until the root node is reached; (4) Concatenate all the selector strings obtained in the above steps to obtain the selector corresponding to the current DOM node in step (1).

4. The web element parsing method based on large model data enhancement as claimed in claim 1, characterized in that: In S2, supervised training is performed on the BERT model, specifically including: Obtain a training set simulating a real-world web page, wherein the data in the training set includes web pages and user tasks; obtain simplified web page information using the method described in S1; The simplified web page information and user tasks are input into the Cross-Encoder model, and the similarity between each virtual DOM node in the simplified web page information and the user task is calculated by using the Cross-Encoder model to calculate the similarity score, so as to obtain the corresponding relationship between the virtual DOM node and the user task, which is used for supervised training of the BERT model.

5. The web element parsing method based on large model data enhancement as described in claim 2 is characterized in that: The declarative web framework includes any one of React, Vue.js and Elm.

6. The web element parsing method based on large model data enhancement as described in claim 4 is characterized in that: The training set simulating real-world web pages is any one of the Mind2Web dataset and the RUSS dataset.

7. The web element parsing method based on large model data enhancement as claimed in claim 1, characterized in that: In S2, the BERT model includes: any one of the RoBERTa model, the DistilBERT model, the ALBERT model, and the ELECTRA model.

8. The web element parsing method based on large model data enhancement as claimed in claim 1, characterized in that: In S4, the large language model includes: any one of the ChatGPT model, the Wenxin Yiyan model, the Tongyi Qianwen model, and the LLama model.

Images