DOM (Document Object Model)-based element identification method and device

Abstract

The invention discloses a DOM-based element identification method and device, and the method comprises the steps: S1, filtering an obtained webpage original HTML according to a set rule to obtain a simplified HTML, and constructing a preliminary DOM tree according to the simplified HTML; s2, performing element type conversion, information fusion and icon information interpretation on webpage elements in the preliminary DOM tree to obtain an extracted DOM tree; and S3, inputting the extracted DOM tree as a part of cue words into a multi-modal large language model MLLM to obtain recognized elements. On the basis of analyzing an original HTML (Hypertext Markup Language), necessary information is reserved, picture description information is supplemented, data integrity is ensured, the MLLM can better understand a webpage and predict operation through the simplified DOM tree, and the analysis efficiency is improved while the use cost is reduced.

Description

Technical Field

[0001] The present application belongs to the field of software technology, and relates to a DOM-based element recognition method and device, and specifically to an application of optimizing a multimodal large language model (MLLM) in identifying elements on a web page by constructing a DOM structure. Background Art

[0002] Online shopping, ticket booking, and information viewing have become an indispensable part of daily life. For unfamiliar websites, people may need to observe and try some controls on the page before they know how to meet their needs, which takes too much time. It is especially difficult for the elderly and other groups who are not familiar with browser operations to operate web pages by themselves. Therefore, some websites use web navigation services to guide people's operations, but these navigation services only provide fixed guidance operation methods and cannot guide users' special needs. In addition, most websites do not have navigation services.

[0003] Multimodal large language models MLLM, such as GPT4o, Claude3.5, Gork2, etc., have powerful text and image understanding capabilities. MLLM can automatically navigate and perform web page operations according to user needs. More and more technology companies provide customers with automatic web page navigation functions by building MLLM agents. However, most web page navigation MLLM agents must first observe web page information and then decide on web page operations. The information of the original web page HTML is too large, up to hundreds of KB or even several MB bytes; and MLLM has a limit on the input bytes, such as 128KB, and cannot support large-scale HTML input.

[0004] Therefore, developers of web navigation agents generally use a more simplified DOM tree to represent the HTML of a web page. The web page elements in the DOM tree are simplified from the original HTML and usually only contain a small number of attributes such as text content and type. However, there are a large number of non-interactive elements in the original web page. Even if the DOM tree is used, the number of bytes is still very high; even if the MLLM can accommodate the input volume, too much text will still cause the MLLM to have hallucinations or information loss. In addition, some types of elements may need to retain more element information, and an overly simplified DOM tree will also cause the MLLM to misjudge.

[0005] In addition, with the continuous advancement of Web technology, web page content has become more and more complex, and the types and styles of web page elements have become more diverse. At the same time, the implementation methods of web pages are also different. The same web page effect may be achieved through a variety of different HTML expressions. For example, a button can be represented by a type, such as <input type＝"checkbox"> You can also use the style sheet CSS to control the state switching of the check box and the display of the image, such as <i class＝"ico-checkbox-square">This variability and randomness also requires that the expression of the DOM tree be able to extract the most intuitive information of the elements. Summary of the invention

[0006] Purpose: In view of at least one of the above technical problems, the present application provides a DOM-based element recognition method and device, which improves the efficiency of element recognition by extracting effective and concise web page information and performing necessary text information fusion.

[0007] Technical solution: To solve the above technical problems, the technical solution adopted in this application is:

[0008] In a first aspect, the present application provides a DOM-based element identification method, the method comprising:

[0009] S1. Filter the original HTML of the acquired web page according to the set rules to obtain the simplified HTML, and construct a preliminary DOM tree according to the simplified HTML;

[0010] S2, performing element type conversion, information fusion, and icon information interpretation on the web page elements in the preliminary DOM tree to obtain a refined DOM tree;

[0011] S3. Input the refined DOM tree as part of the prompt word into the multimodal large language model MLLM to obtain the recognized element.

[0012] In a second aspect, the present application provides a DOM-based element recognition device, the device comprising:

[0013] The streamlining module is used to: filter the original HTML of the acquired web page according to the set rules to obtain the streamlined HTML, and construct a preliminary DOM tree according to the streamlined HTML;

[0014] The refining module is used to: perform element type conversion, information fusion, and icon information interpretation on the web page elements in the preliminary DOM tree to obtain a refined DOM tree;

[0015] The recognition module is used to: input the refined DOM tree as part of the prompt word into the multimodal large language model MLLM to obtain the recognized element.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0017] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0018] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0019] Beneficial effects: The DOM-based element recognition method and device provided by the present application have the following advantages: The present method proposes a method for constructing a DOM based on the original HTML of a web page, extracts effective and concise web page information, performs necessary text information fusion, uses a pre-trained multimodal model to identify image elements, and passes the refined data to MLLM for processing. Based on the analysis of the original HTML, the method retains the necessary information, supplements the information describing the image, ensures data integrity, and enables MLLM to better understand web pages and predict operations through the simplified DOM tree, thereby reducing the cost of use and improving analysis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of an application scenario of a method according to an embodiment of the present application;

[0021] Figure 2 Schematic diagram of a DOM-based element recognition method according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a DOM-based element recognition device provided according to an embodiment of the present application;

[0023] Figure 4 A schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and cannot be used to limit the protection scope of the present application.

[0025] In this application, the professional terms are explained as follows:

[0026] Multimodal Large Language Model (MLLM): Using a powerful large language model (LLM) as the "brain", various multimodal tasks can be performed. The large language model LLM is a deep learning model trained with a large amount of text data, with billions of parameters, capable of understanding and processing natural language. It is a general natural language processing (NLP) model.

[0027] GPT-4o is a language model released by OpenAI for the chatbot ChatGPT, which "can reason about audio, vision, and text in real time.

[0028] Prompt words: text content used to prompt MLLM to provide targeted answers, usually including system prompts used to prompt MLLM in what role, in what scenario, and in what format to generate answers in the current context window; user prompts generally refer to the conversation content entered by the user, used to prompt MLLM to complete tasks, including but not limited to daily conversations, answering questions, translation, writing, and coding.

[0029] Before describing the embodiments of the present application in detail, a specific scenario example is used to reveal the scenario in which the technical solution of the present application can be used. In a possible scenario example, specifically, Figure 1 As shown, the user performs specific operations on the client 110, and a data collection device and a multimodal large language model MLLM are deployed on the client 110. The data collection device is used to collect the original HTML of the web page obtained by the playwright tool on the client 110, filter the obtained original HTML of the web page according to the set rules to obtain the streamlined HTML, and build a preliminary DOM tree according to the streamlined HTML; perform element type conversion, information fusion, and icon information interpretation on the web page elements in the preliminary DOM tree to obtain a refined DOM tree; input the refined DOM tree as a part of the prompt word into the multimodal large language model MLLM to obtain the recognized elements.

[0030] The multimodal large language model MLLM can run on the client terminal side or the server side. The method of the present application can be configured in the multimodal large language model MLLM on the client terminal / or server side, and the client terminal and the server are connected through network transmission to realize multi-terminal interaction.

[0031] The server may be an electronic device with certain computing and processing capabilities. For example, the server may be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server may also be a new technical means capable of realizing the corresponding functions of the embodiments of the specification. For example, it may be a new form of "server" based on quantum computing.

[0032] The client terminal may be an electronic device with network access capability. Specifically, for example, the terminal may be a desktop computer, a tablet computer, a laptop computer, a smart phone, etc. Alternatively, the terminal may also be software that can run in the electronic device.

[0033] The network may be any type of network that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. The one or more networks may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0034] Example 1: Figure 2 As shown, this embodiment provides a DOM-based element recognition method, which can be applied to a multimodal large language model MLLM, which can run in a client terminal. The DOM-based element recognition method includes:

[0035] Step S1: Filter the acquired original HTML of the web page according to the set rules to obtain the simplified HTML, and construct a preliminary DOM tree according to the simplified HTML.

[0036] First, get the original HTML of the web page, such as using the playwright method to get the page content. It should be noted that the original HTML of the web page is usually very large and contains many elements that are not interactive or have no practical meaning, so these web page elements need to be filtered.

[0037] In some embodiments, in step S1, filtering the acquired original HTML of the webpage according to set rules to obtain the simplified HTML includes:

[0038] (1) Filter script types: Scripts are mainly codes rather than elements displayed on web pages. Scripts containing a large amount of code add unnecessary irrelevant text, affecting the effect of MLLM.

[0039] (2) Filter style type: style is used to define the style, not the web page content or interactive elements.

[0040] (3) Filter noscript types: noscript is only displayed when JavaScript is disabled and does not represent the normal content of the web page.

[0041] (4) Filter invisible elements, where the invisible elements include: 1) elements that cannot be displayed, such as elements with a display attribute of none; 2) hidden elements, such as elements with a visibility attribute of hidden; 3) completely transparent elements, such as elements with an opacity attribute of 0. Although these elements appear in the HTML content, users cannot see them on the web page and will not interact with them.

[0042] According to the above rules, the original HTML of the web page is filtered, and only the body node and its child nodes are retained to generate streamlined HTML, as shown in Table 1.

[0043] Table 1. Compact HTML and DOM tree

[0044]

[0045] Constructing a preliminary DOM tree according to the simplified HTML includes: adding a number to each node in the DOM tree, the number is used as a unique identifier of the web page element, and the web page element can be quickly located through the number.

[0046] S2, performing element type conversion, information fusion, and icon information interpretation on the web page elements in the preliminary DOM tree to obtain a refined DOM tree;

[0047] It should be noted that due to the diversity and randomness of web page HTML, some element expressions in the preliminary DOM tree obtained by S1 lack intuitive information and need to be refined. This solution refines DOM element information from three perspectives: element type conversion, information fusion, and icon information interpretation.

[0048] In some embodiments, step S2 may specifically include:

[0049] S21. Convert the original tag types of the web page elements in the preliminary DOM tree into summarized tag types, where the summarized tag types include: button, text, input, icon, link and others; other types of elements are not processed in this method and are filtered out in subsequent processes.

[0050] The label types of web page elements with original label types such as button, datalist, drop-down box, option, checkbox, and submit are uniformly represented by button; the button is used to trigger the user to click or select an interactive operation.

[0051] The tag types of web page elements with original tag types such as table title caption, code code, container div, form form, heading level h1~h6, label text label, list item li, menu menu, navigation nav, ordered list ol, paragraph p, inline span, document title title, and unordered list ul are uniformly represented by text; text text is used to display the web page document structure and text information.

[0052] The tag types of web page elements whose original tag types are input and multi-line textarea are uniformly represented by input; input is used to accept various inputs from users.

[0053] The tag types of web page elements whose original tag types are image img, vector image svg, and canvas are uniformly represented by icon; icon is an element that uses pictures or icons to display information, which can be used for interaction and operation, and can help users quickly understand the operation and structure of the web page.

[0054] The tag types of web page elements whose original tag types are anchor a and link link are uniformly represented by link; the link link is used to connect web page content and resources and provide navigation information.

[0055] This method does not process web page elements whose original tag type is other tag types and have a low degree of interaction, so they are filtered out.

[0056] In the embodiment of the present application, in order to make it easier for MLLM to understand web page elements, the complex web page element types are summarized (Table 2), which can effectively reduce the complexity of tag types and enable MLLM to focus on user interaction and page layout rather than specific tag type details.

[0057] Table 2

[0058]

[0059]

[0060] S22, performing information fusion on web page elements whose summary tag types are input and link;

[0061] In this embodiment, information fusion includes text fusion, hierarchical fusion and feature fusion. Through fusion, more information is displayed, allowing MLLM to better understand the function of the element. Usually, the element that needs text fusion is input or link.

[0062] For a web page element whose summary tag type is input, feature values ​​of multiple features are extracted and combined together as fused text information of the web page element; the multiple features include: built-in text innerText, input value value, placeholder placeholder, title title, name name, and auxiliary information aria-label, aria-labelledby, aria-describedby; although there are so many features used to describe the text information of the input element, there may still be a situation where the information of these features is empty.

[0063] To avoid this situation, when the information of the plurality of features is empty, the elements adjacent to the input web page element are obtained, and the input web page element is merged with the adjacent elements associated with the information as the fusion information of the input web page element in the DOM tree. <label for＝"un"> Username< / label> <input type＝"text"id＝"un"> , extract the text information "user name" of the label element and the label type "input" of the input element, and fuse these two information to get an element, represented as: input, user name, as the fusion information of the element in the DOM tree. For the input type element, not only the features of the element are fused, but also the information of the adjacent elements is fused.

[0064] For web page elements with the summary tag type of link, the text of the web page element itself (built-in text innerText or text content textContent attribute) and the hyperlink href and link relationship rel attributes in the web page element are obtained and merged into the text information of the link web page element. The attributes href and rel can well describe the relationship between the element link and the resource.

[0065] Table 3. Input type subtype mapping

[0066] Subtype type New Types illustrate button button Regular buttons, used to trigger events or submit forms file button File selection button for uploading files image button Image buttons, which are usually displayed with an image as part of a form submission radio button Radio buttons allow users to select one option from a set of choices range button Slider input control, used to select a value within a numeric range reset button Reset button to clear form input values submit button Submit button, used to submit form data checkbox button A checkbox allows users to select multiple options email input Email input control, validates the format is a valid email address number input Numeric input control, allows to enter a numeric value password input Password input control, the input content is displayed in mask form search input Search box, a text input control designed specifically for search functions tel input Phone number input control, verify that the format is a valid phone number text input Ordinary text input control, used to enter short text url input URL input control, verify that the format is a valid URL address readonly text Read-only text input control, the content cannot be modified but can be copied

[0067] S23. Icon information interpretation: interpret the web page elements with the summary tag type of icon, give semantic and functional descriptions, and add them into the DOM tree as text information of such web page elements.

[0068] Icons convey information through pictures, allowing users to quickly understand their functions and meanings without reading text. For example, a magnifying glass icon usually means "search", while a trash can icon means "delete".

[0069] The icon element in the DOM tree lacks a description of the element's function. The functional information of the icon is more reflected in its visual design. For example, the magnifying glass icon is marked as img or svg in the DOM tree, and its function "search" is not recorded in the DOM tree. The lack of an icon function description will cause MLLM to not understand the role of the icon element, especially when the icon itself lacks explanatory text.

[0070] Therefore, it is necessary to interpret the icons on the page, give each icon element a clear semantic and functional description, and add this information as text information of the icon class into the DOM tree.

[0071] In some embodiments, the semantic and functional description of the icon web page element is generated using a fine-tuned trained BLIP multimodal model. In this embodiment, in response to the problem of missing icon functions, this application collected 2,000 web page icons, used GPT4o to simply annotate the functions of these icons, and then manually corrected the annotated data. Using the data, the BLIP (Bootstrapped Language-Image Pre-training) multimodal model is fine-tuned and trained, and the fine-tuned model is used to generate descriptions of web page icons.

[0072] S3. Input the refined DOM tree into MLLM as part of the prompt word to obtain the recognized element.

[0073] The refined DOM tree incorporates more effective information, allowing different types of web page elements to display their characteristics from different dimensions, and strengthening MLLM's understanding of different web page elements. At the same time, the deletion of redundant and non-interactive elements greatly reduces the consumption of text prompts brought by the DOM tree, saving MLLM's reasoning time and cost.

[0074] Example 2: Based on Example 1, Figure 3 As shown, the present application provides a DOM-based element recognition device, the device comprising:

[0075] The streamlining module is used to: filter the original HTML of the acquired web page according to the set rules to obtain the streamlined HTML, and construct a preliminary DOM tree according to the streamlined HTML;

[0076] The refining module is used to: perform element type conversion, information fusion, and icon information interpretation on the web page elements in the preliminary DOM tree to obtain a refined DOM tree;

[0077] The recognition module is used to: input the refined DOM tree as part of the prompt word into the multimodal large language model MLLM to obtain the recognized element.

[0078] The specific functional implementation of each of the above modules can be found in the relevant contents of the method in Example 1 and will not be elaborated here.

[0079] Embodiment 3: Based on Embodiment 1, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 are implemented.

[0080] Embodiment 4: Based on embodiment 1, as Figure 4 As shown, this embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in Embodiment 1 when executing the computer program.

[0081] Embodiment 5: Based on Embodiment 1, this embodiment proposes a hardware system that applies a DOM-based element recognition method in Embodiment 1. The method is applied on the server side.

[0082] The terminal may be an electronic device with network access capability, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television, etc.

[0083] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0084] Embodiment 6: Based on Embodiment 1, this embodiment further provides a computer program product including instructions, which, when executed by a computer, enables the computer to execute a DOM-based element recognition method in Embodiment 1.

[0085] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the implementation methods of this specification, rather than limiting the scope of this application.

[0086] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0087] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0088] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0089] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0090] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0091] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0093] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0095] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0096] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0097] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A DOM-based element recognition method, characterized in that: The method comprises: S1. Filter the original HTML of the acquired web page according to the set rules to obtain the simplified HTML, and construct a preliminary DOM tree according to the simplified HTML; S2, performing element type conversion, information fusion, and icon information interpretation on the web page elements in the preliminary DOM tree to obtain a refined DOM tree; S3. Input the refined DOM tree as part of the prompt word into the multimodal large language model MLLM to obtain the recognized element.

2. The DOM-based element recognition method according to claim 1, characterized in that: In S1, the acquired original HTML of the webpage is filtered according to the set rules to obtain the simplified HTML, including: filtering the script type, style type, noscript type and invisible elements, retaining only the body node and its child nodes, and generating the simplified HTML.

3. The DOM-based element recognition method according to claim 2, characterized in that: The invisible elements include: 1) elements that cannot be displayed; 2) hidden elements; 3) completely transparent elements.

4. The DOM-based element recognition method according to claim 2, characterized in that: Constructing a preliminary DOM tree according to the simplified HTML includes: adding a number to each node in the DOM tree, the number is used as a unique identifier of the web page element, and the web page element can be quickly located through the number.

5. The DOM-based element recognition method according to claim 1, characterized in that: S1 specifically includes: S21, converting the original tag types of the web page elements in the preliminary DOM tree into summarized tag types, wherein the summarized tag types include: button, text, input, icon, link; S22, performing information fusion on web page elements whose summary tag types are input and link; S23. Interpret the web page elements whose summary tag type is icon, assign semantics and functional descriptions, and add them into the DOM tree as text information of such web page elements.

6. The DOM-based element recognition method according to claim 5, characterized in that: S21, converting the original tag type of the web page element in the preliminary DOM tree into a summarized tag type, including: The label types of web page elements with original label types such as button, datalist, drop-down box, option, checkbox, and submit are uniformly represented by button; The tag types of web page elements with original tag types such as table title caption, code code, container div, form form, heading level h1~h6, label text label, list item li, menu menu, navigation nav, ordered list ol, paragraph p, inline span, document title title, and unordered list ul are uniformly represented by text; The tag types of web page elements whose original tag types are input and multi-line textarea are uniformly represented by input; The tag types of web page elements whose original tag types are image img, vector svg, and canvas are uniformly represented by icon; The tag types of web page elements whose original tag types are anchor a and link link are uniformly represented by link; Filter out tags whose original tag type is other tag types.

7. The DOM-based element recognition method according to claim 5, characterized in that In S22, information fusion is performed on the web page elements whose summary tag type is input, including: For a web page element whose summary tag type is input, feature values ​​of multiple features are extracted and combined together as fused text information of the web page element; the multiple features include: built-in text innerText, input value value, placeholder placeholder, title title, name name and auxiliary information aria-label, aria-labelledby, aria-describedby; when the information of the multiple features is empty, the hierarchical adjacent elements of the input web page element are obtained, and the input web page element is fused with the adjacent elements associated with the information to obtain the fused information of the input web page element in the DOM tree.

8. The DOM-based element recognition method according to claim 5, characterized in that In S22, information fusion is performed on the web page elements whose summary tag type is link, including: For the web page element whose summary tag type is link, the built-in text innerText or text content textContent attribute of the web page element itself and the hyperlink href and link relationship rel attributes in the web page element are obtained and merged into the text information of the link link web page element.

9. The DOM-based element recognition method according to claim 5, characterized in that: In S23, the fine-tuned trained BLIP multimodal model is used to generate semantic and functional descriptions of icon web page elements.

10. A DOM-based element recognition device, characterized in that: The device comprises: The streamlining module is used to: filter the original HTML of the acquired web page according to the set rules to obtain the streamlined HTML, and construct a preliminary DOM tree according to the streamlined HTML; The refining module is used to: perform element type conversion, information fusion, and icon information interpretation on the web page elements in the preliminary DOM tree to obtain a refined DOM tree; The recognition module is used to: input the refined DOM tree as part of the prompt word into the multimodal large language model MLLM to obtain the recognized element.

Images