Page tagged text calculation method based on multi-modal large language model

Abstract

The invention discloses a method for calculating a page tagged text based on a multi-modal large language model, which comprises the following steps of: 1, inputting X into an element identification module D, carrying out element identification, outputting an element list, and recording the element list as listenements; step 2, listelements are transmitted into a cue word-representation format module P, and a complete cue word is spliced and recorded as prompt; 4, prompt is transmitted to a teacher large model-information extraction and representation module, and a marked text Y of a page is output; 5, manually auditing the output text, correcting errors, and outputting Y '; step 6, forming a data set by (X, Y '), and finely adjusting the cross-modal large model M of the small parameters to obtain a fine-adjusted calculation model M of the industry page marking text; and step 7, a forward reasoning stage: inputting the X into D, outputting listeemens, inputting the listeemens into P, outputting prompt, inputting the prompt into M, and outputting a page marking text of the X. The visual understanding and expression ability of the tip large model can be realized by the multi-modal large language model with small parameter quantity in a specified scene at low cost.

Description

Technical Field

[0001] This invention belongs to the field of large language model technology, specifically relating to a method for calculating page markup text based on a multimodal large language model. Background Technology

[0002] With the rapid development of large language models and multimodal large language models, various intelligent agents and knowledge bases are emerging in all walks of life, bringing new hope for the innovative development of human society. High-quality industry knowledge bases and high-precision intelligent agents require large models to possess powerful multimodal information understanding capabilities and accurate reasoning abilities. The construction of various knowledge bases involves raw input text combining text and images. When extracting information from this text, it is required not only to output the content information of text and image tables, but also to record the structural information of the text in a reasonable way. Intelligent agents that rely on visual understanding for interaction (such as web agents and mobile agents) need to deeply understand the overall structure and detailed information of the input image in order to accurately and stably output the operation object and operation content.

[0003] Defects and shortcomings of existing technology: First, top-tier multimodal language models such as GPT-4O, Claude 3.7, and Sonnet perform well on the aforementioned tasks. However, these are all closed-source models and cannot be deployed privately. Open-source models still lag significantly behind top-tier closed-source models in practical applications.

[0004] Secondly, even top-tier closed-source models still have shortcomings in understanding certain specific scenarios. Summary of the Invention

[0005] To address the above problems, this invention proposes a method for calculating page markup text based on a multimodal large language model (the markup text refers to the descriptive and recording of page information in a specific language format, similar to the HTML language text used by webpage developers; here, the markup text is primarily for large models). This method can achieve the visual understanding and expressive capabilities of top-tier large models in specified scenarios using a multimodal large language model with a small number of parameters at low cost, and compensate for its shortcomings in specific scenarios.

[0006] The technical solution of the present invention is as follows: A method for calculating page markup text based on a multimodal large language model, comprising the following steps: Step 1: Input X into element recognition module D, perform element recognition, and output a list of elements, denoted as list_elements; Step 2: Pass list_elements to the prompt word-representation format module P, and concatenate them into a complete prompt word, denoted as prompt; Step 3: Input prompt into the teacher big model - information extraction and representation module, and output the marked text Y on the page; Step 4: Manually review the output text, correct any errors, and output Y'; Step 5: (X,Y') constitute the dataset, fine-tune the cross-modal large model M with small parameters, and obtain the fine-tuned computational model M for industry page markup text; Step 6, Forward Inference Phase: X inputs D, outputs list_elements; list_elements inputs P, outputs prompt; prompt inputs M, outputs X's page markup text.

[0007] Preferably, the marked text is a descriptive description and record of page information in a specific language format.

[0008] Preferably, the specific implementation method of the prompt word-representation format module P is as follows: prompt is a string with placeholders, which consists of three parts: an image, an element set, and output requirements. The image is the image format of the input page X, the element set is list_elements, and the output requirements specifically include: JSON structured information; a description of the three-layer relationship of the page: page structure, hierarchical relationship, and relationship between elements; and added attribute information. If a table is included, the table is output in a structured manner.

[0009] Preferably, it is mainly used to identify non-textual page features or content in order to output an element recognition model that aligns with human semantic descriptions. The implementation method includes using the YOLO model as the recognition model, setting the model's category labels to align with human semantic descriptions, performing classification labeling and training to obtain an element recognition model, and outputting element descriptions during forward inference.

[0010] Preferably, the implementation method includes, for a webpage or mobile GUI page, using common developer tools or automated testing tools, capturing native element information, including the DOM tree, and then mapping the element descriptions to semantic descriptions aligned with human understanding before outputting them.

[0011] Preferably, the method further includes identifying the sub-modules of the page before element identification in step one, dividing the page into multiple sub-modules, and then performing text tagging calculations on each sub-module in sequence.

[0012] Preferably, the teacher large model T-information extraction and representation module adopts the GPT-4O model, the industry page mark text calculation model M adopts the qwen2.5 vl 3b model, and the GRP0 loss function is used to fine-tune all parameters of the model.

[0013] Compared with the prior art, the beneficial effects of the present invention are: it can realize the visual understanding and expression capabilities of top-level large models in specified scenarios with a small number of parameters for multimodal large language models at low cost, and make up for the problem of insufficient accuracy in visual understanding of web and other GUI pages. Attached Figure Description

[0014] Figure 1 This is a flowchart of the calculation method described in this invention. Detailed Implementation

[0015] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0016] Example: Figure 1 As shown, a method for calculating page markup text based on a multimodal large language model is described as follows: Let the input text or image be X. First, X is input to the element recognition module D for element recognition, outputting an element list, denoted as list_elements. Second, list_elements is passed to the prompt word-representation format module P, concatenating them into a complete prompt word, denoted as prompt. Fourth, prompt is passed to the teacher large model T-information extraction and representation module, outputting the page markup text Y. Fifth, the output text is manually reviewed, errors are corrected, and Y' is output. Sixth, (X, Y') form a dataset, and the cross-modal large model M with small parameters is fine-tuned. The fine-tuned industry page markup text calculation model M is obtained. Seventh, in the forward inference stage, X is input to D, outputting list_elements; list_elements is input to P, outputting prompt; prompt is input to M, outputting the page markup text of X. Figure 1 As shown.

[0017] The markup text refers to the descriptive and recordable information on a page in a specific language format. (Similar to the HTML language text used by web pages to developers, the markup text here is primarily for the overall web page design.) The specific implementation of the prompt-representation format module P is as follows: `prompt` is a string with placeholders, consisting of an image, an element set, and output requirements. The image is the image format of the input page X. The element set is `list_elements`. The output requirements specifically include: JSON structured information; a description of the three-tiered relationship of the page: page structure, hierarchical relationship, and relationship between elements; added attribute information, including attributes such as "button", "title", and "clickable"; if tables are included, the tables are output in a structured format, which can be a data frame, Markdown, HTML, or other markup language formats.

[0018] 3. The features of the element recognition module D are: It is primarily used to identify non-textual page features or content, and output them in accordance with human-aligned semantic descriptions. Specifically, it can be implemented using a YOLO model for annotation and training, with category labels set to human-aligned semantic descriptions. For web pages or mobile GUI pages, it can also extract native element information and map element descriptions to human-aligned semantic descriptions for output.

[0019] 4. Furthermore, before element recognition in the first step, the present invention can first identify the sub-modules of the page, divide the page into multiple sub-modules, and then perform text marking calculations on each sub-module in turn.

[0020] 5. In the implementation case, the teacher big model T-information extraction and representation module adopts the GPT-4O model, and the industry page markup text calculation model M adopts the qwen2.5 vl 3b model. The GRP0 loss function is used to fine-tune the model with all parameters. In the web page understanding task, the accuracy of the page markup text after fine-tuning reaches 99% on the test set, which exceeds 90% of GPT4O.

[0021] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A computing method for page labeling text based on a multi-modal large language model, characterized in that, The method comprises the following steps: Step one, the X input element recognition module D performs element recognition and outputs an element list, denoted as list_elements; Step two, input list_elements into the prompt-word representation format module P to splice into a complete prompt, denoted as prompt; Step three, input prompt into the teacher large model-information extraction and representation module to output the marked text Y of the page; Step four, manually review the output text, correct errors, and output Y'; Step five, (X, Y') constitutes a data set, fine-tunes the cross-modal large model M with small parameters, and obtains the fine-tuned industry page marked text calculation model M; Step six, forward inference stage, X input D, output list_elements, list_elements input P, output prompt, prompt input M, output the page marked text of X.

2. The multi-modal large language model-based page markup text computing method according to claim 1, characterized in that, The marked text is a description and record of the page information in a specific language format. 3.The multi-modal large language model-based page markup text computing method of claim 1, wherein, The specific implementation method of the prompt-word representation format module P is that prompt is a string with placeholders, which is composed of three parts: picture, element set and output requirement, wherein the picture is the picture format of the input page X, the element set is list_elements, and the output requirement specifically includes: json structured information; describe the three-layer relationship of the page: page structure, hierarchical relationship, and relationship between elements; add attribute information, if there is a table, output the table structure.

4. The multi-modal large language model-based page markup text computing method according to claim 1, characterized in that, The element recognition module D is used to recognize the features or content of non-text pages to align the human semantic description for output.

5. The multi-modal large language model-based page markup text computing method according to claim 4, characterized in that, The implementation method includes using a yolo model as the recognition model, setting the class label of the model to align the human semantic description, performing classification annotation and training to obtain the element recognition model, and outputting the element description in the forward inference stage.

6. The multi-modal large language model-based page markup text computing method according to claim 4, characterized in that, The implementation method includes that for web pages or mobile GUI pages, the original element information is captured through common developer tools or automated testing tools, including DOM tree, and then the element description is mapped to the aligned human semantic description for output. 7.The multi-modal large language model based page markup text computing method of claim 1, wherein, The method further comprises, before element recognition, performing page submodule recognition to divide the page into multiple submodules, and then sequentially performing marked text calculation on each submodule. 8.The multi-modal large language model based page markup text computing method of claim 1, wherein, The teacher large model T-information extraction and representation module adopts a GPT-4O model, the industry page marked text calculation model M adopts a qwen2.5 vl 3b model, and a grpo loss function is used to fine-tune the model with all parameters.

Images