GUI (Graphical User Interface) page markup language and system oriented to large language model
By designing a GUI page markup language and system for large language models, the problem of difficulty in accurately obtaining the location of GUI page elements in the existing technology is solved, safer and more general information acquisition is achieved, and the reasoning ability of large language models is significantly improved.
Patent Information
- Application Number
- CN202411807487.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-10
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art is difficult to accurately obtain the location information of GUI page elements, resulting in poor performance of large language models in scenarios such as automatic execution and automated testing, and data security problems.
A GUI page markup language and system for large language models was designed to obtain accurate element position information by calculating the GUI page information. The system contains four text description module units: a summary of the page content, whether it is a foreground layer and foreground type, module/component, and specific elements and relationships between elements within the module/component.
Through this technical solution, the problem that the large model cannot accurately obtain the location of page elements is compensated, and a safer and more general GUI page information is achieved, which significantly improves the inference ability of the large language model and its performance on cross-modal datasets.
Smart Images
Figure CN119645403A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a support, in particular to a GUI page markup language and system oriented to a large language model, belonging to the technical field of intelligent control. Background Art
[0002] Intelligent automatic execution of software has a wide range of applications in the field of human-computer interaction, such as automated testing scenarios, RPA (robotic process automation), online services, etc.
[0003] Problems with existing technologies:
[0004] 1. Existing software assistants, such as Siri, mainly achieve automated operations through system backend access and function calls. This approach has a big disadvantage, which is that it needs to access the user's privacy and sensitive data, which may cause serious data security issues. It also does not meet the application scenarios such as software automated testing;
[0005] 2. With the rise of big model technology, there have been many research works on software automatic execution, such as AppAgent released by Tencent, but its success rate is far from enough to be put into production practice. The dilemma is as follows:
[0006] (1) The current cross-modal large language model lacks the ability to accurately output the coordinates of page elements. Without the accurate coordinates of the page elements, the execution end cannot perform the final execution.
[0007] (ii) The reasoning ability of the current open-source cross-modal models in China, such as qwen-vl and glm4v, is weaker than that of large language models. A method is needed to enable large language models to understand GUIs in order to implement more complex scenarios and tasks.
[0008] 3. Existing markup languages, such as XPath, XML, HTML, and JSON, are usually lengthy. Directly inputting them into the general big model will consume a large number of tokens, and the general big model has limited ability to understand them. In addition, due to the standardization issues of developers, H5 pages, etc., often lead to the lack of key page description information, resulting in the inability of the big model to accurately understand the page content. In comparison, directly converting the markup language based on the final image presented on the page is closer to the way humans input information when operating mobile phones, and can obtain more complete and accurate information and input it into the big model. Summary of the invention
[0009] The present invention aims at the technical problems existing in the prior art and provides a GUI page markup language and system for a large language model. The technical solution obtains accurate element position information by calculating the GUI page information, thereby making up for the problem that the current large model / cross-modal large model cannot accurately obtain the page element position.
[0010] In order to achieve the above-mentioned purpose, the technical solution of the present invention is as follows: a GUI page markup language and system for a large language model, the system comprising: a text description module unit of "page content summary", a text description module unit of "whether it is a foreground layer & foreground type", a text description module unit of "module / component", and a text description module unit of "specific elements and relationships between elements" in the module / component, a total of four module units, the outputs of the four module units are spliced together in natural language form, and together constitute a GUI page markup language for a large language model.
[0011] The text description module unit of "specific elements and relationships between elements within a module / component" includes descriptions of elements, their attributes, and relationships between elements. The text description module unit of elements including plain text and general icon meanings is denoted as LMmap and is marked in the text with the <> symbol.
[0012] Among them, LMmap is the result of semantic mapping of the original text obtained from OCR recognition. The mapping complies with the principles of identification uniqueness and semantic comprehensibility. The element representation and storage relationship is that the original text / icon element is converted into two structured data for storage in memory. One group consists of [ID, LMmap], where LMmap is the element representation passed to the large model (LLM), and the other group consists of [ID, boundingbox]. Boundingbox is used for the element representation subsequently passed to the execution end. The two groups of data are associated through ID, where ID is the unique identification code assigned to the element, and boundingbox is the coordinate position of the element. The element attribute description is used to describe the visual characteristics and functional characteristics of the element, and is spliced after the element with -. For example, element A-interaction button (functional characteristics), element B-activation option (visual characteristics), etc.
[0013] The relationship description between elements is used to describe the functional relationship between elements. It is mainly used in the description of form-based UI pages. A group of related elements are marked with []. Elements with question-answer relationship are recorded as QA relationship pairs, where the question element is recorded as Q and the answer element is recorded as A. For example: [ <lmmap-q> , <lmmap-a>, <LMmap-Button 1>
[0014] In the text description module unit of "Specific elements and relationships between elements", the calculation methods for element descriptions, element attribute descriptions, and descriptions of relationships between elements include three steps: basic element recognition, element attribute recognition, and recognition of relationships between elements. Among them, basic element recognition refers to the recognition of text elements (text) and non-text icons (icon) in the UI page. For text elements (text), the ocr algorithm can be used, and the recognition of non-text icons (icon) can be achieved through training with the yolo model. Among them, element attribute recognition includes: interactive button recognition, active option recognition.
[0015] Among them, the recognition of relationships between elements includes: layer / foreground recognition, module component recognition, in-module line order conversion, and recognition of QA input selection.
[0016] Among them, attribute recognition and relationship recognition can be implemented by a transformer network that supports both image input and text input. By simultaneously inputting the UI page and the ocr result, the dependency relationships and attributes between elements are output.
[0017] The training dataset construction method for the transformer model is as follows: First, classify according to software types, including mobile banking apps and shopping mall apps. Collect several apps of each type, intercept images, and use annotation tools to annotate the dependency relationships and attribute categories between elements to obtain the dataset.
[0018] The text description module unit of "Page content summary" is used to summarize the content and characteristics of the current page, including the title information of the page, and page attribute categories such as the current home page, entrance page, form page, etc., which can be classified in detail according to the specific software system.
[0019] Among them, the text description module unit of "Page content summary" is specifically composed of a prompt template and a trained cross-modal model. The cross-modal model is composed of a transfromer pre-trained model and a pre-trained ViT visual encoder.
[0020] Among them, the text description module unit of "Whether it is the foreground layer & foreground type" is used to describe the attribute information of the top-level page layer, including foreground layers such as pop-up windows and foreground layers of sub-function pages.
[0021] Among them, the text description module unit of "Module / Component" divides the page into modules or components by a target recognition model and consists of descriptions of elements within the module / component in different paragraphs.
[0022] A calculation method for a GUI page markup language and system for large language models
[0023] It includes four parts: basic element recognition, element attribute recognition, page attribute recognition and element relationship recognition.
[0024] Basic element recognition refers to the recognition of text elements (text) and icons without text (icons) in the UI page, which can be implemented using OCR plus target detection algorithm. Element attribute recognition includes: interactive button recognition, activation option recognition, and page attribute recognition is used to extract summary information and features of the page, including page title description and page type. Relationship recognition between elements includes: layer / foreground recognition, module component recognition, module intra-row sequence conversion, and QA input selection pair recognition. The recognition model is a target recognition model, or a cross-modal model that supports image input. The training data set of the model is constructed as follows: first, classify according to software type, such as mobile banking apps, shopping mall apps, etc. Collect several apps of each type, take screenshots and annotate them to obtain the data set
[0025] Compared with the prior art, the present invention has the following advantages: by calculating the GUI page information, accurate element position information is obtained, which makes up for the problem that the current large model / cross-modal large model cannot accurately obtain the position of page elements. Information extraction directly based on the GUI page is safer and more universal than obtaining information based on back-end access, more accurate than obtaining information based on xpath, and more efficient in information description. Through a GUI page markup language and system for a large language model, a complete and accurate visual understanding of the page is provided for the large language model, which makes up for the problem that the large language model cannot perceive pictures, is conducive to the reasoning ability of the large language model, and reduces the difficulty of retraining the industry's large model on a cross-modal data set. A GUI page markup language and system for a large language model is also suitable as an input for a cross-modal large model, and can significantly improve the accuracy of model perception compared to simple image input. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is an example diagram of the GUI page information calculation method and markup language implementation.
[0027] Figure 2 Schematic diagram of element representation and storage relationship. DETAILED DESCRIPTION
[0028] In order to deepen the understanding of the present invention, the present embodiment is described in detail below with reference to the accompanying drawings.
[0029] Example 1: See Figure 1 , Figure 2 , a GUI page markup language and system for large language models, such as Figure 1 The "Markup Language Examples" in the document include: a text description of "page content summary", a text description of "whether it is a foreground layer and foreground type", a text description of "modules / components", and a text description of specific elements in the modules / components and the relationships between elements.
[0030] 1.1. The text description of "page content summary" is used to summarize the content and characteristics of the current page, for example, the title information of the page, the current page attribute type such as the home page, entry page, form page, etc., which can be classified in detail according to the specific software system.
[0031] 1.2. The text description of "Whether it is a foreground layer & foreground type" is used to describe the attribute information of the top page layer. For example: pop-up window type foreground layer, sub-function page type foreground layer.
[0032] 1.3. The text description of "module / component" is used to divide the page into modules or components, and the elements in different modules / components are described in different paragraphs.
[0033] 1.4. The text description of "specific elements within the module / component and the relationships between elements" includes descriptions of the elements, their attributes and the relationships between elements.
[0034] 1.4.1 Elements containing plain text and a text description of the meaning of a generic icon are denoted as LMmap and are identified in the text using the <> symbol.
[0035] 1.4.1.1 LMmap is the result of semantic mapping of the original text obtained by OCR recognition. The mapping complies with the principles of uniqueness of identification and semantic comprehensibility. Its element representation and storage relationship are as follows: Figure 2 As shown in the figure, the original text / icon element is converted into two types of structure data for storage in memory. One group consists of [ID, LMmap], where LMmap is the element representation passed to the large model (LLM). The other group consists of [ID, boundingbox]. The boundingbox is used for the element representation passed to the execution end. The two groups of data are associated through ID. Among them, ID is the unique identification code assigned to the element, and boundingbox is the coordinate position of the element.
[0036] 1.4.2 The element attribute description is used to describe the visual and functional characteristics of the element, and is concatenated with - after the element. For example, element A-interactive button (functional characteristics), element B-activation option (visual characteristics), etc.
[0037] 1.4.3 The description of the relationship between elements is used to describe the functional relationship between elements. It is mainly used in the description of form-based UI pages. A group of related elements are marked with []. Elements with a question-answer relationship are recorded as a QA relationship pair, with the question element recorded as Q and the answer element recorded as A. For example: [ <lmmap-q> , <lmmap-a>, <LMmap - Button 1>]
[0038] A calculation method for a GUI page markup language and system for large language models, as Figure 1 shown in the calculation steps in
[0039] 2.1 Among them, the basic element recognition refers to the recognition of text elements (text) and non - text icons (icon) in the UI page. It can be implemented by using the ocr plus object detection algorithm. Among them, the element attribute recognition includes, but is not limited to: interactive button recognition, active option recognition, etc. Among them, the page attribute recognition is used to extract the general information and features of the page, including, but is not limited to: page title description, page type. Among them, the element - to - element relationship recognition includes, but is not limited to: layer / foreground recognition, module component recognition, in - module row order conversion, QA input selection pair recognition (a set of element relationships with a question - answer format).
[0040] 2.1 The selection of the recognition model can be an object recognition model or a cross - modal model that supports image input. The construction method of the model's training dataset is as follows: First, classify according to the app type, such as mobile banking apps, shopping mall apps, etc. Collect several apps of each type and obtain the dataset by cropping and annotating the icons.
[0041] It should be noted that the above embodiments are not used to limit the protection scope of the present invention. Equivalent transformations or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention. < / lmmap-q> < / lmmap-q>
Claims
1. A GUI page markup language and system for a large language model, characterized in that: The system includes: a text description module unit of "page content summary" and a text description module unit of "whether it is a foreground layer & foreground type". There are four module units in total, namely, the text description module unit of "module / component" and the text description module unit of "specific elements and relationships between elements" within the module / component. The outputs of the four module units are spliced together in the form of natural language to form a GUI page markup language for a large language model.
2. A GUI page markup language and system for a large language model according to claim 1, characterized in that The text description module unit of "specific elements and relationships between elements within a module / component" includes descriptions of elements, their attributes, and the relationships between them. The elements include plain text and common icon meanings of text description module units, denoted as LMmap, and are identified in the text with the <> symbol. LMmap is the result of semantic mapping of the original text obtained from OCR recognition. The mapping complies with the principles of identity uniqueness and semantic comprehensibility. The element representation and storage relationship is as follows: the original text / icon element is converted into two structured data for storage in memory. One group consists of [ID, LMmap], where LMmap is the element representation passed to the large model (LLM), and the other group consists of [ID, boundingbox]. Boundingbox is used for the element representation subsequently passed to the execution end. The two groups of data are associated through ID, where ID is the unique identification code assigned to the element and boundingbox is the coordinate position of the element. The element attribute description is used to describe the visual and functional characteristics of the element, and is concatenated after the element with -. The relationship description between elements is used to describe the functional relationship between elements. It is mainly used in the description of form-type UI pages. A group of related elements are marked with [] symbols. Elements with question-answer relationships are recorded as QA relationship pairs, with question elements recorded as Q and answer elements as A.
3. A GUI page markup language and system for a large language model according to claim 2, characterized in that In the text description module unit of "Specific Elements and Relationships Between Elements", the calculation method of elements, element attribute descriptions and descriptions of relationships between elements includes three steps: basic element identification, element attribute identification, and relationship identification between elements. The basic element recognition refers to the recognition of text elements (text) and icons without text (icon) in the UI page. The text element (text) can be recognized by OCR algorithm, and the recognition of icons without text can be realized after training with Yolo model. Element attribute recognition includes: interactive button recognition, activation option recognition, The identification of the relationship between elements includes: layer / foreground identification, module component identification, module internal line sequence conversion, QA input selection pair identification, Attribute recognition and relationship recognition can be implemented by a transformer network that supports image input and text input, while inputting UI pages and OCR results, and outputting dependencies and attributes between elements. The training data set for the transformer model is constructed as follows: first, the software is classified according to the type, including mobile banking apps and shopping mall apps. Several apps of each type are collected, images are captured, and the dependencies and attribute categories between elements are annotated using annotation tools to obtain the data set.
4. A GUI page markup language and system for a large language model according to claim 1, characterized in that: The text description module unit of "Page Content Summary" is used to summarize the content and characteristics of the current page, including the page title information, the current page attribute type such as home page, entry page, form page, etc., which can be classified in detail according to the specific software system. The text description module unit of "page content summary" is specifically composed of a prompt word template and a trained cross-modal model, wherein the cross-modal model is composed of a transfromer pre-trained model and a pre-trained ViT visual encoder.
5. A GUI page markup language and system for a large language model according to claim 1, characterized in that: The text description module unit of "whether it is a foreground layer & foreground type" is used to describe the attribute information of the top page layer, including the pop-up window type foreground layer and the sub-function page type foreground layer.
6. A GUI page markup language and system for a large language model according to claim 1, characterized in that: The text description module unit of "module / component" divides the page into modules or components by a target recognition model, and is composed of descriptions of elements in the module / component in different paragraphs.