Multi-modal sample data generation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a multi-modal sample data generation method and device, electronic equipment and a storage medium, and the method comprises the following steps: for a target webpage, obtaining a webpage screenshot and element information of UI elements in the webpage; generating annotation information based on the element information of the UI element; taking the annotation information and a preset task cue word as text input, taking the webpage screenshot as picture input, and inputting the text input and the picture input into a large language model to obtain description information which is output by the large language model and is used for describing the webpage screenshot; and taking the webpage screenshot and the description information as sample data for training the multi-modal large language model. According to the embodiment of the invention, the generation efficiency of the sample data of the multi-modal large language model can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, and particularly relates to a multi-modal sample data generation method and device, electronic equipment and storage medium BACKGROUND

[0002] In the field of artificial intelligence, especially in the combined application of natural language processing and computer vision, multi-modal data generation technology is becoming increasingly important. These data often contain images, text and coordinate information, which are used to train intelligent systems to understand and generate content combining images and text. For example, when developing virtual assistants or automated testing tools, a large amount of annotated Graphical User Interface (GUI) data is needed to train the model so that it can recognize and operate web page elements.

[0003] However, existing GUI multi-modal data generation methods have some limitations. First, these methods often rely on manual annotation, which is not only time-consuming but also costly. Second, data generated by automated tools such as Optical Character Recognition (OCR) or crawlers may lack sufficient expression of GUI element interactivity and visual hierarchy. SUMMARY

[0004] In view of this, the present disclosure proposes a multi-modal sample data generation technical solution.

[0005] According to an aspect of the present disclosure, a multi-modal sample data generation method is provided, comprising:

[0006] For a target web page, obtaining a web screenshot and element information of UI elements in the web page;

[0007] Based on the element information of the UI elements, generating annotation information;

[0008] Inputting the annotation information and a preset task prompt word as text input and the web screenshot as picture input into a large language model to obtain description information for describing the web screenshot output by the large language model;

[0009] Taking the web screenshot and the description information as sample data for training a multi-modal large language model.

[0010] In one possible implementation, the generating annotation information based on the element information of the UI elements comprises:

[0011] Performing at least one filtering operation on the UI elements, including:

[0012] Eliminating UI elements with invalid web links;

[0013] Prune the UI element whose bounding box filling rate is lower than the threshold value;

[0014] Prune the UI element containing a URL link in the HTML attribute;

[0015] Generate annotation information using the element information of the filtered UI element.

[0016] In a possible implementation, the generating the annotation information comprises:

[0017] Integrate the UI elements with nested relationships, retain the position information of the outermost UI element, and integrate other element information of the internal child elements of the outermost UI element into the annotation information of the outermost element.

[0018] In a possible implementation, the preset task prompt word comprises at least one of the following:

[0019] The region description generation prompt word is used to guide the large language model to generate description information of the UI elements in the corresponding region.

[0020] The interactive dialogue generation prompt word is used to guide the large language model to generate an interactive question and answer pair containing the bounding box coordinates.

[0021] In a possible implementation, the sample data is used in a pre-training stage of multi-modal large model training.

[0022] The inputting the annotation information and the preset task prompt word as text input and the webpage screenshot as picture input into the large language model to obtain description information output by the large language model for describing the webpage screenshot comprises:

[0023] The inputting the annotation information and the region description generation prompt word as text input and the webpage screenshot as picture input into the large language model to obtain description information of the UI elements in the corresponding region generated by the large language model, wherein the description information of the UI elements in the corresponding region is used in the pre-training stage.

[0024] In a possible implementation, the sample data is used in an instruction fine-tuning stage of multi-modal large model training.

[0025] The inputting the annotation information and the preset task prompt word as text input and the webpage screenshot as picture input into the large language model to obtain description information output by the large language model for describing the webpage screenshot comprises:

[0026] The annotation information is spliced with an interactive dialogue generation prompt word as text input, the web screenshot is input as picture input, and the interactive question and answer pair generated by the large language model is obtained by inputting into the large language model, the interactive question and answer pair is used for the instruction fine-tuning stage.

[0027] In a possible implementation, the web screenshot and the description information are saved as sample data for training the multi-modal large language model.

[0028] The generated description information is cleaned and arranged to remove repetitive or invalid information.

[0029] The cleaned and arranged description information and the web screenshot are saved as sample data for training the multi-modal large language model.

[0030] According to another aspect of the present disclosure, a multi-modal sample data generation apparatus is provided, comprising:

[0031] An acquisition module is configured to acquire a web screenshot and element information of a UI element in a web page for a target web page.

[0032] An annotation information generation module is configured to generate annotation information based on the element information of the UI element.

[0033] A description information generation module is configured to input the annotation information and a preset task prompt word as text input, and input the web screenshot as picture input into a large language model to obtain description information output by the large language model for describing the web screenshot.

[0034] A sample data generation module is configured to save the web screenshot and the description information as sample data for training a multi-modal large language model.

[0035] In a possible implementation, the annotation information generation module is configured to:

[0036] The UI element is subjected to at least one filtering operation, including:

[0037] UI elements with invalid web links are removed.

[0038] UI elements with a boundary box filling rate lower than a threshold value are removed.

[0039] UI elements with a URL link in an HTML attribute are removed.

[0040] The element information of the filtered UI element is used to generate annotation information.

[0041] In a possible implementation, the annotation information generation module is configured to:

[0042] Integrate the UI elements with the nested relationship, retain the position information of the outermost UI element, and integrate other element information of the internal sub-element position information of the outermost element into the annotation information of the outermost element.

[0043] In a possible implementation, the preset task prompt word comprises at least one of the following:

[0044] The region description generation prompt word is used to guide the large language model to generate description information of the UI element in the corresponding region.

[0045] The interactive dialogue generation prompt word is used to guide the large language model to generate an interactive question and answer pair containing the bounding box coordinates.

[0046] In a possible implementation, the sample data is used for a pre-training phase of multi-modal large model training.

[0047] The description information generation module is configured to:

[0048] The annotation information and the region description generation prompt word are spliced as text input, the web screenshot is input as picture input, and the large language model is input to obtain description information of the UI element in the corresponding region generated by the large language model, and the description information of the UI element in the corresponding region is used for the pre-training phase.

[0049] In a possible implementation, the sample data is used for an instruction fine-tuning phase of multi-modal large model training.

[0050] The description information generation module is configured to:

[0051] The annotation information and the interactive dialogue generation prompt word are spliced as text input, the web screenshot is input as picture input, and the large language model is input to obtain an interactive question and answer pair containing the bounding box coordinates generated by the large language model, and the interactive question and answer pair is used for the instruction fine-tuning phase.

[0052] In a possible implementation, the sample data generation module is configured to:

[0053] The generated description information is cleaned and arranged to remove duplicate or invalid information.

[0054] The cleaned and arranged description information and the web screenshot are saved as sample data for training a multi-modal large language model.

[0055] According to another aspect of the present disclosure, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above method.

[0056] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.

[0057] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, or a non-transitory computer-readable storage medium carrying a computer program, which, when executed by a processor, implements the steps of the above method.

[0058] In the embodiments of the present disclosure, by obtaining the screenshot of the target webpage and the attribute information of the UI element, generating annotations based on these information, and inputting the annotations and the webpage screenshot into the large language model combined with the preset task prompt, the description information is obtained, which is used as the sample data for training the multi-modal large model. Figure 1 Thus, not only the demand for manual annotation is reduced and the efficiency of sample data generation is improved, but also the description information generated by the large language model enhances the diversity and richness of the sample data, so that the trained multi-modal model can more accurately understand and execute complex GUI interaction tasks, effectively solving the problems of high labor cost, insufficient data information, and lack of instruction data in the prior art.

[0059] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0061] Figure 1 A flowchart of a multi-modal sample data generation method according to an embodiment of the present disclosure is shown.

[0062] Figure 2 A flowchart of another multi-modal sample data generation method according to an embodiment of the present disclosure is shown.

[0063] Figure 3 A block diagram of a multi-modal sample data generation apparatus according to an embodiment of the present disclosure is shown.

[0064] Figure 4 A block diagram of a multi-modal sample data generation apparatus according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0065] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings represent the same or similar elements / functionally similar elements. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0066] As used herein, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof.

[0067] When an element is referred to as being "connected", "coupled", "responsive", or "in communication" to another element, it can be directly connected, coupled, responsive, or in communication to the other element, or intervening elements can be present.

[0068] Although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of the present inventive concept.

[0069] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0070] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the disclosure can be omitted. It will be appreciated that the present disclosure can be practiced with the exact details as

[0071] Figure 1 A flowchart of a multi-modal sample data generation method according to an embodiment of the present disclosure is shown. As shown in the figure, the method includes: Figure 1

[0072] In step S11, for a target webpage, a webpage screenshot and element information of a UI element in the webpage are acquired;

[0073] The target webpage can be any webpage. An open-source browser automatic crawling library can be used to process the target webpage and acquire a screenshot of the webpage. The screenshot can include complete visual content of the webpage, providing a basis for subsequent image analysis.

[0074] ​Meanwhile, the HyperText Markup Language (HTML) structure of the webpage can be parsed to extract element information of each User Interface (UI) element in the webpage. The element information can include at least one of the following: bounding box (bbox) coordinates of the element, HTML tag type, content, style, color, etc.

[0075] The element information is crucial for understanding the functionality and interaction logic of the UI elements. For example, the bounding box coordinates can help determine the position of the element in the webpage, the HTML tag and content can provide textual information of the element, and the style information and color can describe the visual appearance of the element.

[0076] By collecting these detailed element information, the characteristics of the UI elements in the webpage can be comprehensively captured, providing rich raw data for subsequent automated annotation and multi-modal data generation.

[0077] In step S12, annotation information is generated based on the element information of the UI elements.

[0078] The annotation information, also known as screen annotation, is used to describe the key UI elements in the webpage screenshot. The annotation information not only includes the position information of each UI element, such as the bounding box coordinates, but also includes the HTML code corresponding to each element. The bounding box coordinates consist of four numbers, representing the coordinate positions of the top-left and bottom-right corners of the element in the webpage screenshot.

[0079] For example, if there is a search box in the webpage, the HTML code of the search box and its bounding box coordinates in the webpage screenshot, such as [100, 200, 150, 250], will be extracted through step S11, indicating the coordinates of the search box in the screenshot. Through step S12, these information is generated as annotations, such as: bbox: [100, 200, 150, 250] element: "" <input type="text" name="s" class="search-input" autocomplete="off" placeholder="输入搜索内容"> .

[0080] The annotation information enables large language models to more accurately understand and describe the UI elements in the webpage screenshot, thereby generating higher quality description information. In addition, the process of automatically generating annotation information reduces the need for manual annotation, improves the efficiency of data generation, and enhances the consistency and accuracy of data through precise coordinates and HTML code, providing a foundation for training more intelligent and efficient multi-modal large models.

[0081] In step S13, the annotation information and a preset task prompt word are input as text, and the web screenshot is input as a picture into a large language model to obtain description information output by the large language model for describing the web screenshot.

[0082] After obtaining the web screenshot and the annotation information, the annotation information and a preset task prompt word are input as text, and the captured web screenshot is input as a picture into a large language model (LLM). The annotation information refers to a description of a key UI element in the web page, and can include HTML code of the UI element and position information in the screenshot, such as bounding box coordinates. The preset task prompt word is an instruction or question for guiding the large language model to complete a specific task, and the specific task can include describing a UI element and simulating user interaction.

[0083] The description information is used to describe the UI element in the current web screenshot through natural language, and can specifically include visual features (color, shape, and text) of the UI element, or functional semantics (search and submission) of the UI element, and is one-to-one corresponding to the position information bbox in the screen annotation to form high-quality sample data of image-text alignment, which can be directly used as a "language" label for subsequent training of a multi-modal large model.

[0084] For example, if the annotation information includes HTML code and bounding box coordinates [437, 34, 493, 17] of a search box, and HTML code and bounding box coordinates [343, 541, 655, 604] of a button, the preset task prompt word can be to require the large language model to describe the functions of these elements and how to use them in user interaction. After inputting this information into the large language model, the model will analyze the text and picture content and generate corresponding description information, such as "there is a blue button with the text 'Search' at the upper left corner coordinates [343, 541, 655, 604] for submitting a search request; and there is a white text input box with the placeholder text 'Enter search content' at the right side coordinates [437, 34, 493, 17] for entering keywords."

[0085] In step S14, the web screenshot and the description information are used as sample data for training a multi-modal large language model.

[0086] The web screenshot and the corresponding description information are combined into a piece of multi-modal sample data for training a multi-modal large model, and the multi-modal large model can process input containing vision and text.

[0087] In the sample data, webpage screenshots serve as the visual modality input, and descriptive information serves as the linguistic modality label; together, they constitute the training samples for image-text alignment. During training, the "webpage screenshot + descriptive information" pair is first input into the multimodal large-scale model. The screenshot is input to the visual encoder to obtain features, and the text decoder predicts the descriptive information character by character based on these features; the ground truth value for each character is the descriptive information itself. By predicting the loss between the descriptive information and the ground truth value, and then backpropagating the loss, the parameters of the multimodal large-scale model are updated.

[0088] After repeatedly training the large model with a large number of samples, the model parameters can map the content in the webpage screenshot to the corresponding descriptive information. Subsequently, no additional annotation is needed to understand new webpage screenshots and output the correct description.

[0089] In this embodiment of the disclosure, by obtaining screenshots of the target webpage and attribute information of UI elements, annotations are generated based on this information. These annotations are then combined with preset task prompts and webpage screenshots and input into a large language model to obtain descriptive information. The descriptive information and webpage screenshots are then used to obtain descriptive information. Figure 1 This data serves as sample data for training a large multimodal model. This not only reduces the need for manual annotation and improves the efficiency of sample data generation, but also enhances the diversity and richness of the sample data through descriptive information generated by the large language model. This enables the trained multimodal model to more accurately understand and execute complex GUI interaction tasks, effectively solving the problems of high manual costs, insufficient data information, and lack of instruction data in existing technologies.

[0090] In one possible implementation, generating annotation information based on the element information of the UI element includes: performing at least one of the following filtering operations on the UI element: removing UI elements with invalid webpage links; removing UI elements with bounding box fill rates below a threshold; removing UI elements whose HTML attributes contain Uniform Resource Locator (URL) links; and generating annotation information using the element information of the filtered UI elements.

[0091] A broken UI element is an element whose resource cannot be loaded successfully. For example, if the src or href that the element depends on returns 4xx / 5xx or times out, the browser cannot render the element and will display a placeholder or blank space. For instance, if the src of an image points to a deleted file, the page will only display a red cross icon.

[0092] Therefore, while crawling HTML, an access request can be sent to each webpage link, and the return code can be verified. If the returned code indicates that the webpage link is invalid, the element is marked as invalid and discarded. When training a large multimodal model, this can prevent the large model from learning "placeholder red crosses" or blank areas as valid UI elements, thereby reducing training noise and improving the model's accuracy in describing truly visible elements.

[0093] UI elements with a bounding box fill rate below the threshold can also be removed. The bounding box fill rate is used to measure the visual density of a UI element, that is, the proportion of the UI element in the bounding box. The specific calculation method is the ratio of the actual non-transparent pixel area inside the bounding box to the total area. This ratio can be determined according to the actual situation. For example, the threshold can be set to 0.3. For example, a button's bbox is 50×50 pixels, but the actual icon is only 10×10. Since the fill rate is below the threshold, it can be removed.

[0094] Therefore, by calculating the fill rate of UI elements in the bounding box, elements below the threshold can be removed. This can eliminate the coordinate deviation caused by "large box with small image" or "click hotspot is much smaller than the label box", making the bounding box consistent with the interactive area. This significantly improves the recognition accuracy of UI elements by the large model trained subsequently and reduces the click error of UI elements by the interactive large model.

[0095] In addition, UI elements whose HTML attributes contain URL links can be removed. These URL links can be absolute or relative paths. Specifically, regular expressions can be used to scan fields containing href or src, and if a match is found, the element can be removed.

[0096] Considering that large models do not output URLs when outputting descriptive information, removing UI elements whose HTML attributes contain URL links can prevent the model from treating irrelevant linked text as task instructions, reduce semantic drift, and improve the accuracy of the trained large model.

[0097] In this embodiment of the disclosure, after removing the above three types of filters, the remaining elements are all high-quality samples that are "renderable, interactive, and semantically relevant". The final generated annotation information is clean, aligned, and has high information density, which can be directly used for pre-training and instruction fine-tuning of multimodal large models, significantly improving training efficiency and the performance of downstream tasks.

[0098] In one possible implementation, generating annotation information includes: integrating UI elements with nested relationships, retaining the position information of the outermost UI element, and integrating the position information of its internal child elements and other element information into the annotation information of the outermost element.

[0099] In generating the annotation information, the UI elements with the nested relationship can be integrated first. The nested relationship here can be that a parent element contains multiple child elements, i.e., an outer element contains an inner element, which is in a hierarchical structure in a document object model (DOM) tree of HTML, and multiple HTML tags share the same visual area.

[0100] To avoid redundancy, only the bounding box (bbox) of the outermost element can be retained as the position information, i.e., the position information of the bounding box of the internally nested UI element is not retained, and all the content of the internal child elements except the position is uniformly merged into the description of the outer element.

[0101] For example, if the outermost element is detected <button>The position of the label is [343, 541, 655, 604], which is nested with one a label and one The final generated annotation information only keeps [343, 541, 655, 604] as the button position, but the HTML description of the button is integrated into < / button> <button class="search"> Search < / button> The content in the child element is retained.

[0102] In the embodiments of the present disclosure, in generating the annotation information, the UI elements with the nested relationship are integrated, only the position information of the outermost UI element is retained, and the element information other than the position information of all the internal child elements is integrated into the annotation information of the outermost element. In this way, the coordinates of the internally nested child elements can be eliminated, the inaccuracy of the position information is avoided, the training noise is reduced, in addition, all the semantics and interactive information of the child elements are still retained, so that the subsequent large model can not only locate the control but also understand the complex structure inside the control, thereby improving the accuracy of instruction execution.

[0103] In a possible implementation, the preset task prompt word includes at least one of the following: a region description generation prompt word for guiding the large language model to generate description information of the UI elements in the corresponding region; and an interactive dialogue generation prompt word for guiding the large language model to generate an interactive question and answer pair containing the bounding box coordinates.

[0104] The preset task prompt word is an instruction template for driving the large language model to complete GUI understanding and interactive tasks of different granularities. The structured text instruction controls the type and format of the data output by the large model. According to the training stage requirements of the multi-modal GUI agent, the present disclosure provides multiple prompt words.

[0105] The region description generation prompt word is used to guide the large language model to perform multi-dimensional description on the specified UI elements. The region description generation prompt word can guide the large language model to generate structured question and answer pairs (qa_pairs), guide the large language model to embed the bounding box coordinates of the UI elements into the question and embed the visual features and functional description into the answer when outputting the result, and construct data samples that can be directly used to train the GUI understanding model (multi-modal large model).

[0106] In generating the description information based on the region description generation prompt word, the large language model takes the web screenshot as the visual context basis, uses the screen annotation to provide the accurate spatial positioning of the UI element (in the form of <bbox>x1,y1,x2,y2< / bbox> bounding box coordinates), and the screen annotation of the region to generate the description information.

[0107] In the generation process, the region description generation prompt word can contain constraint rules for the results generated by the large language model, which contain at least one of the following:

[0108] While maintaining the question and answer relationship, the bounding box of the UI element is placed in the output result question, and the description information of the UI element needs to be included in the answer of the output result. Specifically, the element type recognition (button / input box / icon, etc.), visual feature extraction (color / shape / icon symbol), and text content description (visible text or placeholder) can be fused, for example, for a search box, generate "white rectangular input box with magnifying glass icon and 'Search' placeholder".

[0109] The constraint output result adopts JSON format to output the description information to ensure machine readability, for example, the output of the large language model is required to strictly follow the "question-answer" pair form, which can be structured as follows: { "qa_pairs": [{"question":"...","answer":"..."}]}.

[0110] In the constraint output result contained question, the position information of the UI element in the description information needs to be included, such as "elements in the description region <bbox> 66,85,353,196< / bbox> ".

[0111] The constraint only describes the visual content for pure picture elements, and prohibits the output of irrelevant information such as picture source, to avoid noise pollution of training data, for example, "red flame icon", but not similar to "picture source from……" and other replies.

[0112] Through the strong binding mechanism of spatial coordinates and semantic description, the traditional GUI data fragmentation problem is solved. When the model receives the "description <bbox> 731,83,932,187< / bbox> region" instruction, it will simultaneously associate the visual features (such as blue rectangles) in the corresponding position of the screenshot and the HTML semantics (such as Electric encyclopedia ) in the annotation, output the composite description "blue text link with 'Electric Encyclopedia' words", and realize the natural fusion of visual information and functional context.

[0113] The generated samples can be directly used as pre-training data for multi-modal large models, and the explicit embedding of bounding box coordinates in the question enables the model to learn the mapping logic of "spatial positioning → element description", laying a foundation for subsequent interactive intelligent agent instruction execution.

[0114] Interactive dialogue generation prompts are used to guide the large language model in constructing executable interface operation instructions. These prompts guide the large language model to generate structured question-and-answer pairs (qa_pairs), and in outputting results, to transform user intent into atomic operations and embed bounding box coordinate labels in the responses, thus constructing instruction samples that can be directly used to train interactive GUI agents.

[0115] When generating descriptive information based on interactive dialogue prompts, the large language model uses webpage screenshots as the visual context, leveraging screen annotations to understand the functional semantics (such as HTML tag attributes) and spatial positioning (using...) of UI elements. <bbox>x1,y1,x2,y2< / bbox> (Format the bounding box coordinates) to generate question-and-answer pairs that conform to real interaction logic.

[0116] During the generation process, the interactive dialogue generation prompts may include constraint rules on the results generated by the large language model, and these constraint rules may include at least one of the following:

[0117] User questions must be linked to actual interactive elements, and the generation of fictitious commands is prohibited. For example, a question cannot be generated for a real download button in a screenshot, such as {"user":"download file"}, but cannot be generated for a non-existent "share button".

[0118] Force the multimodal large model to return the original bounding box coordinate labels containing the target elements (format: <bbox>x1,y1,x2,y2< / bbox> ) and visual identifier descriptions, such as "Click on the location" <bbox> 120,85,180,120< / bbox> The blue 'Sign in' button, where the coordinate values ​​must be exactly the same as the screen annotation;

[0119] The output results are constrained to use JSON format to ensure machine parsing, strictly adhering to the structure {"qa_pairs":[{"user":"...","assistant":"..."}]}. For example, for the search function, the following is generated: {"user":"Where to enter keywords","assistant":"In..."} <bbox> 500,50,700,80< / bbox> Enter "} in the magnifying glass input box;

[0120] The operation description must be a single-step atomic action (such as "click" or "input"), and the generation of multi-step compound instructions (such as "click A first and then input B") is prohibited to ensure executability.

[0121] Accuracy is ensured through visual-HTML dual validation, for example, when HTML comments are... <button>Login< / button> Furthermore, when the screenshot shows a blue button, a description of "blue login button" is generated to avoid misrepresenting a gray button as blue.

[0122] Solve the problem of missing GUI instruction data through the strong binding mechanism of intent-operation-space: when the model processes the HTML annotation of the search box and the magnifying glass icon in the screenshot <input placeholder="输入关键词"> , generate the sample {"user":"search content","assistant":"click <bbox> 500,50,700,80< / bbox> the search box"} so that the question and answer pair contains user intent expression, functional positioning, and executable coordinates.

[0123] The generated sample can be directly used for instruction fine-tuning (SFT) of multi-modal large models, such as teaching the model to map "login account" to a specific operation of clicking <bbox> 120,85,180,120< / bbox> ; The explicit embedding of the bounding box label enables the model to learn the conversion ability from natural language instructions to spatial coordinates, providing a practical interaction basis for scenarios such as automated testing and accessibility services.

[0124] A specific example of generating region description prompt words is given below:

[0125] "Given a screenshot and screen annotation, I need you to generate region description question and answer pairs (qa_pairs): Given the bounding box of an element, provide a description of the corresponding region. The region description answer should include the type of UI element (button, input box, icon, drop-down box, etc.), as well as any visible text, and be simple and clear, for example:

[0126] 1) Text link with "About Us" label;

[0127] 2) Dark gray rectangular button with yellow bell icon and text "Reminder";

[0128] 3) Link with 'Popular Chart Release';

[0129] 4) Drop-down element with label "Brand" and down arrow indicating expansion;

[0130] 5) Text input box with placeholder text "What are you looking for?";

[0131] 6) A red flame icon followed by black font "BACK".

[0132] Please note:

[0133] 1. While maintaining the original question and answer relationship, put the bounding box of the UI element into the question and use the region description of the UI element in the answer.

[0134] 2、If there is an image in the bounding box, please briefly describe the image and do not include "Image source from..." in the response.

[0135] 3、The bounding box coordinates in the question should be in the format x1,y1,x2,y2, consistent with the screen annotations.

[0136] 4、Your output must strictly follow the JSON format: { "qa_pairs": [{"question": "...","answer": "..."},...]}."

[0137] Here is a specific example of generating prompt words for interactive dialogues:

[0138] "This is a set of instructions on how to generate question and answer pairs based on screenshots and screen annotations. The specific content is as follows:

[0139] For each UI element, it is necessary to combine its visual features and HTML annotations to better understand the element. Then, based on the full understanding of the page provided by the screenshots and screen annotations, design the question and answer pairs to simulate the interaction between the user and the webpage screen, and generate possible interactive question and answer pairs. Several examples are given:

[0140] 1、User: "I want to search for content", Assistant: "Click the search box x1,y1,x2,y2 to search"

[0141] 2、User: "How do I log in to my account", Assistant: "Click the 'Sign in' link located at x1,y1,x2,y2"

[0142] 3、User: "Download the file in the picture", Assistant: "Click the 'Download' button located at x1,y1,x2,y2"

[0143] 4、User: "Jump to the page of TV series in the suspense category", Assistant: "You can click the 'Suspense' link located at x1,y1,x2,y2"

[0144] 5、User: "Where should I enter my account number?", Assistant: "You can fill in the input box x1,y1,x2,y2"

[0145] At the same time, two points for attention are raised:

[0146] 1、Generate corresponding questions based on the UI elements in the screen annotations and screenshots, and do not generate irrelevant questions.

[0147] 2. The output must strictly follow the JSON format: { "qa_pairs": [{"user": "...", "assistant": "..."},...]}.

[0148] In the embodiments of the present disclosure, based on the same set of webpage screenshots and annotation information, by switching the prompt words, understanding labels and action labels can be simultaneously output, one picture is used for multiple purposes, and data utilization is improved; the region description prompt word can help the multi-modal large model to first learn to speak while looking at the picture, and the interactive dialogue generation prompt word can further teach the model to do things while looking at the picture, the two-level tasks are progressive, and the zero-shot adaptation time of the downstream task is significantly shortened.

[0149] In addition, the position information can be explicitly written into the question and answer pair, and the trained model can directly output the operable position, reducing subsequent parsing errors, improving the accuracy of picture understanding, and improving the click success rate in real webpage automation scenarios.

[0150] In addition, it should be noted that the present disclosure also provides a shared prompt word, which is a prompt word shared by region description generation and interactive dialogue generation, and provides common execution logic for the two types of tasks through unified multi-modal input specifications and role definitions.

[0151] The shared prompt word clearly indicates that the input of the large language model includes two core data: a webpage screenshot, as a visual context carrier; and a screen annotation (Screen Annotation), which provides structured UI element information (including HTML source code and normalized bounding box coordinates, with coordinate format [x_min, y_min, x_max, y_max] and numerical range limited to 0-999).

[0152] A specific example of the shared prompt word is given below:

[0153] "Your name is GUI-Expert, a user interface interaction assistant designed specifically for web pages.

[0154] As a virtual assistant, you can interact with users through the interface of the operating system, assist them in solving requests, and describe the content displayed in the screenshot, etc.

[0155] ##Usage Guide:

[0156] 1. Provide you with a screenshot of a webpage or desktop.

[0157] 2. Provide you with some key UI element annotations detected on the screenshot, labeled as "ScreenAnnotation".

[0158] 3. Each UI element has corresponding HTML code and a corresponding bounding box. The bounding box is represented by four numbers between 0 and 999.

[0159] Your task is to respond to the user request by looking at the screenshot.

[0160] Shared prompt words are used to guide the model to identify visual features (icon style, color layout in the screenshot), semantic attributes (HTML tags such as <button>, the placeholder text) and the spatial positioning (bounding box coordinates) dynamically, and outputs differentiated results based on the type of downstream task.

[0161] When applied to a region description task, the output is like <bbox> 120,300,240,340< / bbox> "Region is a gray input box with a'search' placeholder."

[0162] When applied to an interactive dialogue task, executable instructions such as {"user":"How to search","assistant":"Click <bbox> 250,300,290,340< / bbox> the magnifying glass button" are generated.

[0163] By sharing the prompt words, the dual task can reuse the same input pipeline, reducing the data preprocessing cost.

[0164] In one possible implementation, the sample data is used in the pre-training stage of multi-modal large model training; and the method of inputting the annotation information and the preset task prompt word as text input and the webpage screenshot as picture input into a large language model to obtain description information generated by the large language model for describing the webpage screenshot includes: splicing the annotation information and the region description generation prompt word as text input, inputting the webpage screenshot as picture input into the large language model, and obtaining description information of UI elements in the corresponding region generated by the large language model, the description information of the UI elements in the corresponding region being used in the pre-training stage.

[0165] In this implementation, the sample data is used in the pre-training stage of multi-modal large model training, and when generating the sample data, the screen annotation can be spliced with the region description generation prompt word to form a text instruction, which is combined with the webpage screenshot.

[0166] The core task of the pre-training stage is to enable the multi-modal large model to recognize images and generate image information, i.e., to establish a general alignment between visual regions and functional semantics without specific downstream task constraints.

[0167] To this end, in the embodiments of the present disclosure, the screen annotation obtained in step S12 is spliced with the region description generation prompt word to form a unified text input, and the webpage screenshot remains unchanged as picture input, which is sent into the multi-modal large model together with the text.

[0168] Under the guidance of the prompt word, the multi-modal large model outputs a natural language description, such as "This is a blue rectangular search button with the text 'Search' for submitting a search request." The description contains visual features (color, shape), text content ("Search") and functional semantics (submit search), forming a text-image alignment supervision label, which is directly used for large-scale pre-training.

[0169] In the embodiments of the present disclosure, the sample data is used for pre-training of a multi-modal large model, so that the model learns to associate visual forms, text content and functional semantics, solves the problem of data fragmentation of traditional GUI, and improves the accuracy of the model in a GUI understanding task; the boundary box coordinates in the problem are synchronously trained to improve spatial cognitive ability, laying a foundation for subsequent interaction tasks.

[0170] In a possible implementation, the sample data is used for an instruction fine-tuning stage of multi-modal large model training; and the inputting of the annotation information and a preset task prompt word as text and the inputting of the web screenshot as a picture into a large language model to obtain description information output by the large language model for describing the web screenshot include: splicing the annotation information and an interactive dialogue generation prompt word as text input, inputting the web screenshot as a picture into the large language model, and obtaining an interactive question and answer pair generated by the large language model and containing boundary box coordinates, the interactive question and answer pair being used for the instruction fine-tuning stage.

[0171] In this implementation, the sample data is used for an instruction fine-tuning (SFT) stage of a multi-modal large model; and a target of the instruction fine-tuning stage is to enable the model to not only understand an interface but also act according to a user instruction.

[0172] Specifically, the screen annotation generated in step S12 is spliced with an interactive dialogue generation prompt word to form complete text input, and then the web screenshot is kept unchanged as picture input, and is sent into a multi-modal large model together with the text.

[0173] Under the guidance of the prompt word, the multi-modal large model outputs an interactive question and answer pair, and an answer in the question and answer pair is directly used as a supervision label of the instruction fine-tuning stage, so that the model learns to map a natural language intention to accurate screen coordinates and actions during back propagation.

[0174] In the embodiments of the present disclosure, the sample data is directly used for instruction fine-tuning of a multi-modal large model, so that the model learns to convert a user request into a spatial operation with coordinates, solves the problem of lack of execution ability of a traditional GUI agent, and guarantees instruction accuracy through visual-HTML double verification (for example, avoiding misdescription of a gray button as blue), thereby improving operation accuracy of the trained model in a web interaction task.

[0175] In a possible implementation, the inputting of the web screenshot and the description information as sample data for training of a multi-modal large language model includes: cleaning and arranging the generated description information to remove repeated or invalid information; and saving the cleaned and arranged description information and the web screenshot as sample data for training of the multi-modal large language model.

[0176] When the webpage screenshot and description information are used as sample data for training a multimodal large language model, the description information output by the model can be cleaned and arranged first, and then paired with the corresponding screenshot and saved.

[0177] When cleaning, invalid information can be removed through a rule script or a confidence threshold, for example, descriptions that are completely repeated; entries with valid coordinates but empty text or only meaningless symbols; and incorrect descriptions that do not match the actual UI elements of the screenshot.

[0178] After cleaning, the description information can be arranged, for example, the remaining description information can be saved in a specific format, for example, in the "id+text+bbox" unified format; in addition, the training / validation / test set can be divided according to a specific ratio, for example, the training / validation / test set can be divided according to an 8:1:1 ratio.

[0179] In the embodiments of the present disclosure, the sample data after cleaning and arrangement eliminates noise and maintains the diversity of the alignment of text and pictures, so that the multimodal large language model can quickly converge with a higher signal-to-noise ratio of samples in the subsequent pre-training and instruction fine-tuning stages, and significantly improve the success rate of webpage interaction tasks.

[0180] A specific application scenario of the embodiments of the present disclosure is described below, Figure 2 A flowchart of another multimodal sample data generation method according to an embodiment of the present disclosure is shown. First, webpage information acquisition is performed on the target webpage, which can be achieved by opening the target webpage through a browser, automatically intercepting the current page to obtain a webpage screenshot, and parsing the DOM to extract element information (bbox+HTML code) of all UI elements.

[0181] Then, screen annotation is performed, and filtering operation is performed on the element information:

[0182] Remove elements with invalid links;

[0183] Remove "big box small picture" with a fill rate of less than 0.3;

[0184] Remove elements containing external links.

[0185] After filtering, the remaining elements are only retained according to the "nested integration" principle, and the internal text, class name, etc. are combined into the outermost UI element to form the final screen annotation.

[0186] Then, the shared prompt words, region description generated prompt words, and interactive dialogue generated prompt words are spliced with the screen annotation to generate description information.

[0187] The example includes a picture understanding task, i.e., a pre-training stage. Then, the screen annotation and the region description generation prompt word can be spliced into a text input, together with the webpage screenshot Figure 1 is input into a large language model; and the model outputs description information of the webpage screenshot, which can be used as a pre-training sample together with the webpage screenshot.

[0188] The example also includes a dialogue interaction task, i.e., an instruction fine-tuning stage. Then, the screen annotation and the interactive dialogue generation prompt word can be spliced and input into a large language model again, without changing the screenshot. The large language model generates description information (question and answer pairs) containing a bounding box, which can be used as an instruction fine-tuning sample together with the webpage screenshot.

[0189] Further, the two types of description information are uniformly de-duplicated, checked, and formatted into JSONL, and are divided into training / validation / testing sets at a ratio of 8:1:1 and saved.

[0190] The entire process does not require manual annotation, and can output large-scale, image-text aligned, and high-quality sample data that can be directly used for training of a multi-modal large model.

[0191] Figure 3 A block diagram of a multi-modal sample data generation apparatus according to an embodiment of the present disclosure is shown. As shown in the figure, the apparatus 20 includes: Figure 3

[0192] An acquisition module 21 is configured to acquire a webpage screenshot and element information of UI elements in a webpage for a target webpage;

[0193] An annotation information generation module 22 is configured to generate annotation information based on the element information of the UI elements;

[0194] A description information generation module 23 is configured to input the annotation information and a preset task prompt word as a text input and the webpage screenshot as a picture input into a large language model, to obtain description information for describing the webpage screenshot output by the large language model;

[0195] A sample data generation module 24 is configured to use the webpage screenshot and the description information as sample data for training of a multi-modal large language model.

[0196] In a possible implementation, the annotation information generation module is configured to:

[0197] perform at least one filtering operation on the UI elements as follows:

[0198] remove UI elements with invalid webpage links;

[0199] remove UI elements with a bounding box filling rate lower than a threshold value;

[0200] ​Prune the UI element containing the URL link in the HTML attribute;

[0201] Generate annotation information by using the element information of the filtered UI element.

[0202] In a possible implementation, the annotation information generation module is configured to:

[0203] Integrate the UI elements with nested relationship, retain the position information of the outermost UI element, and integrate other element information of the internal sub-elements of the outermost UI element into the annotation information of the outermost element.

[0204] In a possible implementation, the preset task prompt word includes at least one of the following:

[0205] The region description generation prompt word is used to guide the large language model to generate the description information of the UI element in the corresponding region.

[0206] The interactive dialogue generation prompt word is used to guide the large language model to generate the interactive question and answer pair containing the bounding box coordinates.

[0207] In a possible implementation, the sample data is used for the pre-training phase of the multi-modal large model training.

[0208] The description information generation module is configured to:

[0209] Splice the annotation information and the region description generation prompt word as text input, splice the web screenshot as picture input, input into the large language model, and obtain the description information of the UI element in the corresponding region generated by the large language model, the description information of the UI element in the corresponding region being used for the pre-training phase.

[0210] In a possible implementation, the sample data is used for the instruction fine-tuning phase of the multi-modal large model training.

[0211] The description information generation module is configured to:

[0212] Splice the annotation information and the interactive dialogue generation prompt word as text input, splice the web screenshot as picture input, input into the large language model, and obtain the interactive question and answer pair containing the bounding box coordinates generated by the large language model, the interactive question and answer pair being used for the instruction fine-tuning phase.

[0213] In a possible implementation, the sample data generation module is configured to:

[0214] Clean and arrange the generated description information to remove repeated or invalid information.

[0215] The cleaned and processed description information and the webpage screenshot are saved as sample data for training the multi-modal large language model.

[0216] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0217] The embodiments of the present disclosure also provide an electronic device, including a memory, a processor and a computer program stored in the memory, the processor executes the computer program to implement the steps of the above method.

[0218] The embodiments of the present disclosure also provide a non-volatile computer readable storage medium having a computer program stored thereon, the computer program is executed by a processor to implement the steps of the above method.

[0219] The embodiments of the present disclosure also provide a computer program product, including a computer program or a non-volatile computer readable storage medium carrying a computer program, the computer program is executed by a processor to implement the steps of the above method.

[0220] Figure 4 is a block diagram of a device for multi-modal sample data generation according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Referring to Figure 4 , the device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0221] The device 1900 can also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input output interface 1958 (I / O interface). The device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0222] In an example embodiment, there is also provided a non-transitory computer- readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component(s) 1922 of the apparatus 1900 to implement the above-described methods.

[0223] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0224] Computer programs (or computer readable program instructions), which describe the methods described herein, can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device from a network, for example, the Internet, a local area network, a wide area network, and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0225] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0226] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0227] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0228] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0229] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0230] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.< / button>

Claims

1. A multi-modal sample data generation method, characterized by, The method comprises the following steps: For a target webpage, obtaining a webpage screenshot and element information of UI elements in the webpage; Based on the element information of the UI elements, generating annotation information; Inputting the annotation information and a preset task prompt word as text input and the webpage screenshot as picture input into a large language model to obtain description information output by the large language model for describing the webpage screenshot; Using the webpage screenshot and the description information as sample data for training a multi-modal large language model.

2. The method of claim 1, wherein, The method comprises the following steps: Performing at least one filtering operation on the UI elements: Eliminating UI elements with invalid webpage links; Eliminating UI elements with a bounding box filling rate lower than a threshold value; Eliminating UI elements containing URL links in HTML attributes; Generating annotation information using the element information of the filtered UI elements.

3. The method of claim 1, wherein, The method comprises the following steps: Integrating UI elements with nested relationships, retaining position information of outermost UI elements, and integrating other element information of internal sub-elements into annotation information of the outermost elements.

4. The method of claim 1, wherein, The preset task prompt word comprises at least one of the following: A region description generation prompt word for guiding the large language model to generate description information of UI elements in a corresponding region; An interactive dialogue generation prompt word for guiding the large language model to generate an interactive question and answer pair containing bounding box coordinates.

5. The method of claim 4, wherein, The sample data is used in a pre-training stage of multi-modal large model training; The method comprises the following steps: Splicing the annotation information and the region description generation prompt word as text input, inputting the webpage screenshot as picture input into the large language model, and obtaining description information of UI elements in a corresponding region generated by the large language model, wherein the description information of the UI elements in the corresponding region is used in the pre-training stage.

6. The method of claim 4, wherein, The sample data is used in an instruction fine-tuning stage of multi-modal large model training; The method comprises the following steps: Splicing the annotation information and the interactive dialogue generation prompt word as text input, inputting the webpage screenshot as picture input into the large language model, and obtaining an interactive question and answer pair containing bounding box coordinates generated by the large language model, wherein the interactive question and answer pair is used in the instruction fine-tuning stage.

7. The method of claim 1, wherein, The method comprises the following steps: Cleaning and arranging the generated description information to remove duplicate or invalid information; Saving the cleaned and arranged description information and the webpage screenshot as sample data for training the multi-modal large language model.

8. A multi-modal sample data generating apparatus, characterized by comprising: The method comprises the following steps: An obtaining module is configured to obtain a webpage screenshot and element information of UI elements in the webpage for a target webpage. An annotation information generation module is configured to generate annotation information based on element information of the UI element; A description information generation module is configured to input the annotation information and a preset task prompt word as text input, and input the webpage screenshot as picture input into a large language model to obtain description information output by the large language model for describing the webpage screenshot; A sample data generation module is configured to use the webpage screenshot and the description information as sample data for training a multi-modal large language model.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-8. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images