Large Model Multimodal Output Display Method and Device

Abstract

The embodiments of this specification provide a method and device for multimodal output display of large models. The method for multimodal output display of large models includes: obtaining initial data, inputting the initial data into a large model to obtain target data; extracting the target data based on an extraction rule to determine parsed data; and based on the parsed data, drawing a target image and displaying the target image. By obtaining initial data, inputting the initial data into a large model to obtain target data; extracting the target data based on an extraction rule to determine parsed data; and based on the parsed data, drawing a target image and displaying the target image, the flexibility and diversity of the output data of the large model can be enhanced, thereby facilitating image drawing, improving the drawing efficiency, and increasing the success rate.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of big data technology, and particularly to a method for multi-modal output display of large models. Background Art

[0002] With the successive emergence of large language models such as ChatGPT, Stable Diffusion, and Sora, large language models (LLMs) can not only output text, but also pictures and even videos. The combination of multiple models can generate relatively novel and rich materials in the form of illustrated blogs, publicity manuscripts, etc. Therefore, most current products can achieve the mixed output form of text, pictures, and videos.

[0003] However, with the exploration of the capabilities of general large models, the mixed layout of text, pictures, and videos cannot meet the requirements. Some professional reports require more elements in addition to text and pictures, such as tables, data charts, maps, formulas, etc. Therefore, how to support the multi-modal data output display has become one of the primary problems faced by current large model products. Thus, a better solution is urgently needed. Summary of the Invention

[0004] In view of this, the embodiments of this specification provide a method for multi-modal output display of large models. One or more embodiments of this specification also relate to a multi-modal output display device for large models, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to the first aspect of the embodiments of this specification, a method for multi-modal output display of large models is provided, including:

[0006] Obtain initial data, and input the initial data into a large model to obtain target data;

[0007] Extract the target data based on an extraction rule to determine parsed data;

[0008] Based on the parsed data, draw a target image and display the target image.

[0009] In a possible implementation, obtaining initial data and inputting the initial data into a large model to obtain target data includes:

[0010] Determine text data based on the initial data, and input the text data into the large model to obtain target data; where the target data is in Markdown format.

[0011] In a possible implementation, extracting the target data based on an extraction rule to determine parsed data includes:

[0012] Determine comparison data based on extraction rules, compare the comparison data with target data, and determine a target parser;

[0013] Parse the target data based on the target parser to determine parsed data.

[0014] In one possible implementation, parsing the target data based on the target parser to determine parsed data includes:

[0015] Determine a set character based on the target parser;

[0016] Parse the parsed data from the target data based on the set character; wherein, the parsed data includes JS code data and DOM object data.

[0017] In one possible implementation, based on the parsed data, draw a target image and display the target image, including:

[0018] Determine whether there is an asynchronous loading plugin and determine the plugin loading result;

[0019] Based on the plugin loading result, draw a target image based on the parsed data and display the target image.

[0020] In one possible implementation, based on the plugin loading result, draw a target image based on the parsed data and display the target image, including:

[0021] In the case where the plugin loading result is that there is no asynchronous loading plugin, perform image rendering based on the parsed data to obtain a target image and display the target image.

[0022] In one possible implementation, based on the plugin loading result, draw a target image based on the parsed data and display the target image, including:

[0023] In the case where the plugin loading result is that there is an asynchronous loading plugin, perform image rendering based on the parsed data to obtain an initial rendered image;

[0024] Obtain third-party system data, perform image rendering based on the third-party system data and the initial rendered image to obtain a target image, and display the target image.

[0025] According to the second aspect of the embodiments of the present specification, there is provided a large model multi-modal output display device, including:

[0026] A data processing module, configured to obtain initial data, input the initial data into a large model to obtain target data;

[0027] A data parsing module, configured to extract the target data based on extraction rules to determine parsed data;

[0028] An image drawing module, configured to draw a target image based on the parsed data and display the target image.

[0029] According to a third aspect of the embodiments of the present specification, a computing device is provided, including:

[0030] A memory and a processor;

[0031] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned large model multi-modal output display method are implemented.

[0032] According to a fourth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the above-mentioned large model multi-modal output display method are implemented.

[0033] According to a fifth aspect of the embodiments of the present specification, a computer program is provided. When the computer program is executed on a computer, the computer is made to execute the steps of the above-mentioned large model multi-modal output display method.

[0034] The embodiments of the present specification provide a large model multi-modal output display method and device. The large model multi-modal output display method includes: obtaining initial data, inputting the initial data into a large model to obtain target data; extracting the target data based on an extraction rule to determine parsed data; drawing a target image based on the parsed data and displaying the target image. By obtaining initial data, inputting the initial data into a large model to obtain target data; extracting the target data based on an extraction rule to determine parsed data; drawing a target image based on the parsed data and displaying the target image, the flexibility and diversity of the output data of the large model can be enhanced, thereby facilitating image drawing, improving the drawing efficiency, and increasing the success rate. Description of the Drawings

[0035] Figure 1 is a flowchart of a large model multi-modal output display method provided by an embodiment of the present specification;

[0036] Figure 2 is a schematic diagram of a large model multi-modal output display method provided by an embodiment of the present specification;

[0037] Figure 3 is a structural schematic diagram of a large model multi-modal output display device provided by an embodiment of the present specification;

[0038] Figure 4 is a structural block diagram of a computing device provided by an embodiment of the present specification. Detailed implementation manners

[0039] In the following description, numerous specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the connotation of this specification. Therefore, this specification is not limited by the specific implementations disclosed below.

[0040] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining".

[0042] First, the noun terms related to one or more embodiments of this specification are explained.

[0043] JS: JavaScript, is a lightweight, interpreted, or just-in-time compiled programming language with function-first.

[0044] DOM: Document Object Model, is a programming interface for a web document. It represents the page so that programs can change the structure, style, and content of the document. The DOM represents the document as nodes and objects; in this way, programming languages can interact with the page.

[0045] Markdown: is a lightweight markup language.

[0046] Currently, the output of large models basically adopts the Markdown format. Markdown itself supports the output of text, images, and tables, and some extension plugins support the output of rich text such as charts and videos. When using Markdown for output, all formats are fixed and designed with zero tolerance for errors. Due to the hallucinations and randomness of large models, it is extremely difficult for large models to output in a fixed format and with completely correct results. Moreover, Markdown supports formats such as text, images, tables, videos, and links, but it cannot support flexible third-party application forms such as charts and maps. Markdown has a single-compilation design structure, so all data and formats need to be rendered at once. This results in slow speed or even crashes if a Markdown document contains a large amount of data (more than 2000 lines) in the form of charts or tables.

[0047] Therefore, in this specification, a method for multi-modal output display of large models is provided. This specification also relates to a device for multi-modal output display of large models, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0048] See Figure 1 , Figure 1 which shows a flowchart of a method for multi-modal output display of large models according to an embodiment of this specification, specifically including the following steps.

[0049] Step 101: Obtain initial data, and input the initial data into the large model to obtain target data.

[0050] In a possible implementation, obtaining initial data and inputting the initial data into the large model to obtain target data includes: determining text data based on the initial data, inputting the text data into the large model, and obtaining target data; where the target data is in Markdown format.

[0051] In practical applications, users can send inquiries to the large model, that is, issue tasks to the large model, such as: What is the driving route of vehicle A today? See Figure 2 , and the large model, based on this task, retrieves the route data of the vehicle and outputs data in Markdown format.

[0052] Step 102: Extract the target data based on the extraction rule to determine the parsed data.

[0053] In a possible implementation, extracting the target data based on the extraction rule to determine the parsed data includes: determining comparison data based on the extraction rule, comparing the comparison data with the target data to determine the target parser; and parsing the target data using the target parser to determine the parsed data.

[0054] In practical applications, the large model understands that the output format of Echarts is in JSON format, including modules such as title, text, xAxis, yAxis, series, etc. Problems with the formats of these modules will cause rendering failures. In the embodiments of this specification, all these attributes are built-in, and only the data part needs to be concerned about, as exemplified below.

[0055] :::pie{"title":"My Skills","key":["java","python","scala"],"value":[100,60,70,70]}:::;

[0056] Among them, "pie" is a type of comparison data. If "pie" exists in the data, the corresponding target parser can be used to parse the subsequent data to obtain the parsed data of "pie".

[0057] See Figure 2 , the target parser can also include video format, audio format, map format, and Echarts format parsers.

[0058] In a possible implementation, based on the target parser to parse the target data to determine the parsed data, including: determining the set character based on the target parser; parsing the parsed data from the target data based on the set character; where the parsed data includes JS code data and DOM object data.

[0059] Continuing with the above example, the set character corresponding to the parser of "pie" is "pie", so look for the "pie" character in the Markdown format data. After finding it, change the format of the corresponding data to obtain the parsed data. The format of the parsed data can include formats such as html, css, js, and json.

[0060] It should be noted that the parser can also be extended. The custom parser determines which parser to use according to xxx in ":::xxx". What the parser can finally return is a DOM object in html + js code.

[0061] Step 103: Based on the parsed data, draw the target image and display the target image.

[0062] Specifically, based on the parsed data, draw the target image and display the target image, including: determining whether there is an asynchronous loading plugin and determining the plugin loading result; based on the plugin loading result, draw the target image based on the parsed data and display the target image.

[0063] In a possible implementation, based on the plugin loading result and the parsed data, a target image is drawn and the target image is displayed, including: when the plugin loading result indicates that there is no asynchronous loading plugin, image rendering is performed based on the parsed data to obtain the target image, and the target image is displayed.

[0064] In practical applications, if there is no requirement for asynchronous data conversion, that is, there is no asynchronous loading plugin, then the data can be directly rendered to generate an image and displayed.

[0065] In another possible implementation, based on the plugin loading result and the parsed data, a target image is drawn and the target image is displayed, including: when the plugin loading result indicates that there is an asynchronous loading plugin, image rendering is performed based on the parsed data to obtain an initial rendered image; third-party system data is obtained, and image rendering is performed based on the third-party system data and the initial rendered image to obtain the target image, and the target image is displayed.

[0066] In practical applications, data can also be obtained asynchronously. Therefore, in each return of the large model, an id can be set for the data. For example: :::pie{"id":"xxxxxxxxxxxxx"}:::, after the parser obtains the id, it uses an asynchronous interface to obtain the data, and the obtained data is {"key":["java","python","scala"],"value":[100,60,70,70]}. After obtaining the data, the rendering graph is performed.

[0067] For example, when asking the large model what is the driving route of vehicle A today? After the large model outputs data in Markdown format, the target parser converts the data into data encapsulated by html, css, js, and json. See Figure 2 to determine whether there is an asynchronous loading plugin. If so, set an id for the data, obtain the data output by the large model through the id, perform preliminary rendering, and then retrieve the data of the map software to draw the route and the map together to obtain the target image, and display the target image. Figure 1 to obtain the target image and display the target image.

[0068] The embodiments of this specification provide a method and device for multimodal output display of large models. The method for multimodal output display of large models includes: obtaining initial data, inputting the initial data into the large model to obtain target data; extracting the target data based on an extraction rule to determine parsed data; and based on the parsed data, drawing a target image and displaying the target image. By obtaining initial data, inputting the initial data into the large model to obtain target data, extracting the target data based on an extraction rule to determine parsed data, drawing a target image based on the parsed data, and displaying the target image, the flexibility and diversity of the output data of the large model can be enhanced, thereby facilitating image drawing, improving the drawing efficiency, and increasing the success rate.

[0069] Further, in some scenarios, a powerful display plugin can be encapsulated using html+css+js+json. Thereby enhancing the flexibility and diversity of the output of the large model. For example, for a map component, a component that supports querying the running trajectory can be encapsulated based on amap. In a big data scenario, asynchronous and secondary compilation methods can be used to display complex charts and address the problem of slow rendering speed.

[0070] Corresponding to the above method embodiments, this specification also provides embodiments of a device for multimodal output display of large models. Figure 3 The structural schematic diagram of a device for multimodal output display of large models provided by an embodiment of this specification is shown. As Figure 3 shown, the device includes:

[0071] A data processing module 301, configured to obtain initial data and input the initial data into the large model to obtain target data;

[0072] A data parsing module 302, configured to extract the target data based on an extraction rule to determine parsed data;

[0073] An image drawing module 303, configured to draw a target image based on the parsed data and display the target image.

[0074] In a possible implementation manner, the data processing module 301 is further configured to:

[0075] Determine text data based on the initial data, input the text data into the large model, and obtain target data; wherein, the target data is in Markdown format.

[0076] In a possible implementation manner, the data parsing module 302 is further configured to:

[0077] Determine comparison data based on the extraction rule, compare the comparison data with the target data to determine a target parser;

[0078] Parse the target data based on the target parser to determine the parsed data.

[0079] In a possible implementation, the data parsing module 302 is further configured to:

[0080] Determine a set character based on the target parser;

[0081] Parse the parsed data from the target data based on the set character; wherein, the parsed data includes JS code data and DOM object data.

[0082] In a possible implementation, the image drawing module 303 is further configured to:

[0083] Determine whether there is an asynchronous loading plugin to obtain the plugin loading result;

[0084] Draw a target image based on the parsed data according to the plugin loading result, and display the target image.

[0085] In a possible implementation, the image drawing module 303 is further configured to:

[0086] In the case where the plugin loading result is that there is no asynchronous loading plugin, perform image rendering based on the parsed data to obtain a target image, and display the target image.

[0087] In a possible implementation, the image drawing module 303 is further configured to:

[0088] In the case where the plugin loading result is that there is an asynchronous loading plugin, perform image rendering based on the parsed data to obtain an initial rendered image;

[0089] Obtain third-party system data, perform image rendering based on the third-party system data and the initial rendered image to obtain a target image, and display the target image.

[0090] The embodiments of this specification provide a large model multi-modal output display method and device. The large model multi-modal output display device includes: obtaining initial data, inputting the initial data into the large model to obtain target data; extracting the target data based on an extraction rule to determine parsed data; drawing a target image based on the parsed data, and displaying the target image. By obtaining initial data, inputting the initial data into the large model to obtain target data; extracting the target data based on an extraction rule to determine parsed data; drawing a target image based on the parsed data, and displaying the target image, the flexibility and diversity of the output data of the large model can be enhanced, thus facilitating image drawing, improving the drawing efficiency, and increasing the success rate.

[0091] The above is a schematic solution of a large model multi-modal output display device according to this embodiment. It should be noted that the technical solution of the large model multi-modal output display device belongs to the same concept as the technical solution of the above large model multi-modal output display method. For the details not described in the technical solution of the large model multi-modal output display device, reference can be made to the description of the technical solution of the above large model multi-modal output display method.

[0092] Figure 4 The block diagram of a computing device 400 according to an embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 through a bus 430, and a database 450 is used to store data.

[0093] The computing device 400 further includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0094] In an embodiment of this specification, the above components of the computing device 400 and Figure 4 other components not shown therein may also be connected to each other, for example, through a bus. It should be understood that Figure 4 the block diagram of the computing device shown is for illustrative purposes only and is not a limitation on the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0095] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0096] Among them, the processor 420 is used to execute the following computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above-mentioned large model multimodal output display method are implemented. The above is a schematic solution of a computing device in this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-mentioned large model multimodal output display method belong to the same concept. For the details not described in detail in the technical solution of the computing device, reference can be made to the description of the technical solution of the above-mentioned large model multimodal output display method.

[0097] An embodiment of this specification also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the above-mentioned large model multimodal output display method are implemented.

[0098] The above is a schematic solution of a computer-readable storage medium in this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-mentioned large model multimodal output display method belong to the same concept. For the details not described in detail in the technical solution of the storage medium, reference can be made to the description of the technical solution of the above-mentioned large model multimodal output display method.

[0099] An embodiment of this specification also provides a computer program, wherein when the computer program is executed on a computer, the computer is made to execute the steps of the above-mentioned large model multimodal output display method.

[0100] The above is a schematic solution of a computer program in this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the above-mentioned large model multimodal output display method belong to the same concept. For the details not described in detail in the technical solution of the computer program, reference can be made to the description of the technical solution of the above-mentioned large model multimodal output display method.

[0101] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, removable hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0103] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0104] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0105] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The alternative embodiments do not elaborate on all the details and do not limit the invention to only the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can understand and utilize this specification well. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A large model multimodal output display method, characterized in that: include: Acquire initial data, and input the initial data into a large model to acquire target data; Extract the target data based on the extraction rules to determine parsed data; Based on the analyzed data, draw a target image and display the target image; The step of extracting the target data based on the extraction rule and determining the parsed data includes: Determine comparison data based on the extraction rule, and determine a target parser based on comparing the comparison data with the target data; Parsing the target data based on the target parser to determine parsed data; The step of parsing the target data based on the target parser to determine parsed data includes: Determining a set character based on the target parser; Parsing the target data based on the set character to obtain parsed data; wherein the parsed data includes JS code data and DOM object data; Drawing a target image based on the parsed data and displaying the target image comprises: Determine whether there is an asynchronous loading plug-in and determine the plug-in loading result; Draw a target image based on the parsed data according to the plug-in loading result, and display the target image; The step of drawing a target image based on the parsed data according to the plug-in loading result and displaying the target image includes: When the plug-in loading result is that the asynchronous loading plug-in does not exist, image rendering is performed based on the parsed data to obtain a target image, and the target image is displayed.

2. The method according to claim 1, characterized in that The obtaining of initial data, and inputting the initial data into a large model to obtain target data, comprises: Determine text data based on the initial data, input the text data into the large model, and obtain target data; wherein the target data is in Markdown format.

3. The method according to claim 1, characterized in that The step of drawing a target image based on the parsed data according to the plug-in loading result and displaying the target image includes: When the plug-in loading result indicates that the asynchronous loading plug-in exists, performing image rendering based on the parsed data to obtain an initial rendered image; The third-party system data is obtained, image rendering is performed based on the third-party system data and the initial rendered image to obtain a target image, and the target image is displayed.

4. A large model multimodal output display device, used to implement the steps of the large model multimodal output display method described in any one of claims 1 to 3, characterized in that: include: A data processing module is configured to obtain initial data, and input the initial data into a large model to obtain target data; A data parsing module is configured to extract the target data based on an extraction rule and determine parsed data; The image drawing module is configured to draw a target image based on the parsed data and display the target image.

5. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the large model multimodal output display method described in any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the large model multimodal output display method described in any one of claims 1 to 3.

Images