Multi-modal text generation method, acquisition method, device, equipment and medium

Abstract

To provide a multi-modal text generation method capable of automatically generating a multi-modal text to improve versatility of generated content, and an acquisition method, device, equipment and medium of the multi-modal text.SOLUTION: A multi-modal text generation method comprises: in response to reception of a multimodal text generation request including prompt information, generating text information corresponding to the prompt information using a large language model based on the prompt information; generating image information corresponding to the text information using the large language model based on the text information; and calling a multi-modal text rendering tool using the large language model based on the text information and the image information, and obtaining a multi-modal text including the text information and the image information through rendering.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the technical field of artificial intelligence, particularly to technical fields such as computer vision, deep learning, and large-scale models, and in particular to a method, apparatus, device, medium, and program for generating and acquiring multimodal text. [Background technology]

[0002] With the development of computer and network technologies, the application of deep learning models has become increasingly widespread, and deep learning models have achieved breakthroughs in various fields. Here, AI generated content (AIGC) is an important direction of deep learning. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure aims to provide a method, device, equipment, medium, and program for generating multimodal text and acquiring multimodal text that can automatically generate multimodal text and improve the diversity of generated content. [Means for solving the problem]

[0004] According to a first aspect of the present disclosure, there is provided a method for generating multimodal text, including: in response to receiving a multimodal text generation request including presentation information, generating text information corresponding to the presentation information based on the presentation information using a large-scale language model; generating image information corresponding to the text information based on the text information using the large-scale language model; and invoking a multimodal text rendering tool using the large-scale language model based on the text information and the image information, and rendering to obtain multimodal text including the text information and the image information.

[0005] According to a second aspect of the present disclosure, there is provided a method for obtaining multimodal text, the method including: in response to receiving presentation information, sending a multimodal text generation request including the presentation information; and in response to obtaining multimodal text generated in response to the multimodal text generation request, displaying the multimodal text, wherein the multimodal text is generated using the multimodal text generation method provided by the first aspect of the present disclosure.

[0006] According to a third aspect of the present disclosure, there is provided a multimodal text generation device including: a text generation module that, in response to receiving a multimodal text generation request including presentation information, generates text information corresponding to the presentation information based on the presentation information using a large-scale language model; an image generation module that generates image information corresponding to the text information based on the text information using the large-scale language model; and a multimodal text generation module that invokes a multimodal text rendering tool using the large-scale language model based on the text information and the image information, and performs rendering to obtain multimodal text including the text information and the image information.

[0007] According to a fourth aspect of the present disclosure, there is provided a multimodal text acquisition device including: an information transmitting and receiving module that transmits a multimodal text generation request including the presentation information in response to receiving the presentation information; and a multimodal text display module that displays the multimodal text generated using the multimodal text generation device provided by the third aspect of the present disclosure in response to the information transmitting and receiving module acquiring the multimodal text generated in response to the multimodal text generation request.

[0008] According to a fifth aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform a method for generating multimodal text or a method for obtaining multimodal text provided by the present disclosure.

[0009] According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon computer instructions, the computer instructions being used to cause a computer to perform a method for generating multimodal text or a method for obtaining multimodal text provided by the present disclosure.

[0010] According to a seventh aspect of the present disclosure, there is provided a computer program stored in at least one of a readable storage medium and an electronic device, which, when executed by a processor, realizes the method for generating multimodal text or the method for obtaining multimodal text provided by the present disclosure.

[0011] It should be understood that the contents described in this section are not intended to depict key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood from the following description.

[0012] The drawings are for a better understanding of the invention and are not intended to limit the disclosure. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of an application scene of a method and apparatus for generating multimodal text and a method and apparatus for obtaining multimodal text according to an embodiment of the present disclosure.

[0014] [Figure 2] FIG. 2 is a flowchart of a method for generating multimodal text according to an embodiment of the present disclosure.

[0015] [Figure 3] FIG. 3 is a principle schematic diagram of generating text information according to an embodiment of the present disclosure.

[0016] [Figure 4] FIG. 4 is a principle schematic diagram of generating image information according to an embodiment of the present disclosure.

[0017] [Figure 5] FIG. 5 is a principle schematic diagram of rendering to obtain multimodal text according to an embodiment of the present disclosure.

[0018] [Figure 6] FIG. 6 is an architectural diagram illustrating an implementation of a method for generating multimodal text according to an embodiment of the present disclosure.

[0019] [Figure 7] FIG. 7 is a flowchart of a method for acquiring multimodal text according to an embodiment of the present disclosure.

[0020] [Figure 8] FIG. 8 is a block diagram of a configuration of a multimodal text generation device according to an embodiment of the present disclosure.

[0021] [Figure 9] FIG. 9 is a block diagram of a configuration of a multimodal text acquisition device according to an embodiment of the present disclosure.

[0022] [Figure 10] FIG. 10 is a block diagram of an electronic device for implementing the multimodal text generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. Various details of the embodiments of the present disclosure are included here for ease of understanding, but they should be considered as examples. Therefore, it should be understood that those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description will omit descriptions of well-known functions and configurations.

[0024] Multimodal text is a special text format, and multimodal text is more colorful than regular text. For example, multimodal text may contain characters in different fonts, colors, and sizes, and may contain elements such as images, rings, tables, and videos, which make the text more lively and interesting.

[0025] With the development of deep learning technology, deep learning models are increasingly used as auxiliary tools for content generation in order to improve content generation efficiency. For example, a generative large-scale language model may be used to generate text, and an artificial intelligence drawing tool may be used to generate images. When multimodal text needs to be generated, text may be generated using a large-scale language model, an artificial intelligence drawing tool may be used to generate images, and finally, the generated text may be added to the generated images using a picture editor such as Photoshop. That is, in related technologies, various models or tools must be invoked when generating multimodal text, resulting in technical problems such as low efficiency and high cost of generating multimodal text.

[0026] In order to solve the problems existing in the related art, the present disclosure provides a method for generating multimodal text and a method, apparatus, device, medium, and program for acquiring multimodal text. Hereinafter, application scenarios of the method and apparatus provided by the present disclosure will be described first with reference to FIG.

[0027] FIG. 1 is a schematic diagram of an application scene of the method and apparatus for generating multimodal text and obtaining multimodal text according to an embodiment of the present disclosure.

[0028] As shown in FIG. 1, an application scenario 100 of this embodiment includes a user 110, a terminal device 120, and a server .

[0029] Here, the terminal device 120 may be any electronic device capable of providing an interaction interface, such as a smartphone, a tablet computer, a mobile computer, a desktop computer, etc. The terminal device 120 can be communicatively connected to the server 130 via a network.

[0030] For example, user 110 can input presentation information to terminal device 120 via an interaction interface provided by terminal device 120, and have terminal device 120 generate and present multimodal text based on the presentation information. For example, content sharing client interaction interface generation client applications and the like may be installed on terminal device 120, and user 110 may input presentation information into the interaction interface of these client applications, and these client applications may display multimodal text generated based on the presentation information to user 110.

[0031] In one embodiment, after receiving the presentation information input by the user 110, the terminal device 120 can, for example, transmit a multimodal text generation request 101 including the presentation information to the server 130. In response to receiving the multimodal text generation request 101, the server 130 can, for example, generate multimodal text 102 based on the presentation information therein and feed the generated multimodal text 102 back to the terminal device 120 so that the terminal device 120 can display the multimodal text 102 to the user 110.

[0032] In one embodiment, the server 130 may use a large-scale language model to determine a process for generating multimodal text, and then call a generation tool, a rendering tool, or the like to generate the multimodal text based on the determination result. The generation tool here may be a large-scale language model that makes decisions, or may be another deep learning model, or the like; the present disclosure is not limited thereto. In this way, the server 130 may realize automated generation of multimodal text.

[0033] In one embodiment, server 130 may be a background management server that supports the execution of client applications installed on terminal device 120. Server 130 may also be a cloud server, also called a cloud computing server or cloud host, which is a host product in a cloud computing service system that solves the drawbacks of traditional physical hosts and VPS services (abbreviated as "Virtual Private Server" or "VPS"), such as high management difficulty and poor service scalability. Server 130 may also be a server in a distributed system or a server combined with blockchain.

[0034] It should be noted that the method for generating multimodal text provided by the present disclosure may be executed by the server 130. Accordingly, the device for generating multimodal text provided by the present disclosure may be provided in the server 130. The method for obtaining multimodal text provided by the present disclosure may be executed by the terminal device 120. Accordingly, the device for obtaining multimodal text provided by the present disclosure may be provided in the terminal device 120.

[0035] It should be understood that the number and types of terminal devices 120 and servers 130 in Figure 1 are merely exemplary, and any number and types of terminal devices 120 and servers 130 may be included as required for implementation.

[0036] The method for generating multimodal text provided by the present disclosure will be described in detail below with reference to FIGS.

[0037] FIG. 2 is a flowchart of a method for generating multimodal text according to an embodiment of the present disclosure.

[0038] As shown in FIG. 2, the method 200 for generating multimodal text in this embodiment may include operations S210 to S230.

[0039] In operation S210, in response to receiving a multimodal text generation request including the presentation information, text information corresponding to the presentation information is generated based on the presentation information using a large-scale language model.

[0040] According to an embodiment of the present disclosure, after receiving a multimodal text generation request, the server may analyze the multimodal text generation request, obtain proposal information included in the request, use the proposal information as input to a large-scale language model, and use the text output from the large-scale language model as text information. Here, the large-scale language model may be an existing large-scale language model, and the present disclosure is not limited thereto.

[0041] For example, the presented information may be theme information of the multimodal text to be generated, or keywords of the multimodal text to be generated, and the present disclosure does not limit this. For example, the presented information may be "mobile phones suitable for students," "what interesting tourist spots are there in XX," or the like. The text information may be text content included in the multimodal text to be generated. For example, the text information may be "mobile phones suitable for students include mobile phone A, mobile phone B, mobile phone C, and mobile phone D," or "if you are visiting XX for the first time, recommended tourist spots are tourist spot 1, tourist spot 2, and tourist spot 3," and the present disclosure does not limit this.

[0042] In operation S220, based on the text information, image information corresponding to the text information is generated using a large-scale language model.

[0043] In one embodiment, text information is used as input to a large-scale language model, and an image database is searched based on the text information using the large-scale language model to retrieve images matching the text information as image information. Here, the image database can hold a large number of images. For example, based on the text information "Cell phones suitable for students include cell phone a, cell phone b, cell phone c, and cell phone d," it is possible to search for image fig1 corresponding to cell phone a, image fig2 corresponding to cell phone b, image fig3 corresponding to cell phone c, and image fig4 corresponding to cell phone d. In this embodiment, image fig1, image fig2, image fig3, and image fig4 can be used as image information.

[0044] In this embodiment, the large-scale language model is a model that has been pre-trained to have the ability to connect to external tools (eg, function-calling capabilities) including image databases.

[0045] In one embodiment, the large-scale language model is a pre-trained multimodal large-scale language model, and in this embodiment, text information and presentation information for image generation are input to the large-scale language model, and image information is generated by the large-scale language model. For example, the input to the large-scale language model may be "generate an image for the text 'Mobile phones suitable for students include mobile phone a, mobile phone b, mobile phone c, and mobile phone d'", and the presentation information for image generation may be "generate an image for the text '", but the present disclosure is not limited thereto.

[0046] In operation S230, a multimodal text rendering tool is invoked based on the text information and image information using a large-scale language model to render and obtain a multimodal text including the text information and image information.

[0047] According to an embodiment of the present disclosure, the large-scale language model is a model capable of connecting to external tools, including a multimodal text rendering tool, through pre-training. For example, a calling interface for the external tool can be provided to the large-scale language model, and text information and image information can be input to the large-scale language model, and presentation information for calling the multimodal text rendering tool can be input to the large-scale language model. Based on the presentation information, the large-scale language model can use the text information and image information as input parameters for the calling interface of the multimodal text rendering tool and output multimodal text fed back from the calling interface. In operation S230, the large-scale language model can act as an agent for calling the multimodal text rendering tool.

[0048] For example, an automatic multi-step read-only tool or a function-calling function may be employed to provide a large-scale language model with the ability to connect to external tools, and the present disclosure is not limited thereto.

[0049] Here, the multimodal text rendering tool may be, for example, the Rich Text open source library or the wxParse component, and the present disclosure is not limited thereto.

[0050] According to embodiments of the present disclosure, it is possible to realize integrated and automated generation of multimodal text based on a large-scale language model, which can improve the efficiency and automation of multimodal text generation compared to technical solutions that require manually invoking different models to generate text and images and using an image editor to add text to images. In the process of generating multimodal text, embodiments of the present disclosure do not require users to learn invocation techniques for different models or tools, and only require the provision of presentation information, thereby reducing the cost of generating multimodal text, favoring the dissemination of information in the form of multimodal text, and also improving the dissemination of large-scale language models.

[0051] The principle of generating text information will be further expanded below with reference to Fig. 3. Fig. 3 is a schematic diagram illustrating the principle of generating text information according to an embodiment of the present disclosure.

[0052] In one embodiment, when generating text information, for example, a large-scale language model may determine whether text needs to be generated based on search results. If it is determined that text needs to be generated based on the search results, a search expansion generation task is performed by the large-scale language model to generate text based on the search results. In this manner, the generated text information is a fusion of search results, which can improve the timeliness, accuracy, and / or authenticity of the generated text information. This is because a search can retrieve highly timely and true information that the large-scale language model has not learned. Furthermore, generating text based on search results can improve the diversity of the generated text information. This is because when text is generated directly using a large-scale language model, the generated text is usually highly similar. By generating text in conjunction with search results, the large-scale language model can refer to its unlearned knowledge when generating text.

[0053] 3, in embodiment 300, when a multimodal text generation request is received, the server can first use a large-scale language model 310 to process the presented information 301 in the multimodal text generation request, and generate first determination information 320 using the large-scale language model. The first determination information 320 includes first instruction information 321 indicating whether a first database needs to be searched.

[0054] Here, the first database may be a database corresponding to a search engine, and the present disclosure is not limited thereto. In this embodiment, the input of the large-scale language model may be constituted by the presentation information 301 in the multimodal text generation request and the presentation information for prompting the large-scale language model to generate a determination result for searching the first database. Here, the presentation information for prompting the large-scale language model to generate a determination result for searching the first database may be, "Do you need to generate and search multimodal text based on the following presentation information?"

[0055] In one embodiment, when the first instruction information indicates that the first database needs to be searched, the first determination information 320 may further include, for example, a search statement 322. Accordingly, the presentation information for prompting the large-scale language model to generate a determination result for searching the first database may also be used to prompt the large-scale language model to generate a search statement, for example, the presentation information may be "Based on the presentation information below, do you need to generate and search text for multimodal text? If so, please provide a search statement."

[0056] When the first determination information generated by the large-scale language model 310 indicates that a first database needs to be searched, the embodiment 300 can use the large-scale language model 310 to call a call interface corresponding to the first database 330 to perform a data search, and generate text information based on the search results. Specifically, the large-scale language model 310 can generate the text information by performing a Retrieval-Augmented Generation (RAG) task. For example, the query statement 322 can be input to the large-scale language model 310, and presentation information instructing the large-scale language model to perform the Retrieval-Augmented Generation task can also be input to the large-scale language model 310, so that the large-scale language model 310 can perform the RAG task based on the query statement.

[0057] For example, in response to the input information including presentation information instructing the execution of a search expansion generation task, the large-scale language model 310 can, based on its function of invoking an external tool, use the search statement 322 in the input information as an input parameter of a call interface corresponding to the first database 330. After receiving the searched information fed back from the call interface, the large-scale language model 310 can use the searched information to guide the generation of the text information 302 and output the generated text information 302.

[0058] For example, the process of using a large-scale language model to perform a search expansion generation task may include inputting a search statement and presentation information instructing the large-scale language model to perform the search task into the large-scale language model, and using the search statement as an input parameter of a call interface corresponding to a first database via the large-scale language model. After receiving the searched information fed back via the call interface, the large-scale language model can directly output the searched information. Then, the output searched information and the presentation information in the multimodal text generation request can be used as inputs for the large-scale language model, and the large-scale language model can execute a text generation task based on the input information, and output the generated text as text information after executing the text generation task.

[0059] In one embodiment, as shown in FIG. 3 , if the first instruction information indicates that there is no need to search the first database, the embodiment can directly use the large-scale language model 310 to perform the text generation task, such that the large-scale language model 310 generates text information 302 based on the presented information 301.

[0060] For example, the presentation information 301 can be used as input information for the large-scale language model 310, and the presentation information 301 can be processed by the large-scale language model 310, and for example, the large-scale language model 310 can perform text prediction based on the presentation information 301 and output the predicted text as generated text information 302.

[0061] In one embodiment, in the processing step in which the large-scale language model outputs text information, the information input to the large-scale language model may further include presentation information for instructing the format of the generated text information so that the format of the generated text information better meets actual needs. For example, the format of the text information may include a table format, a summary format, etc., and the present disclosure is not limited thereto.

[0062] For example, when no presentation information related to the task to be performed is input, the large-scale language model 310 performs the text generation task by default, i.e., performs text prediction based on the input information, and sets the predicted text as the generated text.

[0063] In one embodiment, when generating the first decision information using the large-scale language model, the information input to the large-scale language model may further include, for example, presentation information indicating a decision rule, and the large-scale language model generates a decision result based on the decision rule. For example, the decision rule may be, "If the presentation information includes the name of an item that changes rapidly, such as a mobile phone, a search is required." It should be understood that, for ease of understanding of the present disclosure, the above decision rules are merely exemplary, and any decision rule can be set according to actual needs, and the present disclosure is not limited thereto.

[0064] The principle of generating image information will be further expanded below with reference to Fig. 4. Fig. 4 is a schematic diagram illustrating the principle of generating image information according to an embodiment of the present disclosure.

[0065] In one embodiment, when generating image information, a determination can be made as to whether or not an image needs to be retrieved by search, for example, using a large-scale language model. If it is determined that an image needs to be retrieved by search, an image search task is performed using the large-scale language model to search for images matching the text information. In this manner, the generated image information is obtained by search, and the timeliness, accuracy, and / or authenticity of the obtained image information can be improved. This is because the search can retrieve timely and authentic information that has not been learned by a deep learning model, such as a large-scale language model. For example, if there is a real image (e.g., an image related to a certain drama) that matches the text information, the retrieved image is typically more authentic than an image generated by a deep learning model.

[0066] 4, in embodiment 400, after generating text information 401, the server may first process the text information 401 using a large-scale language model 410, and then generate second determination information 420 through the large-scale language model. The second determination information 420 includes second instruction information 421 indicating whether a second database needs to be searched.

[0067] Here, the second database may be an image database corresponding to a search engine, and the present disclosure is not limited thereto. In this embodiment, the input information of the large-scale language model may be constituted by text information 401 and presentation information for presenting to the large-scale language model to generate a determination result for searching the second database. Here, the presentation information for presenting to the large-scale language model to generate a determination result for searching the second database may be, "Whether an image of the following text information can be obtained by searching?"

[0068] In one embodiment, when the large-scale language model 410 is used to generate the second determination information, the input information of the large-scale language model 410 may further include the presentation information in the multimodal text generation request, thereby increasing the richness of the reference information when the large-scale language model generates the second determination information and improving the accuracy of the second determination information generated by the large-scale language model.

[0069] In one embodiment, when the second instruction information 421 indicates that a second database needs to be searched, the second determination information 420 may further include, for example, search parameters 422. Accordingly, the prompting information for prompting the large-scale language model to generate a determination result for searching the second database is also used to prompt the large-scale language model to generate search parameters, for example, the prompting information may be "Can an image of the following text information be obtained by searching? If so, please provide search parameters."

[0070] If the second determination information generated by the large-scale language model 410 indicates that a second database needs to be searched, the embodiment 400 can use the large-scale language model 410 to call a call interface corresponding to the second database 430 to perform data search, and output the searched image as image information 402.

[0071] For example, in response to the input information including presentation information instructing the execution of an image search task, the large-scale language model 410 can, based on the function of the large-scale language model to call an external tool, set the search parameters 422 in the input information as input parameters of a call interface corresponding to the second database 430. After receiving the searched image fed back from the call interface, the large-scale language model 410 can output the searched image as image information 402.

[0072] In one embodiment, if the second instruction information indicates that there is no need to search the second database, the second determination information 420 may further include an image description 423. The image description 423 may be a basis for generating an image. In this embodiment, the large-scale language model is directly used to perform an image generation task based on the image description 423, so that the large-scale language model can generate image information.

[0073] In one embodiment, as shown in FIG. 4, when the second instruction information 421 indicates that there is no need to search the database, the embodiment can also generate image information 402 by using the image description sentence 423 as input information for the Text-to-Image model 440 and executing an image generation task based on the image description sentence 423 using the Text-to-Image model 440.

[0074] Here, the text-to-image model 440 may include, for example, a model built based on a large-scale language model or a stationary diffusion model, but the present disclosure is not limited thereto. The text-to-image model 440 and the large-scale language model 410 can constitute a system for providing multimodal text generation functionality.

[0075] In one embodiment, when generating the second decision information using the large-scale language model, the information input to the large-scale language model may further include, for example, presentation information instructing a decision rule, such that the large-scale language model generates a decision result based on the decision rule. For example, the decision rule may be, "If the text information contains real-world content, search for images." It should be understood that, for ease of understanding of the present disclosure, the above decision rule is merely exemplary. Any decision rule may be set according to actual needs, and the present disclosure is not limited thereto. For example, if the text information is "Create an avatar for the Year of the Dragon," and the text information does not contain any real-world objects, the second instruction information included in the second decision information indicates that the second database does not need to be searched. If the text information is "A still photo of actor A in the play XX," the second instruction information included in the second decision information indicates that the second database needs to be searched.

[0076] An embodiment of the present disclosure generates second decision information using a large-scale language model and acquires image information in different ways depending on different situations indicated by the second instruction information in the second decision information, thereby ensuring the accuracy of the image information (embodied by acquiring image information through search) while improving the diversity of the image information (embodied by generating image information).

[0077] Hereinafter, the principle of obtaining multimodal text by rendering will be further expanded with reference to Fig. 5. Fig. 5 is a schematic diagram of the principle of obtaining multimodal text by rendering according to an embodiment of the present disclosure.

[0078] According to an embodiment of the present disclosure, before invoking a multimodal text rendering tool to obtain multimodal text through rendering, layout information for the multimodal text can be generated based on text information and image information using a large-scale language model. Then, the multimodal text is rendered based on the layout information. Compared to rendering multimodal text with a fixed layout, rendering multimodal text using the layout information generated by the large-scale language model allows the form of the multimodal text to be adaptively determined according to the content, which is advantageous in improving the diversity and richness of the content form of the multimodal text obtained through rendering.

[0079] 5 , in this embodiment 500, after obtaining text information 501 and image information 502, a large-scale language model 510 is used to perform a layout generation task based on the text information 501 and image information 502, and layout information 520 for the multimodal text is generated by the large-scale language model 510. Then, the large-scale language model 510 is used to invoke a multimodal text rendering tool 530 based on the layout information 520, which is then used to render and obtain multimodal text 540.

[0080] Specifically, input information for the large-scale language model 510 can be obtained based on the text information 501 and the image information 502. For example, the text information 501 and the image information 502 can be mapped to the same feature space and then spliced ​​together, and the features obtained by splicing them can be used as input information. The large-scale language model 510 processes the input information to perform a layout generation task and obtain layout information 520. The layout information may include, for example, position information of bounding boxes in the multimodal text of the text information 501 and the image information 502, and the position information of each bounding box further includes indication information indicating a correspondence between the bounding box and the text information 501 or the image information 502. For example, the indication information can indicate whether text information or image information corresponds to the corresponding bounding box.

[0081] In one embodiment, after obtaining the layout information 520, the layout information 520, the text information 501, and the image information 502 are used as inputs to the large-scale language model 510. Based on the function of the large-scale language model invoking its external tool, the layout information 520, the text information 501, and the image information 502 are used as input parameters for the calling interface of the multimodal text rendering tool 530. The multimodal text 540 is fed back from the calling interface of the multimodal text rendering tool 530, and the multimodal text 540 is output as output data of the large-scale language model 510.

[0082] In one embodiment, when the large-scale language model 510 performs a layout generation task, in addition to inputting the text information 501 and the image information 502 into the large-scale language model 510, the input information of the large-scale language model may further include instruction information instructing the large-scale language model 510 to perform the layout generation task, and the presentation information may be, for example, "Please generate layout information for a multimodal text including the following text and image." It should be understood that the presentation information is merely an example for facilitating understanding of the present disclosure, and the present disclosure is not limited thereto.

[0083] In one embodiment, when the large-scale language model 510 performs a layout generation task, the input information of the large-scale language model may further include, for example, size information of the predetermined multimodal text, which may include, for example, the height and width of the multimodal text, thereby making the layout information generated by the large-scale language model 510 more reasonable. The size information of the predetermined multimodal text may, for example, be adapted to a client application that displays the multimodal text or to a screen size of a terminal device that displays the multimodal text, and the present disclosure is not limited thereto.

[0084] In one embodiment, before invoking the multimodal text rendering tool using a large-scale language model, base image information for the multimodal text may be determined based on image information, where the base image information may be understood as a background image containing the multimodal text. For example, this embodiment may use a large-scale language model to search for base image information from a base image database based on the image information. For example, the color of the background image included in the determined base image information should be distinguishable from the color of the image included in the image information, thereby improving the readability of the generated multimodal text. For example, the color of the background image included in the determined base image information may match the color of the image included in the image information, where the match may refer to the color systems of both satisfying a predetermined color system coordination condition to improve the aesthetics of the generated multimodal text.

[0085] After obtaining the base image information, in this embodiment, the large-scale language model 510 can call a multimodal text rendering tool based on the layout information and base image information. Specifically, the layout information, map information, text information, and image information are input to the large-scale language model, and based on the function of the large-scale language model to call an external tool, the layout information, base image information, text information, and image information are input parameters to the calling interface of the multimodal text rendering tool, and the multimodal text fed back from the calling interface of the multimodal text rendering tool is received and output as output data of the large-scale language model.

[0086] This embodiment first determines base image information based on image information, and then renders multimodal text based on the layout information and base image information, thereby making the rendered multimodal text more in line with actual needs and favorably improving the readability and aesthetics of the rendered multimodal text.

[0087] The principle by which the method for generating multimodal text provided by the embodiment of the present disclosure can be realized will be described below with reference to Fig. 6. Fig. 6 is a system architecture diagram of the method for generating multimodal text according to the embodiment of the present disclosure.

[0088] 6, a system architecture 600 for realizing a method for generating multimodal text may include a layer 610 for implementing a workflow for generating multimodal text, a central control layer 620, and a tool layer 630. Here, the workflow for generating multimodal text includes operations S611 to S613 that are performed depending on a large-scale language model.

[0089] Operation S611 generates text. Specifically, the large-scale language model generates text information based on the input proposal information proposal 601. In the process of generating the text information, the large-scale language model may first generate the aforementioned first determination information. If the first determination information indicates that a database needs to be searched, the large-scale language model may act as an agent of the search engine 621 in the central control layer 620 to search for knowledge from the database 622. Then, the large-scale language model may generate text information based on the searched knowledge.

[0090] Operation S612 generates an image. Specifically, the large-scale language model can generate image information based on the text information generated by operation S611. In the process of generating the image information, the large-scale language model can first generate the above-mentioned second determination information. If the second determination information indicates that a database needs to be searched, the large-scale language model can call an image search tool included in the image generation tool 631 in the tool layer 630 to search the database 622 in the central control layer 620 as an agent to search for image information. If the second determination information indicates that a database does not need to be searched, the system can call the image generation tool included in the image generation tool 631 to generate image information.

[0091] Operation S613 generates multimodal text. Specifically, the large-scale language model can generate multimodal text based on the text information generated by operation S611 and the image information generated by operation S612. In the process of generating the multimodal text, the large-scale language model can call a layout generation tool (which may be understood as the large-scale language model itself) included in the multimodal text generation tool 632 in the tool layer 630 to generate layout information for the multimodal text. At the same time, the large-scale language model can call a background search tool included in the multimodal text generation tool 632 to search for base image information that matches the image information. Then, the large-scale language model can call an automatic filling and image rendering tool (i.e., a multimodal text rendering tool) included in the multimodal text generation tool 632 to perform rendering based on the base image information, text information, image information, and layout information, thereby obtaining multimodal text 602.

[0092] In the system architecture 600, tools such as image search, image generation, and multimodal text generation are embedded into the overall system framework, and a large-scale language model is used to simulate the decision-making process as an agent. In the multimodal text generation process, the large-scale language model generates execution links representing the tool invocation sequence, and the corresponding tools are invoked one by one in order to generate text information, image information, and multimodal text, ultimately generating the multimodal text. The system architecture 600 can access products such as search engines and provide new knowledge to these products.

[0093] In the system architecture 600, the large-scale language model is used as an agent to achieve overall control of the multimodal text generation process, automatically scheduling each tool to execute each step, and achieving automated generation of multimodal text. Furthermore, using the large-scale language model as an agent allows the large-scale language model to learn specific character expression methods, making the generated multimodal text more realistic and significantly reducing the AI-like feel of the generated multimodal text. Furthermore, in the system architecture 600, the large-scale language model supports access to search engines, effectively alleviating the problem of the outdated nature of the knowledge learned by the model, thereby making the generated multimodal text more time-sensitive.

[0094] In the system architecture 600, the large-scale language model can perform decision-making tasks, specifically, can determine in real time how to perform the next step according to the actual situation, rather than performing step by step according to a fixed method or process, so that different step flows can be performed for different tasks, thereby improving the richness and diversity of the final generated multimodal text content.

[0095] In one embodiment, in the process of generating multimodal text, the method provided by the present disclosure can also generate an information stream for the large-scale language model to embody the input information and output information of the large-scale language model during the process of generating multimodal text (specifically, during the process of performing each linking step). For example, the information stream includes multiple sets of information, each set of information including the input information and output information of the large-scale language model, and the multiple sets of information are arranged in the order in which the large-scale language model outputs its output information. For example, in one embodiment, the information stream can be expressed as {presentation information, first determination information + search statement}, {search statement, text information}, {text information + presentation information, second determination information + search parameters}, {search parameters, image information}, {image information + text information, layout information}, or {image information + text information + layout information, multimodal text}.

[0096] After generating the information stream, this embodiment can also display the generated information stream for the salesperson to analyze the rationality of the information stream. After the salesperson obtains a rational information stream through analysis, they can perform a selection operation on the information stream. The method of this embodiment can also determine the information stream as a target information stream in response to the selection operation on the information stream. The target information stream can be used as a sample for continuously optimizing the large-scale language model, and the large-scale language model can be fine-tuned based on the target information stream. In this way, the decision-making ability and information generation ability of the large-scale language model can be continuously improved, and the quality of the generated multimodal text can be continuously improved.

[0097] In one embodiment, in addition to generating information corresponding to the task to be performed, the large-scale language model may also generate, for example, a confidence level for the generated information corresponding to the task to be performed. Accordingly, when the large-scale language model performs a task, the input information may further include, for example, prompting information to output the confidence level of the generated information. For example, in a task of generating text information, the input information of the large-scale language model may further include prompting information such as "Please provide the confidence level of the generated text information." Alternatively, training may be performed so that the large-scale language model directly generates the confidence level without providing prompting information for outputting the confidence level. That is, generating the confidence level is the default task of the large-scale language model. It should be understood that when the large-scale language model generates at least one of first determination information, text information, second determination information, image information, layout information, and multimodal text, it can output a confidence level corresponding to the at least one piece of information.

[0098] If the large-scale language model further generates a confidence level, the method of this embodiment may compare the confidence level corresponding to the at least one piece of information with a confidence threshold. If the generated confidence level is lower than the confidence threshold, the method may return to generating the at least one piece of information using the large-scale language model. That is, the task of generating the at least one piece of information using the large-scale language model is re-executed until the confidence level of the at least one piece of information generated by the large-scale language model is equal to or greater than the confidence threshold. This method allows the large-scale language model to have backoff reflection capabilities, which is beneficial to improving the quality and accuracy of the final generated multimodal text and also minimizes the possibility of failing to generate multimodal text.

[0099] Based on the multimodal text generation method provided by the present disclosure, the present disclosure further provides a multimodal text acquisition method, which will be described in detail below with reference to FIG.

[0100] FIG. 7 is a flowchart of a method for acquiring multimodal text according to an embodiment of the present disclosure.

[0101] As shown in FIG. 7, the method 700 for obtaining multimodal text in this embodiment may include operations S710 to S720.

[0102] In operation S710, in response to receiving the presentation information, a multimodal text generation request including the presentation information is sent.

[0103] According to an embodiment of the present disclosure, the presentation information may be input by a user to a terminal device via an interaction interface provided by the terminal device. After receiving the input presentation information, the terminal device may generate a multimodal text generation request including the presentation information and send the multimodal text generation request to a server. As described above, the presentation information may be topic information of the multimodal text to be generated, keywords of the multimodal text to be generated, etc., and further description thereof will be omitted here.

[0104] In operation S720, in response to obtaining the multimodal text generated in response to the multimodal text generation request, the multimodal text is displayed.

[0105] Here, the multimodal text is generated using the above-described multimodal text generation method. For example, after receiving a multimodal text generation request, the server can generate multimodal text using the above-described multimodal text generation method and feed the multimodal text back to the terminal device. After receiving the multimodal text, the terminal device can display the multimodal text.

[0106] According to the multimodal text acquisition method provided by the present disclosure, in the process of generating multimodal text, the user only needs to provide presentation information and does not need to perform other operations, which can improve the automation degree of multimodal text acquisition and is advantageous for improving the user experience.

[0107] Based on the multimodal text generation method provided by the present disclosure, the present disclosure further provides a multimodal text generation device, which will be described in detail below with reference to FIG.

[0108] FIG. 8 is a block diagram of a configuration of a multimodal text generation device according to an embodiment of the present disclosure.

[0109] As shown in FIG. 8, the multimodal text generating device 800 of this embodiment may include a text generating module 810, an image generating module 820 and a multimodal text generating module 830.

[0110] In response to receiving a multimodal text generation request including the presentation information, the text generation module 810 generates text information corresponding to the presentation information using a large-scale language model based on the presentation information. In one embodiment, the text generation module 810 is used to perform the above-mentioned operation S210, and the description thereof will be omitted here.

[0111] The image generation module 820 generates image information corresponding to the text information based on the text information using a large-scale language model. In one embodiment, the image generation module 820 is used to perform the above-mentioned operation S220, and the description thereof will be omitted here.

[0112] The multimodal text generation module 830 invokes a multimodal text rendering tool using a large-scale language model based on the text information and the image information to render and obtain a multimodal text including the text information and the image information. In one embodiment, the multimodal text generation module 830 is used to perform the above-mentioned operation S230, and the description thereof will be omitted here.

[0113] According to an embodiment of the present disclosure, the text generation module 810 may include a first determination generation submodule and a first text generation submodule. The first determination generation submodule is used to process the presented information using a large-scale language model to generate first determination information, where the first determination information includes first instruction information indicating whether a first database needs to be searched, and if the first instruction information indicates that the first database needs to be searched, the first determination information further includes a search statement. In response to the first instruction information indicating that the first database needs to be searched, the first text generation submodule performs a search expansion generation task based on the search statement using the large-scale language model to generate text information.

[0114] According to an embodiment of the present disclosure, the text generation module 810 may further include a second text generation sub-module for performing a text generation task based on the presented information using a large-scale language model to generate text information in response to the first instruction information indicating that there is no need to search the first database.

[0115] According to an embodiment of the present disclosure, the image generation module 820 may include a second decision generation sub-module and a first image generation sub-module. The second decision generation sub-module processes input information using a large-scale language model to generate second decision information, where the input information is obtained based on the presentation information and the text information, and the second decision information includes second instruction information indicating whether a second database needs to be searched. If the second instruction information indicates that the second database needs to be searched, the second decision information further includes search parameters. In response to the second instruction information indicating that the second database needs to be searched, the first image generation sub-module performs an image search task based on the search parameters using the large-scale language model to obtain image information.

[0116] According to an embodiment of the present disclosure, if the second instruction information indicates that there is no need to search the second database, the second determination information further includes an image description. The image generation module 820 may further include a second image generation sub-module for performing an image generation task based on the image description using a Text-to-Image model to generate image information in response to the second instruction information indicating that there is no need to search the second database.

[0117] According to an embodiment of the present disclosure, the multimodal text generation module 830 may include a layout generation submodule and a rendering submodule. The layout generation submodule performs a layout generation task based on text information and image information using a large-scale language model to generate layout information for the multimodal text. The rendering submodule invokes a multimodal text rendering tool based on the layout information using the large-scale language model to render and obtain the multimodal text.

[0118] According to an embodiment of the present disclosure, the multimodal text generation device 800 may further include a base image determination module for determining base image information for the multimodal text based on image information. The rendering submodule specifically invokes a multimodal text rendering tool based on the layout information and the base image information using a large-scale language model to render and obtain the multimodal text.

[0119] According to an embodiment of the present disclosure, the information generated by the large-scale language model includes generated information corresponding to a task performed by the large-scale language model and a confidence level for the generated information, and the generated information includes at least one of text information, image information, and multimodal text. The multimodal text generation device 800 may further include a scheduling module that, in response to the confidence level for the generated information being lower than a confidence threshold, schedules the module that generates the generated information to re-execute the task of generating the generated information using the large-scale language model.

[0120] According to an embodiment of the present disclosure, the multimodal text generation device 800 may further include an information stream generation module, a display module, and an information stream determination module. The information stream generation module generates an information stream for the large-scale language model in the process of generating multimodal text, and the information stream indicates input information and output information of the large-scale language model. The display module displays the information stream. The information stream determination module determines the information stream as a target information stream in response to a selection operation on the information stream. Here, the target information stream is used to fine-tune the large-scale language model.

[0121] Based on the multimodal text acquisition method provided by the present disclosure, the present disclosure further provides a multimodal text acquisition device, which will be described in detail below with reference to FIG.

[0122] FIG. 9 is a block diagram of a configuration of a multimodal text acquisition device according to an embodiment of the present disclosure.

[0123] As shown in FIG. 9, the multimodal text acquisition device 900 of this embodiment may include an information transmitting and receiving module 910 and a multimodal text display module 920.

[0124] In response to receiving the presentation information, the information transmitting and receiving module 910 transmits a multimodal text generation request including the presentation information. In one embodiment, the information transmitting and receiving module 910 is used to perform the above-mentioned operation S710, and the description thereof will be omitted here.

[0125] The multimodal text display module 920 displays the multimodal text in response to the information transmitting and receiving module receiving the multimodal text generated in response to the multimodal text generation request, where the multimodal text is generated using the multimodal text generation device described above. In one embodiment, the multimodal text display module 920 is used to perform the operation S720 described above, and will not be described again here.

[0126] In addition, the collection, storage, use, processing, transmission, provision, disclosure, application, etc. of user personal information in the technical solution disclosed herein complies with the provisions of relevant laws and regulations, takes necessary security measures, and is not contrary to public order and morals. In the technical solution disclosed herein, the permission or consent of the user is obtained before obtaining or collecting the user's personal information.

[0127] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0128] 10 shows a schematic block diagram of an example electronic device 1000 for implementing a method for generating or acquiring multimodal text according to an embodiment of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0129] 10, the device 1000 includes a computing unit 1001, which may perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 1002 or loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may further store various programs and data necessary for the operation of the device 1000. The computing unit 1001, the ROM 1002, and the RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0130] The components of the device 1000 are connected to an I / O interface 1005, which includes an input unit 1006 such as a keyboard, a mouse, etc., an output unit 1007 such as various types of displays, speakers, etc., a storage unit 1008 such as a magnetic disk, an optical disk, etc., and a communication unit 1009 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 enables the device 1000 to exchange information and data with other devices via a computer network such as the Internet and / or various electrical networks.

[0131] The computing unit 1001 may be any of a variety of general-purpose and / or specialized processing modules having processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, computing units running various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs each of the methods and processes described above, such as the multimodal text generation method or the multimodal text acquisition method. For example, in some embodiments, the multimodal text generation method or the multimodal text acquisition method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, some or all of the computer program may be loaded and / or installed into the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, it may perform one or more steps of the method for generating a multimodal text or the method for obtaining a multimodal text described above. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the method for generating a multimodal text or the method for obtaining a multimodal text in any other suitable manner (e.g., via firmware).

[0132] Various embodiments of the systems and techniques described herein above may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that can be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and that can receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0133] Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, so that when the program code is executed by the processor or controller, the functions and operations specified in the flowcharts and / or block diagrams are performed. The program code may be executed entirely on a device, partially on a device, partially on a device as a separate software package, and partially on a remote device, or entirely on a remote device or server.

[0134] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program for use in or in connection with an instruction execution system, device, or electronic device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or electronic device, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection of one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] To provide for user interaction, a computer may implement the systems and techniques described herein and include a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the computer. Other types of devices may also provide for user interaction; for example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and may receive input from the user in any form (including voice input, speech input, or tactile input).

[0136] The systems and techniques described herein can be implemented in a computing system including background components (e.g., a data server), or a computing system including middleware components (e.g., an application server), or a computing system including front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system including any combination of such background, middleware, or front-end components. The components of the system can be connected to each other by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include, by way of example, a local area network (LAN), a wide area network (WAN), and the Internet.

[0137] The computer system may include a client and a server. The client and server are generally separate and typically interact via a communication network. The relationship between the client and the server is established by a computer program running on the corresponding computer and having a client-server relationship. Here, the server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in a cloud computing service system and solves the problems of traditional physical hosts and VPS services (also referred to as "Virtual Private Server" or "VPS"), such as high management difficulty and poor service scalability. The server may be a server in a distributed system or a server combined with blockchain.

[0138] It should be understood that various types of flows shown above may be used, and operations may be rearranged, added, or deleted. For example, the operations described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this specification is not limited thereto.

[0139] The above specific embodiments do not limit the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. In response to receiving a multimodal text generation request including presentation information, generating text information corresponding to the presentation information based on the presentation information using a large-scale language model; generating image information corresponding to the text information using the large-scale language model based on the text information; and invoking a multimodal text rendering tool based on the text information and the image information using the large-scale language model to render and obtain a multimodal text including the text information and the image information. How to generate multimodal text.

2. generating text information corresponding to the presentation information using a large-scale language model based on the presentation information, processing the presentation information using the large-scale language model to generate first determination information, the first determination information including first indication information indicating whether a first database needs to be searched, and when the first indication information indicates that the first database needs to be searched, the first determination information further includes a search statement; in response to the first instruction information indicating that the first database needs to be searched, performing a search expansion generation task based on the search statement using the large-scale language model to generate the text information. The method of claim 1.

3. generating text information corresponding to the presentation information using a large-scale language model based on the presentation information, and, in response to the first instruction information indicating that the first database does not need to be searched, performing a text generation task based on the presentation information using the large-scale language model to generate the text information. The method of claim 2.

4. generating image information corresponding to the text information using the large-scale language model based on the text information, processing input information using the large-scale language model to generate second determination information, the input information being obtained based on the presentation information and the text information, the second determination information including second instruction information indicating whether a second database needs to be searched, and when the second instruction information indicates that the second database needs to be searched, the second determination information further includes search parameters; In response to the second instructional information indicating that the second database should be searched, performing an image retrieval task using the large-scale language model based on the search parameters to obtain the image information. The method of claim 1.

5. When the second instruction information indicates that there is no need to search the second database, the second determination information further includes an image description sentence, and generating image information corresponding to the text information using the large-scale language model based on the text information includes: and, in response to the second instruction information indicating that the second database does not need to be searched, executing an image generation task based on the image description sentence using a Text-to-Image model to generate the image information. The method according to claim 4.

6. Invoking a multimodal text rendering tool using the large-scale language model based on the text information and the image information to obtain a multimodal text including the text information and the image information, performing a layout generation task based on the text information and the image information using the large-scale language model to generate layout information for the multimodal text; and invoking the multimodal text rendering tool based on the layout information using the large-scale language model to render the multimodal text. The method of claim 1.

7. determining based on the image information, basis image information for the multimodal text; Invoking the multimodal text rendering tool based on the layout information using the large-scale language model to render the multimodal text includes invoking the multimodal text rendering tool based on the layout information and the basis image information using the large-scale language model to render the multimodal text. The method according to claim 6.

8. The information generated by the large-scale language model includes generated information corresponding to a task performed by the large-scale language model and a confidence level for the generated information, and the generated information includes at least one of text information, image information, and multimodal text; The method further includes, in response to a confidence in the generated information being below a confidence threshold, re-performing a task of generating the generated information using the large scale language model. The method according to any one of claims 1 to 7.

9. generating an information stream for the large-scale language model in the process of generating the multimodal text, the information stream representing input information and output information of the large-scale language model; displaying the information stream; and in response to a selection operation on the information stream, determining the information stream as a target information stream for fine-tuning the large-scale language model. The method according to any one of claims 1 to 7.

10. in response to receiving the presentation information, sending a multimodal text generation request including the presentation information; and in response to obtaining multimodal text generated in response to the multimodal text generation request, displaying the multimodal text; The multimodal text was generated using the method of claim 1. How to get multimodal text.

11. a text generation module that, in response to receiving a multimodal text generation request including presentation information, generates text information corresponding to the presentation information based on the presentation information using a large-scale language model; an image generation module that generates image information corresponding to the text information by using the large-scale language model based on the text information; a multimodal text generation module that invokes a multimodal text rendering tool using the large-scale language model based on the text information and the image information to obtain a multimodal text including the text information and the image information through rendering. A multimodal text generator.

12. The text generation module includes: a first decision generating submodule for processing the presentation information using the large-scale language model to generate first decision information, the first decision information including first indication information indicating whether a first database needs to be searched, and when the first indication information indicates that the first database needs to be searched, the first decision information further includes a search statement; a first text generation submodule that, in response to the first instruction information indicating that the first database needs to be searched, performs a search expansion generation task based on the search statement using the large-scale language model to generate the text information.

12. The apparatus of claim 11.

13. The text generation module further includes a second text generation sub-module that, in response to the first instruction information indicating that the first database does not need to be searched, performs a text generation task based on the presentation information using the large-scale language model to generate the text information.

13. The apparatus of claim 12.

14. The image generation module includes: a second decision generating submodule for processing input information using the large-scale language model to generate second decision information, the input information being obtained based on the presentation information and the text information, the second decision information including second instruction information indicating whether a second database needs to be searched, and when the second instruction information indicates that the second database needs to be searched, the second decision information further includes a search parameter; a first image generation submodule that, in response to the second instruction information indicating that the second database needs to be searched, performs an image retrieval task based on the search parameters using the large-scale language model to obtain the image information.

12. The apparatus of claim 11.

15. When the second instruction information indicates that the second database does not need to be searched, the second determination information further includes an image description sentence, and the image generating module: and a second image generation sub-module that, in response to the second instruction information indicating that the second database does not need to be searched, executes an image generation task based on the image description sentence using a Text-to-Image model to generate the image information.

15. The apparatus of claim 14.

16. The multimodal text generation module includes: a layout generation submodule that performs a layout generation task based on the text information and the image information using the large-scale language model to generate layout information for the multimodal text; a rendering submodule for invoking the multimodal text rendering tool based on the layout information using the large-scale language model to render the multimodal text; 12. The apparatus of claim 11.

17. Further comprising a base image determination module for determining base image information for the multimodal text based on the image information; wherein the rendering submodule invokes the multimodal text rendering tool based on the layout information and the base image information using the large-scale language model, and performs rendering to obtain the multimodal text; 17. The apparatus of claim 16.

18. The information generated by the large-scale language model includes generated information corresponding to a task performed by the large-scale language model and a confidence level for the generated information, and the generated information includes at least one of text information, image information, and multimodal text; The apparatus further includes a scheduling module that, in response to a confidence level for the generated information being lower than a confidence threshold, schedules the module for generating the generated information to re-execute a task of generating the generated information using the large-scale language model. Apparatus according to any one of claims 11 to 17.

19. an information stream generation module for generating an information stream for the large-scale language model in the process of generating the multimodal text, the information stream representing input information and output information of the large-scale language model; a display module for displaying said information stream; an information stream determination module for determining, in response to a selection operation on the information stream, the information stream as a target information stream for fine-tuning the large-scale language model. Apparatus according to any one of claims 11 to 17.

20. an information transmitting / receiving module that transmits a multimodal text generation request including the presentation information in response to receiving the presentation information; and a multimodal text display module that displays the multimodal text generated by the device according to any one of claims 11 to 17 in response to the information transmitting and receiving module acquiring the multimodal text generated in response to the multimodal text generation request. A multimodal text acquisition device.

21. At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor performs the method according to any one of claims 1 to 7 and 10. electronic equipment.

22. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: The computer instructions are used to cause the computer to carry out the method according to any one of claims 1 to 7 and 10. A non-transitory computer-readable storage medium.

23. A computer program stored on at least one of a readable storage medium and an electronic device, which, when executed by a processor, implements the method according to any one of claims 1 to 7 and 10.

Images