Multi-modal controllable text generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a multi-mode controllable text generation method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining a text control element and a visual control element, carrying out at least two times of text generation reasoning based on the text control element and the visual control element, and obtaining at least two text generation intermediate results, and splicing the at least two text generation intermediate results to obtain a thinking chain prompt, iteratively performing text generation reasoning based on the text control element, the thinking chain prompt and the visual control element until the text generation intermediate results converge, and taking the convergent text generation intermediate results as text generation results, and splicing a text generation intermediate result output by the current round of iteration into a thinking chain prompt input by the current round of iteration to serve as a thinking chain prompt input by the next round of iteration. According to the method, the training cost is low, the generated text context semantics are smooth, and the text can be matched with visual information, so that higher-level multi-modal control is realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of natural language generation, and in particular to a multimodal controllable text generation method, device, electronic device, storage medium and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Controllable text generation refers to the process of controlling the specific attributes or content of the generated text by introducing certain constraints or guiding factors during the text generation process. This technology allows users to specify the style, theme, emotion and other characteristics of the generated text, so that the generated text can better meet the expected needs.

[0004] However, in the related art, the controllable text generation method has high training cost and high price, the generated text context semantics are not smooth, and it is impossible to match the text with the visual information, thus failing to achieve a higher level of multimodal control. Summary of the invention

[0005] In view of this, the purpose of the present disclosure is to propose a multimodal controllable text generation method, device, electronic device, storage medium and program product, which at least to a certain extent solve one of the technical problems in the related art.

[0006] Based on the above purpose, the first aspect of the exemplary embodiment of the present disclosure provides a multimodal controllable text generation method, which is applied to a server, and the method includes:

[0007] Identify textual and visual control elements;

[0008] Based on the text control element and the visual control element, perform at least two text generation reasonings to obtain at least two text generation intermediate results;

[0009] Splicing the intermediate results generated from the at least two texts to obtain a thought chain prompt;

[0010] Based on the text control elements, the thinking chain prompts and the visual control elements, the text generation reasoning is iterated until the intermediate results of text generation converge, and the converged text generation intermediate results are used as the text generation results, wherein the text generation intermediate results output by this round of iteration are spliced ​​into the thinking chain prompts input by this round of iteration as the thinking chain prompts input by the next round of iteration.

[0011] Based on the same inventive concept, the second aspect of the exemplary embodiment of the present disclosure provides a multimodal controllable text generation device, including:

[0012] An element determination module configured to determine text control elements and visual control elements;

[0013] An intermediate result determination module is configured to perform at least two text generation reasonings based on the text control element and the visual control element to obtain at least two text generation intermediate results;

[0014] A thinking chain prompt determination module is configured to splice the intermediate results generated by the at least two texts to obtain a thinking chain prompt;

[0015] The iteration module is configured to iteratively perform text generation reasoning based on the text control elements, the thinking chain prompts and the visual control elements until the intermediate results of text generation converge, and use the converged text generation intermediate results as the text generation results, wherein the text generation intermediate results output by this round of iteration are spliced ​​into the thinking chain prompts input by this round of iteration as the thinking chain prompts input by the next round of iteration.

[0016] Based on the same inventive concept, a third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0017] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0018] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the method described in the first aspect.

[0019] From the above, it can be seen that the embodiments of the present disclosure provide a multimodal controllable text generation method, device, electronic device, storage medium and program product, the method comprising: determining a text control element and a visual control element, performing at least two text generation reasonings based on the text control element and the visual control element, obtaining at least two text generation intermediate results, splicing the at least two text generation intermediate results to obtain a thinking chain prompt, iterating the text generation reasoning based on the text control element, the thinking chain prompt and the visual control element until the text generation intermediate result converges, and using the converged text generation intermediate result as the text generation result, wherein the text generation intermediate result output by this round of iteration is spliced ​​into the thinking chain prompt input by this round of iteration as the thinking chain prompt input by the next round of iteration. The present disclosure has low training cost, the generated text context semantics are fluent, and the text can be matched with visual information, thereby achieving a higher level of multimodal control. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of an application scenario of a multimodal controllable text generation method provided by an exemplary embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a process for generating a multimodal controllable text provided by an exemplary embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of the structure of a multimodal controllable text generation device provided for an exemplary embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a graphical user interface for generating multimodal controllable text provided by an exemplary embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of a graphical user interface in simplified output mode for generating multimodal controllable text provided by an exemplary embodiment of the present disclosure;

[0027] Figure 7A schematic diagram of a graphical user interface in detailed output mode for generating multimodal controllable text provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0035] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "a" before an element does not exclude the existence of multiple such elements.

[0036] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0037] As described in the background technology, in the related art, the training cost of multimodal controllable text generation method is high, the generated text context semantics is not smooth, and it is impossible to match the text with the visual information, so as to achieve a higher level of multimodal control.

[0038] Specifically, in the related art, a controllable text generation method based on a large language model, first, constructs an opinion database according to opinion data. Then, the first hot event of the first media platform and the first hot content of the first hot event are input into the hot content summary model to obtain the first summary content of the first hot event. Next, according to the opinion database, the first post of the first hot event and the first summary content of the first hot event, the post filtering model is used to filter the first post of the first hot event that is inconsistent with the opinion database. Finally, the filtered first post of the first hot event, the first summary content of the first hot event and the target user group information are input into the text generation large model to obtain controllable text.

[0039] However, in order to generate more controllable text, this method may need to combine multiple models and algorithms. Although it improves the accuracy and efficiency of the generated text, it requires a lot of computing resources.

[0040] In the related art, a zero-shot multimodal controllable text generation method, in the inference stage, for text control, ZeroGen first calculates the cosine similarity between each candidate token and the text control word, and integrates it into the original probability distribution to achieve text control. Then, for visual control, it uses CLIP to evaluate the similarity between k candidate tokens and k candidate sentences and images spliced ​​from the generated text, and further integrates this probability distribution into the original probability distribution.

[0041] However, this method CLIP calculates the similarity at the image feature level instead of directly evaluating the correlation between text and image, which may lead to inaccurate text control effects. In addition, CLIP trains the model by maximizing the similarity of positive samples and minimizing the similarity of negative samples, but this method may not be fully adaptable to all types of text and image data. These factors may affect the performance and accuracy of ZeroGen in multimodal controllable text generation, resulting in unsmooth semantic context of the generated text.

[0042] In the related art, a controllable text generation method based on a pre-trained language model is proposed. The core of this method is to train a discriminator model of topics, emotions and writing styles, and use the probability decomposition of the Bayesian formula to combine the output probability of the pre-trained language model and the output probability of the discriminator model. Compared with the model that is directly trained to meet the constraints, the method of this invention does not need to make any changes to the pre-trained language model itself, but guides the model to generate content that meets the constraints through the attribute discriminator in the model reasoning stage.

[0043] However, although this method saves the computational resources of training large-scale pre-trained language models, it is unable to match text with visual information to achieve a higher level of multimodal control.

[0044] In order to solve the above problems, the present disclosure provides a multimodal controllable text generation method, device, electronic device, storage medium and program product solution, which specifically include:

[0045] Determine text control elements and visual control elements, perform at least two text generation reasonings based on the text control elements and the visual control elements, obtain at least two text generation intermediate results, splice the at least two text generation intermediate results to obtain a thinking chain prompt, iterate text generation reasoning based on the text control elements, the thinking chain prompt and the visual control elements until the text generation intermediate results converge, and use the converged text generation intermediate results as the text generation results, wherein the text generation intermediate results output from this round of iteration are spliced ​​into the thinking chain prompts input from this round of iteration as the thinking chain prompts input for the next round of iteration. This scheme makes full use of the pre-training capability of the large language model and does not require additional large-scale data training, thereby reducing the cost and time of model training. Secondly, this scheme uses the powerful natural language generation capability of the large language model to guide text generation through thought chain prompts to ensure that the generated text is fluent and consistent with the contextual semantics. In addition, this scheme combines the image-text matching model in visual control and generates chain thought prompts through multiple iterations to gradually improve the matching and consistency between text and visual information. Referring to Table 1, when the chain thought prompts are removed, the BLEU-1, BLEU-4, ROUGE-L and CIDEr scores decrease slightly. BLEU-1, BLEU-4, ROUGE-L and CIDEr are four commonly used evaluation indicators in the field of natural language processing (NLP), which are used for different tasks and application scenarios. This shows that the model is less accurate in generating text that matches the given control content. Secondly, the cyclic iterative generation is removed. This means that the generation process has only one round. At the same time, the thinking chain prompts are also removed because it is impossible to construct a thinking chain prompt when only one generation is performed. After removing the cyclic iterative generation, all indicators have dropped significantly, which shows that iterative generation is crucial to improving the accuracy and quality of the output. Therefore, by generating chain thinking prompts through multiple iterations, a higher level of multimodal control can be achieved.

[0046] Table 1 Natural language processing index evaluation table

[0047]

[0048] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0049] refer to Figure 1 , which is a schematic diagram of an application scenario of the multimodal controllable text generation method provided by an exemplary embodiment of the present disclosure.

[0050] This application scenario includes a terminal device 101 and a server 102. The terminal device 101 and the server 102 may be connected via a wired or wireless communication network to achieve data interaction.

[0051] The terminal device 101 may be an electronic device with data transmission and multimedia input / output functions close to the user side, including but not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of implementing the above functions. The electronic device may include a processor and a display screen with a touch input function, the display screen is used to present a graphical user interface, the graphical user interface can display an application interface, and the processor is used to process application data, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0052] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0053] In some exemplary embodiments, the multimodal controllable text generation method may be run on the terminal device 101 or the server 102 .

[0054] When the multimodal controllable text generation method is run on the server 102 , the server 102 is used to provide a multimodal controllable text generation service to a user of the terminal device 101 .

[0055] The terminal device 101 determines the text control element and the visual control element and sends them to the server 102;

[0056] The server 102 receives the text control element and the visual control element sent by the terminal device, and performs at least two text generation inferences based on the text control element and the visual control element to obtain at least two text generation intermediate results;

[0057] The server 102 splices the intermediate results generated by the at least two texts to obtain a thought chain prompt;

[0058] The server 102 iteratively performs text generation reasoning based on the text control elements, the thinking chain prompts and the visual control elements, and sends several output results of the generated reasoning to the terminal device 101. The server 102 continues until the text generation intermediate results converge, and then uses the converged text generation intermediate results as the text generation results and sends the text generation results to the terminal device 101. The server 102 splices the text generation intermediate results output from this round of iteration into the thinking chain prompts input for this round of iteration as the thinking chain prompts input for the next round of iteration.

[0059] refer to Figure 2 , a multimodal controllable text generation method is applied to a server, and the method comprises the following steps:

[0060] Step S210: Determine text control elements and visual control elements.

[0061] The following describes how to determine text control elements and visual control elements:

[0062] In this exemplary embodiment, a graphical user interface is provided by a terminal, and the content displayed by the graphical user interface includes an input interface and an output interface, and the input interface includes a text control control and a visual control control;

[0063] Identify textual and visual control elements, including:

[0064] In response to an input operation on the text control control, the input text control element is acquired; in response to an input operation on the visual control control, the input visual control element is acquired.

[0065] During specific implementation, determine the text control elements and visual control elements, including:

[0066] As a specific example, refer to Figure 5 The content displayed by the graphical user interface includes an input interface and an output interface. The input interface includes a text control control and a visual control control, wherein the user can input natural language through the text control control or click on the text control control to select preset text options, such as emotional text options (such as positive, negative, etc.) or thematic text options (such as scenery, animals or plants, etc.); the user can click on the visual control control to select files such as photos or videos from the local computer and upload them to the input interface.

[0067] Step S220: Based on the text control element and the visual control element, perform at least two text generation reasonings to obtain at least two text generation intermediate results.

[0068] Next, we introduce the method of obtaining each text generation reasoning in at least two text generation intermediate results:

[0069] In this exemplary embodiment, at least two text generation inferences are performed based on the text control element and the visual control element to obtain each text generation inference in at least two text generation intermediate results, including:

[0070] Inference is performed based on the text control element through a large language model to obtain several candidate texts, the several candidate texts are matched with the visual control element based on a contrastive language image model, and one of the candidate texts is used as a text to generate an intermediate result.

[0071] In specific implementation, the comparison language image model includes:

[0072] The Contrastive Language Image Processing (CLIP) is a multimodal pre-trained neural network model that aims to associate images and texts through contrastive learning. Its core idea is to use a large amount of unlabeled image and text data to pre-train the model, so as to achieve mutual understanding and alignment between images and texts.

[0073] In the above exemplary embodiment, a method for obtaining each text generation reasoning in at least two text generation intermediate results is introduced. Next, a method for obtaining several candidate texts is introduced:

[0074] In this exemplary embodiment, a large language model is used to perform inference based on the text control element to obtain several candidate texts, including:

[0075] Determine a number of natural language instructions of the text control element, take the number of natural language instructions as observation parameters of one of the candidate texts based on the large language model, update the probability distribution of the observation parameters, obtain the posterior probability of one of the candidate texts, perform probability calculation on the posterior probability, and obtain one of the candidate texts.

[0076] In specific implementation, the large language model includes:

[0077] Large Language Models (LLMs) are deep learning-based AI models designed to understand and generate human language. They are trained on large amounts of text data and are able to perform a wide range of natural language processing (NLP) tasks such as translating text, answering questions conversationally, classifying words and generating them based on knowledge gained from different datasets.

[0078] In a specific implementation, determining a plurality of natural language instructions of the text control element, and using the plurality of natural language instructions as observation parameters of one of the plurality of candidate texts based on the large language model, includes:

[0079] Through the natural language instructions input by the user on the terminal device, the server can understand the natural language instructions based on the large language model, representing the natural language instructions of text control. 1:n Used to directly represent text controls.

[0080] In a specific implementation, updating the probability distribution of the observation parameter to obtain the posterior probability of one of the candidate texts includes:

[0081] Due to the powerful ability of large language models in understanding natural language instructions, in order to maximize P(X 1:n |C T ) to search for X 1:n , which can be regarded as maximizing P(X 1:n |I 1:n ), which can be proved by the following calculation formula:

[0082] According to Bayes' theorem:

[0083]

[0084] Thanks to the powerful ability of large language models in understanding natural language instructions, these models can effectively and accurately generate text containing the desired text control content. Therefore, P(I|C T ) and P(C T |I) are close to 1, so they can be regarded as 1. Therefore,

[0085]

[0086] As a specific example, suppose that we want to prompt the model to generate positive text, and input the sentence "Generate some positive text" into the large language model, it can effectively and accurately generate the required positive text.

[0087] In a specific implementation, the posterior probability is probability calculated to obtain a candidate text from the plurality of candidate texts, including:

[0088] By comparing the posterior probabilities of the candidate texts, the candidate text with the largest posterior probability is selected as the output.

[0089] In the above exemplary embodiment, a method of obtaining several candidate texts is introduced. The following describes a method of obtaining texts and generating intermediate results:

[0090] In this exemplary embodiment, matching the candidate texts with the visual control element based on the contrastive language image model and using one of the candidate texts as text to generate an intermediate result includes:

[0091] The matching values ​​of the visual control element and each of the candidate texts are determined respectively; and the candidate text with the largest matching value is used as the text to generate an intermediate result.

[0092] In a specific implementation, respectively determining the matching value of the visual control element and each of the candidate texts includes:

[0093] A plurality of candidate texts and visual control elements are input into a contrastive language image model (CLIP) model, and a matching score between the image and the text is output to obtain a plurality of matching scores of the candidate texts and the visual control elements.

[0094] In a specific implementation, the candidate text with the largest matching value is used as the text to generate an intermediate result, including:

[0095] Select the text with the highest score from several candidate texts and use it as the current output.

[0096] Step S230: splicing the intermediate results generated by the at least two texts to obtain a thought chain prompt.

[0097] Here are some tips for thinking chains:

[0098] When implementing it, the thinking chain prompts include:

[0099] The thought chain prompt is an improved prompting strategy for improving the performance of large language models (LLMs) in complex reasoning tasks. It achieves this goal by guiding the model to generate a series of intermediate reasoning steps, thereby helping the model to better understand and solve complex arithmetic, common sense, and symbolic reasoning problems.

[0100] In the above exemplary embodiment, the thought chain prompt is introduced. The following describes the method of obtaining the thought chain prompt:

[0101] In a specific implementation, the step of generating intermediate results of at least two texts and splicing them to obtain a thought chain prompt includes:

[0102] According to the generation order of the at least two text-generated intermediate results, the at least two text-generated intermediate results are spliced ​​to obtain the thought chain prompt.

[0103] Step S240, based on the text control elements, the thinking chain prompts and the visual control elements, iteratively perform text generation reasoning until the intermediate results of text generation converge, and use the converged text generation intermediate results as the text generation results, wherein the text generation intermediate results output by this round of iteration are spliced ​​into the thinking chain prompts input by this round of iteration as the thinking chain prompts input by the next round of iteration.

[0104] In this exemplary embodiment, step S240 specifically includes:

[0105] The text control element and the first thought chain prompt are input into the large language model together, and the large language model is used to make inferences based on the text control element and the first thought chain prompt to obtain a number of candidate texts;

[0106] Inputting the visual control element and several candidate texts into a contrastive language image model, matching the several candidate texts and the visual control element based on the contrastive language image model, and taking one of the several candidate texts as a first text to generate an intermediate result;

[0107] The intermediate result generated by the first text is spliced ​​with the first thinking chain prompt to obtain a second thinking chain prompt;

[0108] The text control element and the second thought chain prompt are input into the large language model together, and the large language model is used to make inferences based on the text control element and the second thought chain prompt to obtain a number of candidate texts;

[0109] Inputting the visual control element and several candidate texts into a contrastive language image model, matching the several candidate texts and the visual control element based on the contrastive language image model, and taking one of the several candidate texts as a second text to generate an intermediate result;

[0110] The intermediate result generated by the first text is spliced ​​with the second thinking chain prompt to obtain a third thinking chain prompt;

[0111] The text control elements and the third thought chain prompt are input into the large language model together, and the large language model is used to make inferences based on the text control elements and the third thought chain prompt to obtain a number of candidate texts;

[0112] Inputting the visual control element and several candidate texts into a contrastive language image model, matching the several candidate texts and the visual control element based on the contrastive language image model, and taking one of the several candidate texts as a third text to generate an intermediate result;

[0113] …

[0114] And so on, the loop is iterated until the intermediate result of text generation converges, and the converged intermediate result of text generation is used as the text generation result.

[0115] In specific implementation, convergence is determined by comparing the similarity of model outputs at different time points. If the model gives almost the same output for the same input after multiple trainings, the model can be considered to have converged.

[0116] Next, we introduce the visualization method of the text generation result:

[0117] In this exemplary embodiment, based on the text control element, the thought chain prompt and the visual control element, iteratively performing text generation reasoning until the text generation intermediate result converges, and taking the converged text generation intermediate result as the text generation result, includes:

[0118] The text generation intermediate results generated in each iteration are sequentially displayed on the output interface, and the text generation results are displayed on the output interface.

[0119] In a specific implementation, the text generation intermediate results generated in each iteration are sequentially displayed on the output interface, and the text generation results are displayed on the output interface, including:

[0120] The specific embodiment of step S210 is as follows. Figure 5 In the input interface displayed by the graphical user interface, when the user inputs text control elements through the text control control and inputs visual control elements through the visual control control, by clicking the text generation control in the input interface, the output result of each round of iteration and the final generation result generated after the model converges can be displayed in the output interface displayed by the graphical user interface, and the user can use the mode switching control in the output interface to select between the simplified mode and the detailed mode output by the output interface. When performing text output in the simplified mode, refer to Figure 6 The output interface will only display the output results of each iteration and the final generated results. When using the detailed mode for text output, refer to Figure 7 In addition to displaying the output results and final generated results of each round of iteration, the output interface also displays the input thinking chain prompts and generated candidates for each round of iteration.

[0121] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0122] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0123] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a multimodal controllable text generation device.

[0124] refer to Figure 3 , the multimodal controllable text generation device comprises:

[0125] An element determination module 310, configured to determine text control elements and visual control elements;

[0126] The intermediate result determination module 320 is configured to perform at least two text generation reasonings based on the text control element and the visual control element to obtain at least two text generation intermediate results;

[0127] A thought chain prompt determining module 330 is configured to splice the intermediate results generated by the at least two texts to obtain a thought chain prompt;

[0128] The iteration module 340 is configured to iteratively perform text generation reasoning based on the text control elements, the thinking chain prompts and the visual control elements until the intermediate results of text generation converge, and use the converged text generation intermediate results as the text generation results, wherein the text generation intermediate results output by this round of iteration are spliced ​​into the thinking chain prompts input by this round of iteration as the thinking chain prompts input by the next round of iteration.

[0129] In this exemplary embodiment, the element determination module 310 is specifically configured as follows:

[0130] In response to an input operation on the text control control, the input text control element is acquired; in response to an input operation on the visual control control, the input visual control element is acquired.

[0131] In this exemplary embodiment, the intermediate result determination module 320 is specifically configured as follows:

[0132] Determine a number of natural language instructions of the text control element, take the number of natural language instructions as observation parameters of one of the candidate texts based on the large language model, update the probability distribution of the observation parameters, obtain the posterior probability of one of the candidate texts, perform probability calculation on the posterior probability to obtain one of the candidate texts, respectively determine the matching values ​​of the visual control element and each of the candidate texts, and take the candidate text with the largest matching value as the text generation intermediate result.

[0133] In this exemplary embodiment, the thought chain prompt determination module 330 is specifically configured as follows:

[0134] According to the generation order of the at least two text-generated intermediate results, the at least two text-generated intermediate results are spliced ​​to obtain the thought chain prompt.

[0135] In this exemplary embodiment, the iteration module 340 is specifically configured as follows:

[0136] Based on the text control elements, the thinking chain prompts and the visual control elements, text generation reasoning is performed iteratively, and the text generation intermediate results generated in each iteration are displayed in sequence on the output interface until the text generation intermediate results converge, and the converged text generation intermediate results are used as text generation results, and the text generation results are displayed on the output interface, wherein the text generation intermediate results outputted in this round of iteration are spliced ​​into the thinking chain prompts inputted in this round of iteration as the thinking chain prompts inputted in the next round of iteration.

[0137] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0138] The device of the above embodiment is used to implement the corresponding multimodal controllable text generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0139] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal controllable text generation method described in any of the above embodiments is implemented.

[0140] Figure 4A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0141] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0142] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0143] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0144] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0145] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0146] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0147] The electronic device of the above-mentioned embodiment is used to implement the corresponding multimodal controllable text generation method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0148] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multimodal controllable text generation method described in any of the above embodiments.

[0149] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0150] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0151] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multimodal controllable text generation method described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0152] Based on the same inventive concept, corresponding to the multimodal controllable text generation method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the multimodal controllable text generation method. Corresponding to the execution subject corresponding to each step in each embodiment of the multimodal controllable text generation method, the processor that executes the corresponding step can belong to the corresponding execution subject.

[0153] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the multimodal controllable text generation method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0154] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0155] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0156] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0157] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0158] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified in the boxes in the flowchart and / or block diagram.

[0160] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0161] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0162] In addition, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0163] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0165] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0166] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0167] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0168] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

[0169] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A multimodal controllable text generation method, characterized in that: include: Identify textual and visual control elements; Based on the text control element and the visual control element, perform at least two text generation reasonings to obtain at least two text generation intermediate results; Splicing the intermediate results generated from the at least two texts to obtain a thought chain prompt; Based on the text control elements, the thinking chain prompts and the visual control elements, the text generation reasoning is iterated until the intermediate results of text generation converge, and the converged text generation intermediate results are used as the text generation results, wherein the text generation intermediate results output by this round of iteration are spliced ​​into the thinking chain prompts input by this round of iteration as the thinking chain prompts input by the next round of iteration.

2. The method according to claim 1, characterized in that The performing at least two text generation inferences based on the text control element and the visual control element to obtain each text generation inference in at least two text generation intermediate results includes: Inferring based on the text control elements through a large language model to obtain a number of candidate texts; The candidate texts are matched with the visual control element based on a contrastive language image model, and one of the candidate texts is used as text to generate an intermediate result.

3. The method according to claim 2, characterized in that The large language model is used to infer based on the text control element to obtain several candidate texts, including: Determine a plurality of natural language instructions of the text control element, and use the plurality of natural language instructions as observation parameters of a candidate text among the plurality of candidate texts based on the large language model; Updating the probability distribution of the observed parameters to obtain a posterior probability of one of the candidate texts; Probability calculation is performed on the posterior probability to obtain a candidate text among the plurality of candidate texts.

4. The method according to claim 2, characterized in that: The matching of the plurality of candidate texts with the visual control element based on the contrastive language image model and taking one of the plurality of candidate texts as text to generate an intermediate result includes: Determining a matching value between the visual control element and each of the candidate texts; The candidate text with the largest matching value is used as the text to generate an intermediate result.

5. The method according to claim 1, characterized in that The step of splicing the intermediate results generated from the at least two texts to obtain a thought chain prompt includes: According to the generation order of the at least two text-generated intermediate results, the at least two text-generated intermediate results are spliced ​​to obtain the thought chain prompt.

6. The method according to claim 1, characterized in that Providing a graphical user interface through a terminal, wherein the content displayed by the graphical user interface includes an input interface and an output interface, wherein the input interface includes a text control control and a visual control control; The determining of the text control element and the visual control element comprises: In response to an input operation on the text control component, acquiring the input text control element; In response to an input operation on the visual control component, obtaining the input visual control element; The iterative text generation reasoning based on the text control element, the thought chain prompt and the visual control element until the text generation intermediate result converges, and the converged text generation intermediate result is used as the text generation result, including: The text generation intermediate results generated in each iteration are sequentially displayed on the output interface; The text generation result is displayed on the output interface.

7. A multimodal controllable text generation device, characterized in that: include: An element determination module configured to determine text control elements and visual control elements; An intermediate result determination module is configured to perform at least two text generation reasonings based on the text control element and the visual control element to obtain at least two text generation intermediate results; A thinking chain prompt determination module is configured to splice the intermediate results generated by the at least two texts to obtain a thinking chain prompt; The iteration module is configured to iteratively perform text generation reasoning based on the text control elements, the thinking chain prompts and the visual control elements until the intermediate results of text generation converge, and use the converged text generation intermediate results as the text generation results, wherein the text generation intermediate results output by this round of iteration are spliced ​​into the thinking chain prompts input by this round of iteration as the thinking chain prompts input by the next round of iteration.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.

Images