Method for image-text conversation, electronic device, storage medium and program product

Abstract

The application provides a picture-text conversation method, an electronic device, a storage medium and a program product, and relates to the technical field of picture-text conversation. The method comprises the following steps: obtaining first conversation information; inputting the first conversation information into a target picture-text conversation model to obtain first conversation response information corresponding to the first conversation information; wherein the first conversation response information comprises any one of picture information, text information or picture-text mixed information; wherein the target picture-text conversation model is trained based on sample conversation information and corresponding sample conversation response information; and the sample conversation response information at least comprises picture-text mixed conversation response information. In the embodiment of the application, the target picture-text conversation model has the capability of generating picture-text mixed content, can generate picture-text mixed content, and thus enriches the human-computer interaction content.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a picture-text conversation method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] Large language models represented by GPT series have brought great impact on people's daily production and life due to their powerful text generation capabilities. At the same time, image generation technologies represented by GAN, VAE and Diffusion have also developed rapidly. In human-computer conversation, one of the important means to enrich machine conversation content is that the machine generates mixed content of pictures and texts. Currently, some researches have tried to integrate multi-modal generation tasks into the same model, such as OFA and CoDi. However, these methods can only selectively generate a certain modality (image or text), and actually cannot realize the machine to automatically generate mixed content of pictures and texts.

[0003] Therefore, there is an urgent need for a new picture-text conversation method. SUMMARY

[0004] Embodiments of the present application provide a picture-text conversation method, an electronic device, a storage medium and a program product to at least partially solve the problems in the related art.

[0005] The first aspect of the embodiments of the present application provides a picture-text conversation method, comprising:

[0006] obtaining first conversation information;

[0007] inputting the first conversation information into a target picture-text conversation model to obtain first conversation response information corresponding to the first conversation information; wherein the first conversation response information comprises any one of picture information, text information or mixed picture-text information;

[0008] wherein the target picture-text conversation model is trained based on sample conversation information and corresponding sample conversation response information, and the sample conversation response information at least includes mixed picture-text conversation response information.

[0009] Optionally, the inputting the first conversation information into the target picture-text conversation model to obtain the first conversation response information corresponding to the first conversation information comprises:

[0010] inputting the first conversation information into the target picture-text conversation model, encoding the first conversation information through the target picture-text conversation model to generate a conversation response sequence; the conversation response sequence comprises a plurality of to-be-decoded elements, and a decoding result of the plurality of to-be-decoded elements constitutes the first conversation response information;

[0011] The target text-to-image dialogue model is used to decode the multiple elements to be decoded sequentially to obtain the first dialogue response information.

[0012] Optionally, the plurality of elements to be decoded includes a plurality of elements representing images and a plurality of elements representing text; the elements representing images include: an image start generation element arranged in sequence and a plurality of query elements, wherein the image start generation element is used to instruct the target text-to-image dialogue model to start generating an image, and the plurality of query elements are used to instruct the target text-to-image dialogue model to generate an image corresponding to the first dialogue information;

[0013] The target graph-text dialogue model is used to sequentially decode the plurality of elements to be decoded to obtain the first dialogue response information, including:

[0014] When the element to be decoded is an element representing text, the corresponding text is generated through the target graph-text dialogue model;

[0015] When the current element to be decoded is the image start generation element, the target image-text dialogue model extracts the multiple query elements located after the image start generation element, generates multiple corresponding prompt word elements based on the multiple query elements, and generates the image corresponding to the first dialogue information based on the multiple prompt word elements.

[0016] Among them, the plurality of prompt word elements correspond one-to-one with the plurality of prompt word elements, and the text and images generated by the target graphic-text dialogue model constitute the first dialogue response information that is a mixture of text and graphics.

[0017] Optionally, the target text-to-image dialogue model includes at least: a pre-trained large language model, an image encoding module, and an image decoding module; extracting the plurality of query elements located after the starting element of the image generation, generating the plurality of corresponding prompt word elements based on the plurality of query elements, and generating the image corresponding to the first dialogue information based on the plurality of prompt word elements, including:

[0018] The pre-trained large language model extracts the multiple query elements located after the initial element generation of the image, generates multiple corresponding prompt word elements based on the multiple query elements, and sends the multiple prompt word elements to the image decoding module;

[0019] The image decoding module decodes the multiple prompt word elements to obtain the image corresponding to the first dialogue information.

[0020] Optionally, the element to be decoded following the last query element among the plurality of query elements is decoded by the target graph-text dialogue model according to the following steps:

[0021] receiving, by the image encoding module, the image corresponding to the first dialogue information sent by the image decoding module;

[0022] sending, by the image encoding module, the image corresponding to the first dialogue information to the pre-trained large language model;

[0023] decoding, by the target image-text dialogue model, the to-be-decoded elements located after the last query element in combination with the image corresponding to the first dialogue information and the generated decoding result, the generated decoding result including the decoding results of the to-be-decoded elements located before the first query element in the plurality of query elements.

[0024] Optionally, the target image-text dialogue model further includes an image generation connector connected between the pre-trained large language model and the image decoding module, and the method further includes:

[0025] converting, by the image generation connector, the format of the prompt word element into a format supported by the image decoding module;

[0026] inputting the prompt word element into the image decoding module, including:

[0027] inputting the prompt word element in the converted format into the image decoding module.

[0028] Optionally, the target image-text dialogue model further includes an image understanding connector connected between the image encoding module and the pre-trained large language model;

[0029] sending, by the image encoding module, the image corresponding to the first dialogue information to the pre-trained large language model, including:

[0030] sending, by the image encoding module, the image corresponding to the first dialogue information to the image understanding connector;

[0031] converting, by the image understanding connector, the image corresponding to the first dialogue information into a format supported by the pre-trained large language model, and sending the image in the converted format to the pre-trained large language model.

[0032] Optionally, obtaining the first dialogue information includes:

[0033] obtaining the first dialogue text information in the form of natural language.

[0034] The second aspect of the embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the graphic-text conversation method according to the first aspect of the present application.

[0035] The third aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps in the graphic-text conversation method according to the first aspect of the present application.

[0036] The fourth aspect of the embodiment of the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the steps in the graphic-text conversation method according to the first aspect of the present application.

[0037] In the embodiment of the present application, the target graphic-text conversation model is used to process the obtained first conversation information, so that the first conversation response information corresponding to the first conversation information can be obtained, and the first conversation response information comprises any one of picture information, text information, or mixed graphic-text information. In the embodiment of the present application, the target graphic-text conversation model has the ability to generate mixed graphic-text, and can generate mixed graphic-text first conversation response information, pure text conversation response information, or pure image first conversation response information, thereby enriching the human-computer interaction content. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 is a flowchart of a graphic-text conversation method according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of another graphic-text conversation method according to an embodiment of the present application;

[0041] Figure 3 is a model structure diagram of a target graphic-text conversation model in a graphic-text conversation method according to an embodiment of the present application;

[0042] Figure 4 is a structural block diagram of a graphic-text conversation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the above objectives, characteristics and advantages of the present application more apparent, more comprehensible, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0044] In recent years, important progress has been made in the research of computer vision, deep learning, machine learning, image-text dialogue, image recognition and other technologies based on artificial intelligence. Artificial intelligence (AI) is a new science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks and many other technology categories. Computer vision, as an important branch of artificial intelligence, specifically enables machines to recognize the world. Computer vision technology generally includes face recognition, liveness detection, fingerprint recognition and anti-fraud verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image-text dialogue, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, three-dimensional reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning, and other technologies. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as security, city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile imaging, cloud services, smart home, wearable devices, driverless vehicles, autonomous driving, intelligent medical care, face payment, face unlocking, fingerprint unlocking, face and certificate verification, smart screens, smart televisions, cameras, mobile Internet, network live streaming, beauty, makeup, medical cosmetology, intelligent temperature measurement, and other fields.

[0045] With reference to Figure 1 , a flowchart of a method for image-text dialogue according to an embodiment of the present application is shown. The method for image-text dialogue provided by the present embodiment can include the following steps:

[0046] S101, obtaining first dialogue information.

[0047] In the present embodiment, the first dialogue information can be input by a user or an intelligent model such as an expert model.

[0048] Specifically, the input method of the user can include voice input, text input, etc.

[0049] Specifically, the first dialogue information input by the intelligent model can be selected and randomly generated from a template library that has been constructed. The first dialogue information input by the intelligent model can be randomly selected from various dialogue information that already exists on the network.

[0050] S102, input the first dialogue information into a target image-text dialogue model to obtain first dialogue response information corresponding to the first dialogue information.

[0051] The first dialogue response information includes any one of picture information, text information, or image-text mixed information.

[0052] In the embodiment of the application, the first dialogue information input into the target image-text dialogue model can be first dialogue text in natural language form.

[0053] Specifically, in the embodiment of the application, the basic framework of the target image-text dialogue model is a large language model, such as GTP.

[0054] In the embodiment of the application, the first dialogue information can be a question, and correspondingly, the first dialogue response information can be an answer to the question or content related to the question.

[0055] In the embodiment of the application, the target image-text dialogue model has image and text understanding, generation capabilities, and knowledge representation and reasoning capabilities.

[0056] Specifically, in the embodiment of the application, the target image-text dialogue model can understand the input text content and extract useful information as the context of the dialogue. At the same time, the target image-text dialogue model also has the ability to generate images and can generate related images according to the dialogue content. At the same time, the target image-text dialogue model also has the ability to generate natural language replies that conform to the context.

[0057] In the embodiment of the application, the target image-text dialogue model also has certain knowledge representation and reasoning capabilities. By introducing an external knowledge base or a pre-training model, the target image-text dialogue model can query and reason external knowledge, thereby providing more accurate and comprehensive answers.

[0058] In the embodiment of the application, the target image-text dialogue model can also perform context understanding and reasoning to ensure that the answers can be connected to the previous dialogue content, express clearly, and have logicality.

[0059] In the embodiment of the application, by processing the obtained first dialogue information through the target image-text dialogue model, the first dialogue response information corresponding to the first dialogue information can be obtained. The first dialogue response information includes any one of picture information, text information, or image-text mixed information. In the embodiment of the application, the target image-text dialogue model has the ability to generate image-text mixed information, and can generate image-text mixed first dialogue response information, pure text dialogue response information, or pure image first dialogue response information, thereby enriching the content of human-computer interaction.

[0060] Referring to Figure 2, a flowchart of a picture-text dialogue method according to an embodiment of the present application is shown, and specifically, the picture-text dialogue method can include the following steps:

[0061] S201, obtaining first dialogue text information in a natural language form.

[0062] In the embodiment of the present application, the first dialogue information is text information in a natural language form, so that the target picture-text dialogue model can understand and reason based on the text information in the natural language form to obtain corresponding response information.

[0063] S202, inputting the first dialogue information into the target picture-text dialogue model, encoding the first dialogue information through the target picture-text dialogue model, and generating a dialogue response sequence.

[0064] The dialogue response sequence includes a plurality of to-be-decoded elements, and the decoding results of the plurality of to-be-decoded elements constitute the first dialogue response information.

[0065] Specifically, in the embodiment of the present application, the target picture-text dialogue model can generate a dialogue response sequence through a self-recurrent manner. The specific steps can be represented as follows: first, input the question into the target picture-text dialogue model as the initial input, the target picture-text dialogue model encodes and understands the initial input to learn the corresponding semantic and grammatical information, and generates the first to-be-decoded element, then the target picture-text dialogue model can use the previously generated to-be-decoded element as input, and predict the next most possible to-be-decoded element through a probability distribution. After generating the next to-be-decoded element, it is added to the generated sequence, and then the sequence is input into the target picture-text dialogue model again. The target picture-text dialogue model predicts the next to-be-decoded element based on the updated sequence, and repeats the above steps until the stopping condition is reached (for example, the length of the generated sequence reaches a specified value or a specific end symbol is generated), and the sequence generated when the stopping condition is reached is the dialogue response sequence.

[0066] Through this self-recurrent generation manner, the target picture-text dialogue model can gradually construct a coherent and complete dialogue response sequence. Since the target picture-text dialogue model generates a new to-be-decoded element based on the previously generated to-be-decoded element in the process of generating a new to-be-decoded element, a dialogue response sequence including a plurality of to-be-decoded elements corresponding to the first dialogue information can be generated.

[0067] S203, decoding the plurality of to-be-decoded elements through the target picture-text dialogue model in sequence to obtain the first dialogue response information.

[0068] In the embodiment of the present application, the target image-text dialogue model can decode the dialogue response sequence in an autoregressive manner to obtain the first dialogue response information. The specific steps can be represented as follows: first, input the first decoding element in the dialogue response sequence into the target image-text dialogue model for decoding to obtain the decoding result of the first decoding, and then the target image-text dialogue model can combine the generated decoding result to decode the next decoding element. Among them, for the plurality of query elements included in the dialogue response sequence, which are used to generate the same image (i.e. the image corresponding to the first dialogue information), the plurality of query elements are regarded as a whole, and the plurality of query elements are extracted by the image-text dialogue model to generate the image corresponding to the first dialogue information, which is not decoded in an autoregressive manner. The other decoding elements (for example: elements representing text) except the plurality of query elements regarded as a whole are decoded in an autoregressive manner.

[0069] Through this autoregressive generation manner, the target image-text dialogue model can decode the coherent and complete first dialogue response information step by step. Since the target image-text dialogue model considers the previously generated decoding result when decoding each decoding element, meaningful first dialogue response information related to the first dialogue information can be generated.

[0070] The plurality of decoding elements include a plurality of image representing elements and a plurality of text representing elements. The image representing elements include an image start generating element and a plurality of query elements arranged in sequence, the image start generating element is used to instruct the target image-text dialogue model to start generating an image, and the plurality of query elements are used to instruct the target image-text dialogue model to generate the image corresponding to the first dialogue information.

[0071] In the embodiment of the present application, each query element corresponds to a prompt word element, and the plurality of prompt word elements corresponding to the plurality of query elements are used to generate the image corresponding to the first dialogue information.

[0072] In the embodiment of the present application, the plurality of image representing elements and the plurality of text representing elements can be sequentially taken as the current decoding element, and the following steps are executed by the target image-text dialogue model:

[0073] S2031, when the current decoding element is a text representing element, generating the corresponding text by the target image-text dialogue model.

[0074] S2032, when the current to-be-decoded element is an image start generation element, extracting the plurality of query elements located after the image start generation element through the target image-text dialogue model, generating a plurality of prompt word elements corresponding to the plurality of query elements, and generating an image corresponding to the first dialogue information according to the plurality of prompt word elements. Wherein, the plurality of query elements are extracted in parallel. Because the plurality of query elements are used to generate the same image (i.e. the image corresponding to the first dialogue information), the plurality of query elements can be regarded as a whole, the target image-text dialogue model extracts the plurality of query elements in parallel, processes the plurality of query elements in parallel, generates a plurality of prompt word elements corresponding to the plurality of query elements, and generates the image corresponding to the first dialogue information according to the plurality of prompt word elements.

[0075] Wherein, the text and the image generated by the target image-text dialogue model constitute the first dialogue response information mixed with image and text.

[0076] In the embodiment of the application, because the target image-text dialogue model has the ability to generate dialogue response information mixed with image and text, the dialogue response sequence generated by the target image-text dialogue model can contain both elements representing text and elements representing image.

[0077] In the embodiment of the application, the target image-text dialogue model can decode the plurality of to-be-decoded elements in the generated dialogue response sequence one by one while generating the dialogue response sequence in a self-recurrent manner.

[0078] In the embodiment of the application, after generating the image corresponding to the first dialogue information, an image stop generation element can be generated to make the target image-text dialogue model continue to decode the next to-be-decoded element.

[0079] In the embodiment of the application, a method for image-text dialogue is also provided. In the method, the target image-text dialogue model includes at least a pre-trained large language model, an image encoding module, and an image decoding module, as shown in Figure 3 Figure 3 The model structure diagram of the target image-text dialogue model in the method for image-text dialogue of the embodiment of the application is shown; wherein, the pre-trained large language model is obtained by training the large language model using text data.

[0080] In the embodiment of the application, the pre-trained large language model at least has the ability to understand the semantics of text.

[0081] In the embodiment of the application, the image encoding module can adopt a CLIP model.

[0082] ​The CLIP (Contrastive Language-Image Pretraining) model is a neural network model that integrates text and image information, and can realize multi-modal semantic matching and reasoning tasks. Specifically, the CLIP model can associate a natural language description with an image, thereby establishing a semantic connection. Through training, the CLIP model can understand the correspondence between images and text, so that the model can perform semantic reasoning and matching based on the given image or description.

[0083] The image decoding module can use the Stable Diffusion model, which can generate images based on text prompts as input.

[0084] Specifically, the above step S2032 includes:

[0085] S20321, when the current to-be-decoded element is an image start generation element, extracting the plurality of query elements through the pre-trained large language model, generating a plurality of prompt word elements corresponding to the plurality of prompt word elements according to the plurality of prompt word elements, and sending the plurality of prompt word elements to the image decoding module.

[0086] In the embodiments of the present application, the first dialogue information can be encoded by the pre-trained large prediction model to obtain a dialogue response sequence containing a plurality of query elements, and a plurality of prompt word elements can be obtained based on the plurality of query elements. A control signal can be obtained based on the plurality of prompt word elements, which can be used to generate an image corresponding to the first dialogue information by the image decoding module based on the plurality of prompt word elements.

[0087] S20322, inputting the plurality of prompt word elements into the image decoding module, and decoding the plurality of prompt word elements to obtain an image corresponding to the first dialogue information.

[0088] In the embodiments of the present application, the control signal output by the large language model can be decoded into an image corresponding to the first dialogue information by the Stable Diffusion model.

[0089] S20323, sending the image corresponding to the first dialogue information to the pre-trained large language model.

[0090] S20324, decoding the to-be-decoded elements located after the last query element by the target image-text dialogue model in combination with the image corresponding to the first dialogue information and the generated decoding result, the generated decoding result including the decoding results of the to-be-decoded elements located before the first query element in the plurality of query elements.

[0091] In the embodiment of the present application, the image corresponding to the first dialogue information is sent to the pre-trained large language model, and the pre-trained large language model decodes the next to-be-decoded element (i.e., the to-be-decoded element after the to-be-decoded element after the last query element) in combination with the image corresponding to the first dialogue information and the generated decoding result (including the decoding result of each to-be-decoded element located before the first query element in the plurality of query elements).

[0092] As described above, the plurality of query elements as a whole are ultimately decoded into the image corresponding to the first dialogue information. Therefore, the next to-be-decoded element is the to-be-decoded element after the last query element in the plurality of query elements.

[0093] The decoding manner of the next decoding element depends on the type of the next decoding element. When the next to-be-decoded element is a text element, the corresponding text is generated by the target image-text dialogue model; when the next to-be-decoded element is an image start generation element, the decoding manner of the current to-be-decoded element being an image start generation element can be referred to.

[0094] In the embodiment of the present application, during the training process, an end-to-end training method can be used to train the pre-trained large language model, the image encoding module, and the image decoding module.

[0095] In the embodiment of the present application, during the training process, the parameters of the target image-text dialogue model can be updated according to the first loss value corresponding to the image in the image-text mixed data and the predicted target image, and the second loss value corresponding to the text in the image-text mixed data and the prompt word element. The target image-text dialogue model obtained by training can obtain a prompt word based on the first dialogue information, and can generate an image matching the first dialogue information based on the prompt word.

[0096] In the embodiment of the present application, the target image-text dialogue model further includes an image generation connector connected between the pre-trained large language model and the image decoding module; the method further includes:

[0097] The format of the prompt word element is converted into a format supported by the image decoding module through the image generation connector.

[0098] The prompt word element is input into the image decoding module, including inputting the format-converted prompt word element into the image decoding module.

[0099] In the embodiment of the present application, an image generation connector can be connected between the large language model and the image decoding module, and the image generation connector is used to convert the format of the prompt word element output by the large language model into a format that can be processed by the image decoding module.

[0100] In the embodiment of the present application, the target image-text dialogue model further comprises an image understanding connector connected between the image encoding module and the pre-trained large language model.

[0101] In this case, the image corresponding to the first dialogue information is sent to the pre-trained large language model by the image encoding module, comprising:

[0102] The image corresponding to the first dialogue information sent by the image decoding module is received by the image encoding module;

[0103] The image corresponding to the first dialogue information is sent to the image understanding connector by the image encoding module.

[0104] The image corresponding to the first dialogue information is converted into a format supported by the pre-trained large language model by the image understanding connector, and the image after format conversion is sent to the pre-trained large language model.

[0105] In the embodiment of the present application, an image understanding connector can be connected between the image encoding module and the pre-trained large language model, and the image understanding connector is used to convert the format of the image output by the image encoding module (i.e. the image corresponding to the first dialogue information) into a format that the pre-trained large language model can process.

[0106] In the embodiment of the present application, in the training process, an end-to-end training method can be used to train the image encoding module, the image understanding connector, the pre-trained large language model, the image generation connector and the image decoding module. Thus, in the embodiment of the present application, the end-to-end training is used to make the image-text dialogue model obtained by training have stronger data and model scale-up capability, and the model capability can be continuously improved as the model increases and the data increases.

[0107] The embodiment of the present application also provides an image-text dialogue method. In the method, the target image-text dialogue model is obtained by training a pre-trained image-text dialogue model, the target image-text dialogue model is trained using image-text mixed data, and the target image-text dialogue model is used to obtain a prompt word from the image-text mixed data, and the prompt word is used to generate an image matched with the image-text mixed data.

[0108] In the embodiment of the present application, the image-text dialogue model is pre-trained based on the llama model on a large amount of image-text mixed data, and the training purpose is to obtain an image-text dialogue model that can obtain a prompt word from image-text mixed data.

[0109] In the embodiments of the present application, the image-text mixed data refers to a data pair corresponding to an image and text.

[0110] Further, in the embodiments of the present application, the target image-text dialogue model is trained according to the following steps:

[0111] S11, input the question data in the sample image-text mixed question and answer data into the target image-text dialogue model to obtain predicted image-text mixed answer data.

[0112] S12, update the parameters of the target image-text dialogue model according to the image-text mixed answer data in the sample image-text mixed question and answer data and the predicted image-text mixed answer data.

[0113] In the embodiments of the present application, the image-text dialogue model can be further trained based on sample image-text mixed question and answer data composed of sample questions and corresponding sample image-text mixed answer data.

[0114] The training data can also include sample text dialogue data and sample text-to-image dialogue data, wherein the sample text dialogue data is composed of sample text questions and sample text answers. The sample text-to-image dialogue data is composed of sample text questions and sample image answers.

[0115] Based on the same inventive concept, the embodiments of the present application provide an image-text dialogue device, which refers to Figure 4 , Figure 4 is a structural block diagram of the image-text dialogue device provided by the embodiments of the present application, and the device comprises:

[0116] The acquisition module 401 is configured to obtain first dialogue information.

[0117] The input module 402 is configured to input the first dialogue information into a target image-text dialogue model to obtain first dialogue response information corresponding to the first dialogue information; wherein the first dialogue response information includes any one of picture information, text information or image-text mixed information.

[0118] The target image-text dialogue model is trained based on sample dialogue information and corresponding sample dialogue response information, and the sample dialogue response information at least includes image-text mixed dialogue response information.

[0119] Optionally, the input module 402 specifically includes:

[0120] The encoding submodule is configured to encode the first dialogue information by using the target image-text dialogue model to generate a dialogue response sequence, wherein the dialogue response sequence comprises a plurality of to-be-decoded elements, and a decoding result of the plurality of to-be-decoded elements constitutes the first dialogue response information.

[0121] The decoding submodule is configured to decode the plurality of to-be-decoded elements in sequence by using the target image-text dialogue model to obtain the first dialogue response information.

[0122] Optionally, the plurality of to-be-decoded elements comprises a plurality of image representation elements and a plurality of text representation elements; the image representation elements comprise an image start generation element and a plurality of query elements arranged in sequence, the image start generation element is used to instruct the target image-text dialogue model to start generating an image, and the plurality of query elements are used to instruct the target image-text dialogue model to generate an image corresponding to the first dialogue information.

[0123] The decoding submodule is configured to:

[0124] when the current to-be-decoded element is a text representation element, generate corresponding text by using the target image-text dialogue model;

[0125] when the current to-be-decoded element is an image start generation element, extract the plurality of query elements located after the image start generation element by using the target image-text dialogue model, generate a plurality of prompt word elements corresponding to the plurality of query elements, and generate an image corresponding to the first dialogue information according to the plurality of prompt word elements;

[0126] wherein the plurality of prompt word elements correspond to the plurality of prompt word elements one by one, and the text and the image generated by the target image-text dialogue model constitute image-text mixed first dialogue response information.

[0127] Optionally, the target image-text dialogue model at least comprises a pre-trained large language model, an image encoding module, and an image decoding module; and the decoding submodule is specifically configured to:

[0128] extract the plurality of query elements located after the image start generation element by using the pre-trained large language model, generate a plurality of prompt word elements corresponding to the plurality of query elements, and send the plurality of prompt word elements to the image decoding module;

[0129] decode the plurality of prompt word elements by using the image decoding module to obtain an image corresponding to the first dialogue information.

[0130] Optionally, the to-be-decoded element located after the last query element in the plurality of query elements is decoded by the target image-text dialogue model according to the following steps.

[0131] receive, by the image encoding module, the image corresponding to the first dialogue information sent by the image decoding module;

[0132] send, by the image encoding module, the image corresponding to the first dialogue information to the pre-trained large language model;

[0133] decode, by the target image-text dialogue model, the to-be-decoded element located after the last query element in combination with the image corresponding to the first dialogue information and the generated decoding result, the generated decoding result including the decoding results of the to-be-decoded elements located before the first query element in the plurality of query elements.

[0134] Optionally, the target image-text dialogue model further includes an image generation connector connected between the pre-trained large language model and the image decoding module; the device further includes:

[0135] a first conversion module configured to convert the format of the prompt word element into a format supported by the image decoding module through the image generation connector;

[0136] inputting the prompt word element into the image decoding module includes:

[0137] inputting the converted prompt word element into the image decoding module.

[0138] Optionally, the target image-text dialogue model further includes an image understanding connector connected between the image encoding module and the pre-trained large language model; sending, by the image encoding module, the image corresponding to the first dialogue information to the pre-trained large language model includes:

[0139] sending, by the image encoding module, the image corresponding to the first dialogue information to the image understanding connector;

[0140] converting, by the image understanding connector, the image corresponding to the first dialogue information into a format supported by the pre-trained large language model, and sending the converted image to the pre-trained large language model.

[0141] Optionally, the target image-text dialogue model is obtained by training a pre-trained image-text dialogue model, and the target image-text dialogue model is obtained according to the following steps:

[0142] input the sample dialogue information in the sample dialogue information pair into the target text-image dialogue model to obtain predicted text-image mixed response information;

[0143] update parameters of the target text-image dialogue model according to the text-image mixed dialogue response information in the sample dialogue information pair and the predicted text-image mixed dialogue response information.

[0144] Optionally, the obtaining module is specifically configured to:

[0145] obtain the first dialogue text information in a natural language form.

[0146] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0147] Based on the same inventive concept, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the text-image dialogue method of any of the above embodiments when executing the computer program.

[0148] Based on the same inventive concept, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in the text-image dialogue method of any of the above embodiments.

[0149] Based on the same inventive concept, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to implement the steps in the text-image dialogue method of any of the above embodiments.

[0150] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to.

[0151] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0152] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It is understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device or other programmable processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable processing device, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0153] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable processing device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0154] These computer program instructions can also be loaded onto a computer or other programmable processing device to cause a series of operational steps to be performed on the computer or other programmable processing device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable processing device provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0155] Although preferred embodiments of the present application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments without departing from the basic inventive concepts disclosed. Accordingly, it is intended to embrace all such modifications and variations as fall within the scope of the present application.

[0156] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0157] The above provides a kind of graphic conversation method, electronic equipment, storage medium and program product provided by the present application, the principle and implementation of the present application are described in detail in this paper, the above example is only for helping to understand the method and its core idea of the present application;For the general technical personnel in the art, according to the idea of the present application, there will be changes in specific implementation and application range, and the above-mentioned content of the specification should not be understood as the limitation of the present application.

Claims

1. A method of conversing with a graphic, characterized by, The method comprises: obtaining first dialogue information; inputting the first dialogue information into a target image-text dialogue model to obtain first dialogue response information corresponding to the first dialogue information; wherein the first dialogue response information comprises any one of picture information, text information or image-text mixed information; a basic framework of the target image-text dialogue model is a large language model; wherein the target image-text dialogue model is trained based on sample dialogue information and corresponding sample dialogue response information, and the sample dialogue response information at least comprises image-text mixed dialogue response information; the inputting the first dialogue information into the target image-text dialogue model to obtain the first dialogue response information corresponding to the first dialogue information comprises: inputting the first dialogue information into the target image-text dialogue model, encoding the first dialogue information through the target image-text dialogue model to generate a dialogue response sequence; the dialogue response sequence comprises a plurality of to-be-decoded elements, and decoding results of the plurality of to-be-decoded elements constitute the first dialogue response information; decoding the plurality of to-be-decoded elements through the target image-text dialogue model to obtain the first dialogue response information; in the process of generating a new to-be-decoded element by the target image-text dialogue model, a new to-be-decoded element is generated according to a previously generated to-be-decoded element; the plurality of to-be-decoded elements comprise a plurality of image representation elements and a plurality of text representation elements; the image representation elements comprise an image start generation element and a plurality of query elements arranged in sequence; the image start generation element is used to instruct the target image-text dialogue model to start generating an image, and the plurality of query elements are used to instruct the target image-text dialogue model to generate an image corresponding to the first dialogue information; decoding the plurality of to-be-decoded elements through the target image-text dialogue model to obtain the first dialogue response information comprises: generating corresponding text through the target image-text dialogue model when the current to-be-decoded element is a text representation element; extracting the plurality of query elements located after the image start generation element through the target image-text dialogue model when the current to-be-decoded element is the image start generation element, generating a plurality of prompt word elements corresponding to the plurality of query elements, and generating an image corresponding to the first dialogue information according to the plurality of prompt word elements; wherein the plurality of prompt word elements correspond to the plurality of prompt word elements one by one, and the text and the image generated by the target image-text dialogue model constitute image-text mixed first dialogue response information.

2. The method of claim 1, wherein, the target image-text dialogue model at least comprises a pre-trained large language model, an image encoding module and an image decoding module; extracting the plurality of query elements located after the image start generation element, generating a plurality of prompt word elements corresponding to the plurality of query elements, and generating an image corresponding to the first dialogue information according to the plurality of prompt word elements comprises: extracting, by the pre-trained large language model, the plurality of query elements located after the image starting generation element, generating a plurality of prompt word elements according to the plurality of query elements, and sending the plurality of prompt word elements to the image decoding module; decoding, by the image decoding module, the plurality of prompt word elements to obtain the image corresponding to the first dialogue information.

3. The method of claim 2, wherein, The to-be-decoded element located after the last query element in the plurality of query elements is decoded by the target image-text dialogue model according to the following steps: receiving, by the image encoding module, the image corresponding to the first dialogue information sent by the image decoding module; sending, by the image encoding module, the image corresponding to the first dialogue information to the pre-trained large language model; decoding, by the target image-text dialogue model, the to-be-decoded element located after the last query element in combination with the image corresponding to the first dialogue information and the generated decoding result, the generated decoding result including the decoding results of the to-be-decoded elements located before the first query element in the plurality of query elements.

4. The method of claim 2, wherein, The target image-text dialogue model further includes an image generation connector connected between the pre-trained large language model and the image decoding module; the method further includes: converting, by the image generation connector, the format of the prompt word elements into a format supported by the image decoding module; inputting the prompt word elements into the image decoding module, including: inputting the format-converted prompt word elements into the image decoding module.

5. The method of claim 2, wherein, The target image-text dialogue model further includes an image understanding connector connected between the image encoding module and the pre-trained large language model; sending, by the image encoding module, the image corresponding to the first dialogue information to the pre-trained large language model, including: sending, by the image encoding module, the image corresponding to the first dialogue information to the image understanding connector; converting, by the image understanding connector, the image corresponding to the first dialogue information into a format supported by the pre-trained large language model, and sending the format-converted image features to the pre-trained large language model.

6. The method according to any one of claims 1 to 5, characterized in that, The target image-text dialogue model is obtained by training a pre-trained image-text dialogue model, and the target image-text dialogue model is trained according to the following steps: inputting the sample dialogue information in the sample dialogue information pair into the target image-text dialogue model to obtain a predicted image-text mixed response information; updating the parameters of the target image-text dialogue model according to the image-text mixed dialogue response information in the sample dialogue information pair and the predicted image-text mixed response information.

7. The method according to any of claims 1 to 5, characterized in that, obtaining the first dialogue information, including: obtaining the first dialogue text information in natural language form.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the image-text dialogue method of any one of claims 1-7.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the image-text dialogue method of any one of claims 1-7.

10. A computer program product comprising a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method of the graphical conversation as claimed in any of claims 1 to 7.

Images