Multi-modal model generation method, multi-modal processing method, and device

Abstract

A multi-modal model generation method, a multi-modal processing method, and a device. The multi-modal model generation method comprises: acquiring modal information; determining a description text corresponding to the modal information, and by means of a pre-trained basic model, generating input information and output information on the basis of the description text to obtain an autonomously-generated training sample; constructing a multi-modal model on the basis of the basic model, and finely tuning the multi-modal model on the basis of the autonomously-generated training sample; and by means of the finely-tuned multi-modal model, generating input information and output information on the basis of the modal information, and continuing to finely tune the multi-modal model. In this way, the model can have the ability to process multi-modal information, thereby reducing the time spent for manual annotation, and improving the overall training efficiency, and rich detail features in the modal information can be mined, thereby improving the accuracy of the model.

Description

Multimodal model generation method, multimodal processing method and device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 8, 2023, with application number 202310995052.6 and application name “Multimodal model generation method, multimodal processing method and device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a method for generating a multimodal model, a multimodal processing method and a device. Background Art

[0003] With the widespread application of large models in the field of artificial intelligence, they have demonstrated strong processing capabilities in language understanding, generation, interaction, and reasoning, and a variety of application scenarios have emerged.

[0004] For example, a conversational model is a large model with conversational capabilities that generates corresponding conversational context based on the user's previous conversation, meeting the needs of intelligent conversation in different scenarios. Some technologies can use a pre-trained language model combined with fine-tuning for downstream tasks to create a usable conversational model.

[0005] However, pre-trained language models are usually only capable of processing textual information and are unable to directly process information in other modalities, such as images, audio, and video. This results in the model's input being single and unable to meet user needs. Some technologies have also attempted to use multimodal information when training models, but the overall efficiency and accuracy of training still need to be improved.

[0006] Summary of the Invention

[0007] The main purpose of the embodiments of the present application is to provide a method for generating a multimodal model, a multimodal processing method and a device to improve the training efficiency and accuracy of the multimodal model.

[0008] In a first aspect, an embodiment of the present application provides a method for generating a multimodal model, comprising:

[0009] Acquire modal information, where the modal information includes content of any one or more modalities other than text;

[0010] Determine a description text corresponding to the modal information, and generate input information and output information based on the description text using a pre-trained basic model to obtain a self-generated training sample, wherein the self-generated training sample includes: the modal information, the input information, and the output information;

[0011] Constructing a multimodal model based on the basic model, and fine-tuning the multimodal model based on the self-generated training samples;

[0012] The multimodal model is fine-tuned to generate input information and output information based on the modal information, and the multimodal model is further fine-tuned; wherein the fine-tuned multimodal model is used to determine the output information based on the modal information and the input information.

[0013] In a second aspect, an embodiment of the present application provides a method for generating a multimodal dialogue model, comprising:

[0014] Acquire modal information, where the modal information includes content of any one or more modalities other than text;

[0015] Determining a description text corresponding to the modal information, and generating a preceding and following conversation context based on the description text using a pre-trained language model to obtain a self-generated training sample, wherein the self-generated training sample includes: the modal information, the preceding and following conversation context;

[0016] Constructing a multimodal dialogue model based on the language model, and fine-tuning the multimodal dialogue model based on the self-generated training samples;

[0017] The fine-tuned multimodal dialogue model is used to generate a preceding dialogue context and a following dialogue context based on the modal information, and the multimodal dialogue model is further fine-tuned; wherein the fine-tuned multimodal dialogue model is used to determine the following dialogue context based on the modal information and the preceding dialogue context.

[0018] In a third aspect, an embodiment of the present application provides a multimodal processing method, including:

[0019] Acquire information to be processed, where the information to be processed includes modal information and input information, where the modal information includes content in any one or more modalities other than text;

[0020] Determining output information corresponding to the information to be processed based on the multimodal model;

[0021] Wherein, the multimodal model is generated by any of the methods described above.

[0022] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0023] at least one processor; and

[0024] a memory communicatively coupled to the at least one processor;

[0025] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0026] The multimodal model generation method, multimodal processing method, and device provided in the present application can obtain modal information, where the modal information includes content of any one or more modalities other than text, determine the descriptive text corresponding to the modal information, and generate input information and output information based on the descriptive text through a pre-trained basic model to obtain self-generated training samples. The self-generated training samples include: the modal information, the input information, and the output information. A multimodal model is constructed based on the basic model, and the multimodal model is fine-tuned based on the self-generated training samples. The fine-tuned multimodal model generates input information and output information based on the modal information, and the multimodal model is further fine-tuned, thereby enabling the model to process multimodal information. The model's multimodal understanding ability can be used for iterative fine-tuning, and only a small amount of preliminary annotation is required to complete model training, reducing the time spent on manual annotation and improving overall training efficiency. In addition, by first generating training samples using descriptive text and then generating training samples using the modal information itself, it is possible to mine the rich detailed features in the modal information and improve the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0028] FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0029] FIG2 is a flow chart of a method for generating a multimodal dialogue model provided in an embodiment of the present application;

[0030] FIG3 is a schematic diagram of an interactive data annotation method provided in an embodiment of the present application;

[0031] FIG4 is a schematic diagram showing the principles of a multimodal dialogue model provided in an embodiment of the present application;

[0032] FIG5 is a flow chart of an optimization model provided in an embodiment of the present application;

[0033] FIG6 is a schematic diagram of the principle of contrastive learning provided by an embodiment of the present application;

[0034] FIG7 is a schematic diagram illustrating the principles of a method for generating a multimodal dialogue model provided in an embodiment of the present application;

[0035] FIG8 is a flow chart of another method for generating a multimodal dialogue model provided in an embodiment of the present application;

[0036] FIG9 is a flow chart of a method for generating a multimodal model provided in an embodiment of the present application;

[0037] FIG10 is a flow chart of a multimodal dialogue method according to an embodiment of the present application;

[0038] FIG11 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0039] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0040] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0041] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0042] First, let’s explain the terms involved in this application:

[0043] Large Language Model (LLM): also known as a large-scale language model, currently generally refers to a deep learning model with billions to trillions or even more parameters, which usually requires the use of large-scale data and computing resources for training.

[0044] Self-instructing: refers to using self-instruction to generate data to fine-tune the model, that is, allowing the model to fine-tune by learning some features of the data or performing certain tasks.

[0045] Multimodal: refers to data that contains multiple modalities (such as text, images, voice, video, etc.).

[0046] Instructional fine-tuning: refers to fine-tuning the model using manually provided instructions or tasks.

[0047] Supervised Fine-tuning (SFT): Fine-tuning the model using supervised data.

[0048] Large models refer to deep learning models with massive model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large models, also known as foundation models, are pre-trained on large, unlabeled corpora to produce models with over 100 million parameters. These models are adaptable to a wide range of downstream tasks and exhibit good generalization capabilities, such as large language models (LLMs) and multi-modal pre-training models.

[0049] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0050] In the field of natural language processing, large language models can be applied to scenarios such as conversation, translation, and code generation, and have certain language understanding and reasoning capabilities. Currently, models generally only support text input and cannot directly use information from other modalities, such as images, video, and audio. Compared to pure text data, bimodal, trimodal, or even multimodal interactive data can effectively improve the processing capabilities of large models and meet the needs of different scenarios. To achieve the model's multimodal processing capabilities, the model can be trained using multimodal data. However, manual annotation of data from other modalities is required before training, which is inefficient and costly.

[0051] In some technologies, model training can be achieved by generating data. However, due to the lack of details and richness of the generated data, the model cannot accurately extract more fine-grained content, so that the model training still has certain deficiencies in terms of alignment and data accuracy.

[0052] In view of this, an embodiment of the present application provides a multimodal instruction fine-tuning method for a model, which can fully utilize the text capabilities of an existing, pre-trained plain text language model, combine basic data, self-instructions and manual feedback, etc., to iteratively fine-tune the model so that the model has multimodal processing capabilities.

[0053] Taking a multimodal dialogue model as an example, the method of the embodiment of the present application can be used to enable the model to have the ability to obtain the following dialogue context based on the previous dialogue context and modal information.

[0054] The modal information may include content of any one or more modalities other than text. Optionally, the modalities other than text may be images, audio, video, etc. The modal information may include content of one or more modalities, for example, a single image, a single audio, a single video, etc., or a combination of any two or more of images, audio, and video. For example, one piece of modal information may include a combination of images, audio, and video.

[0055] The context and subcontext can be textual content belonging to different characters during a conversation. The context can be user input, while the subcontext can be responses from other characters (customer service representatives, chatbots, etc.). The context and subcontext are typically presented as questions and answers, but are not limited to these. For example, the context can be a statement, and the subcontext can be a response to that statement.

[0056] Specifically, modal information such as the target image can be obtained, and the descriptive text corresponding to the target image can be determined through manual annotation or other means. Since the pre-trained language model (such as LLM) already has text understanding capabilities, the descriptive text can be directly processed to generate the context of the conversation and the context of the conversation. The context of the conversation and the context of the conversation are used together with the target image as self-generated training samples to fine-tune the model so that the fine-tuned model has the ability to process images. The target image is then directly processed according to the fine-tuned model to regenerate the context of the conversation and the context of the conversation. The model is further fine-tuned so that the model has the ability to handle multi-modality. In the fine-tuning process, the existing capabilities of the model are used for iterative processing to reduce the time spent on manual annotation and improve the overall training efficiency. In addition, by first generating samples using the descriptive text and then generating samples using the target image itself, the rich detailed features in the image can be mined. The generated training samples cover the rich content in the image, so that the model can extract the image content more accurately and meticulously, thereby improving the accuracy of the model.

[0057] In addition to self-generated training samples, models can also be fine-tuned by combining basic training samples with data-augmented training samples. Basic training samples can include the context and background of conversations manually annotated from target images. Data augmentation primarily involves shuffling images and conversations from multiple existing training samples. Shuffling can involve sorting images and conversations separately, but in different ordering schemes, to simulate real-world scenarios where users ask questions out of order.

[0058] After integrating the above-mentioned various forms of training samples, the model can be adapted to various situations through fine-tuning, thereby improving the overall training effect of the model.

[0059] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of the present application. As shown in Figure 1, multiple modal information, such as images, audio, and video, can be collected from the Internet or through other means. Basic data, including description text, conversation context, and conversation context, can be manually annotated. The modal information, conversation context, and conversation context are used as basic training samples. The basic training samples are spliced ​​together in a random order to obtain spliced ​​training samples. In addition, based on the description text, the conversation context and conversation context are generated based on a pre-trained language model and combined with the modal information to obtain self-generated training samples.

[0060] Based on the training samples obtained above, the model is fine-tuned to enable it to process modal information. In the next stage, the fine-tuned model can directly process modal information to generate conversation context and continuation, thereby obtaining new multimodal training samples. After manually screening and rewriting the multimodal training samples, the model is further fine-tuned. The fine-tuned model has better performance and can be used to generate new multimodal training samples and continue fine-tuning until the model meets the requirements. The fine-tuned model can be used to derive the corresponding continuation based on modal information and conversation context, thus being applied to various human-computer interaction scenarios such as intelligent customer service.

[0061] The following detailed description of some embodiments of the present application is provided in conjunction with the accompanying drawings. The following embodiments and features thereof may be combined with one another unless they conflict with each other. Furthermore, the sequence of steps in the following method embodiments is provided for illustrative purposes only and is not intended to be a strict limitation.

[0062] FIG2 is a flow chart of a method for generating a multimodal dialogue model provided in an embodiment of the present application. The execution subject of the method in this embodiment can be applied to any device with data processing capabilities, such as a terminal device or a server. In practical applications, the method in this embodiment can be implemented on the cloud, deployed locally, implemented on a client, implemented on an IOT (Internet of Things) device, etc. As shown in FIG2 , the method may include:

[0063] Step 201: Acquire modal information, where the modal information includes any one or more modal contents except text.

[0064] Optionally, modalities other than text may include images, audio, and video. The modal information may include content of one or more modalities, for example, a single image, a single audio, a single video, or a combination of any two or more of images, audio, and video. For example, one piece of modal information may include a combination of images, audio, and video.

[0065] The modal information can be obtained from the Internet or from a related server or database. For example, when it is necessary to train an intelligent customer service dialogue model for an e-commerce platform, the corresponding modal information can be obtained from the e-commerce platform.

[0066] Step 202: Determine the descriptive text corresponding to the modal information, and generate a conversation context and a conversation context based on the descriptive text through a pre-trained language model to obtain a self-generated training sample. The self-generated training sample includes: the modal information, the conversation context, and the conversation context.

[0067] The preceding conversation context and the following conversation context may serve as input information and output information of the language model, respectively. The language model may serve as a basic model for further constructing a multimodal model, such as a multimodal conversation model.

[0068] Optionally, the descriptive text corresponding to the modal information may be text that describes the modal information in a fine-grained manner. The descriptive text may include one or more items of content contained in the modal information, as well as the attributes of each item of content. For example, for an image, the fine-grained descriptive text may include all objects appearing in the image, as well as the color, action, position, and relationships of each object.

[0069] In this step, a pre-trained language model can be used to obtain self-generated training samples based on the description text. The training samples obtained here do not need to be manually labeled, but are generated by the model itself, so they are recorded as self-generated training samples.

[0070] Among them, the pre-trained language model can be LLM, which has certain language processing capabilities and can generate corresponding questions based on the description text and can also give answers based on the questions.

[0071] Alternatively, the description text can be directly input into the language model, which generates one or more questions as the conversation context. The questions and description text are then input into the language model to generate answers as the conversation context. In this embodiment of the present application, the conversation context and conversation context are not limited to questions and answers. For example, the conversation context can also be a statement, and the conversation context can be a response to the statement.

[0072] For example, in a customer service scenario, the preceding conversation context is the user's input, and the following conversation context is the customer service response. Conversations can be single-turn or multi-turn. In multi-turn scenarios, the preceding conversation context can include previous rounds of conversation as well as questions from the current round, allowing for more accurate responses based on contextual information.

[0073] In addition to directly generating the preceding and following conversations based on the description text, prompt words may also be used when generating the preceding and following conversations.

[0074] Alternatively, a self-instructing approach can be used to generate self-generated training samples. Self-instructing can utilize contextual prompts to automatically generate new instructions. In this embodiment, the new instructions can refer to the generated conversation context or the generated conversation context.

[0075] Optionally, generating the preceding and following conversations based on the description text through a pre-trained language model in this step may include: obtaining prompt words, inputting the description text and the prompt words into the pre-trained language model to obtain the preceding conversation; and inputting the description text and the obtained preceding conversation into the language model to obtain the following conversation.

[0076] Among them, the prompt words can be content used to constrain the generated conversation context, for example, they can include the number, style, question direction, etc. of the conversation context required to be generated by the model.

[0077] For example, the modal information is an image, and the description text obtained for the image is: "The picture shows a man and a dog playing Frisbee on a lawn. The man is lying down, and the dog jumps up and stays in the air, trying to catch the Frisbee with its mouth. The man is wearing a red crew-neck sweatshirt, blue shorts, and red sneakers. The dog is a black and white Border Collie. The Frisbee is orange in the upper left corner of the picture. The background is green grass and woods."

[0078] The prompt is, "Please help me generate 10 questions and answers based on the content of this image. The questions should be diverse and natural, and the answers should be detailed and accurate. Do not ask questions or answer information that is not described in the image above."

[0079] The pre-trained language model can generate questions that meet the requirements based on the aforementioned description text and prompt words.

[0080] Furthermore, in order to generate diverse training samples, a variety of questions can be generated by adjusting the prompt words, such as adding or replacing question styles (for example, brief, humorous, rigorous, etc.) and / or question directions (color, relationship between things, time, creation, etc.). Optionally, the prompt words can be adjusted manually or automatically. There are many ways to automatically adjust. For example, you can set a prompt word template and generate new prompt words by replacing the question style or question direction in the template, or automatically generate prompt words through a language model.

[0081] Optionally, when generating the following dialogue context, it can also be combined with prompt words. For example, the prompt word can be "The generated answer should be concise and accurate, and the word count should be controlled within 100 words", so that the model can generate the following dialogue context that meets the requirements based on the description text, the previous dialogue context and the prompt word.

[0082] This approach leverages the text understanding capabilities of pre-trained language models to automatically generate new conversation contexts and subcontexts. These contexts and subcontexts are then combined with modal information to generate rich, multimodal, self-generated training samples. This allows us to generate a large amount of multimodal conversation data from a small amount of baseline data, which can be used for subsequent fine-tuning, enabling the model to handle multimodal input.

[0083] In addition to the aforementioned self-generated training samples, other types of training samples can also be combined to fine-tune the model to further improve the model's performance.

[0084] Optionally, the method further includes: determining the preceding and following conversations by manual annotation based on the modal information to obtain basic training samples, wherein the obtained basic training samples and self-generated training samples are both multiple; splicing at least two original training samples to obtain spliced ​​training samples, wherein the original training samples are training samples arbitrarily selected from multiple basic training samples and multiple self-generated training samples.

[0085] The following first describes the basic training samples.

[0086] Optionally, any basic training sample may include: modal information, preceding conversation context, and following conversation context. For example, the modal information may be an image, the preceding conversation context may be a question about the image, such as "What color shoes is the person in the picture wearing?", and the following conversation context may be the corresponding answer, such as "Red shoes."

[0087] The preceding and following conversations can both be annotated manually. Alternatively, the preceding and following conversations can be annotated by different people, simulating a multimodal conversation in a real-world scenario. Alternatively, the descriptive text of the modal information can also be annotated by different people.

[0088] FIG3 is an interactive diagram of data annotation provided by an embodiment of the present application. As shown in FIG3, the server can obtain a small amount of modal information through the network and cooperate with the terminal to perform fine-grained annotation. Fine-grained annotation can include two parts: dialogue annotation and fine-grained annotation. Dialogue annotation mainly marks the context of the dialogue based on the modal information, such as questions and answers. Fine-grained annotation is to describe in detail all the content and attributes that appear in the modal information in language to obtain a description text. Questions, answers, and description texts can all be annotated by different people.

[0089] Optionally, determining the preceding and following conversations by manual annotation based on the modal information to obtain basic training samples may include:

[0090] A first display instruction is sent to a first terminal, where the first display instruction is used to display the modal information to a first annotator; the conversation context annotated by the first annotator based on the modal information is obtained and returned by the first terminal; a second display instruction is sent to a second terminal, where the second display instruction is used to display the modal information and the annotated conversation context to a second annotator; and the conversation context annotated by the second annotator based on the modal information and the conversation context is obtained and returned by the second terminal.

[0091] Correspondingly, determining the descriptive text corresponding to the modal information may include: sending a third display instruction to a third terminal, wherein the third display instruction is used to display the modal information to a third annotation personnel; and obtaining the descriptive text annotated by the third annotation personnel based on the modal information and returned by the third terminal.

[0092] Specifically, the first terminal, the second terminal, and the third terminal can be different terminals, and the first annotator, the second annotator, and the third annotator can be different annotators. The first terminal, the second terminal, and the third terminal can display a page for annotation to the corresponding annotator, and the page can display modal information and a window for inputting annotation content. After obtaining the annotated content of the annotator, the annotated content can be sent to the server, which will aggregate the content to obtain the corresponding training sample.

[0093] By sending modal information to different terminals and having different annotators annotate the conversation context, conversation context, and description text, we can effectively simulate multimodal conversations in real scenarios, improve the accuracy of training samples, make the fine-tuned model more in line with actual needs, and improve the model's conversation effect.

[0094] In another optional implementation, a basic training sample of the description class can also be constructed directly based on the modality information and the corresponding description information. In the basic training sample of the description class, the upper part of the dialogue is an instruction to describe the modality information, and the lower part of the dialogue is the description information.

[0095] Exemplarily, basic training samples of the description class may include: modal information, such as an image; instructions for describing the image, such as "Please help me describe this picture in detail", as the context of the conversation; descriptive text, such as "The picture shows a man and a dog playing Frisbee on the lawn,...", as the context of the conversation.

[0096] In this way, there is no need to manually label the context of the conversation and the context of the conversation. Only some fixed instructions for instructing the model to describe the image are directly used as the context of the conversation, and the aforementioned description text is directly used as the context of the conversation. Since the description text is manually annotated text, it has higher accuracy. The training samples generated directly from the description text can also be used as a type of basic training samples to fine-tune the model and improve the diversity of the samples and the overall accuracy.

[0097] The following describes the spliced ​​training samples.

[0098] Optionally, there may be multiple basic training samples and self-generated training samples. In practical applications, multiple modal information can be collected to construct multiple training samples. The same modal information can generate one basic training sample / self-generated training sample, or multiple basic training samples / self-generated training samples. For example, different questions can be asked about the same image, resulting in different answers.

[0099] After obtaining multiple training samples, different training samples can be spliced ​​together. For ease of description, in the embodiment of the present application, the training samples to be spliced ​​together are recorded as original training samples. The original training samples can be basic training samples or self-generated training samples.

[0100] Optionally, multiple basic training samples and multiple self-generated training samples can be put into a sample pool, at least two original training samples can be randomly sampled from the sample pool, and the conversation context can be randomly determined, and indication information indicating the modal information can be inserted before the conversation context for splicing operation.

[0101] Alternatively, the sample pool may contain only one type of training samples, for example, only basic training samples, and the self-generated training samples do not participate in the splicing operation.

[0102] After obtaining at least two original training samples, splicing the at least two original training samples to obtain a spliced ​​training sample may include:

[0103] Extracting modal information of each original training sample from the at least two original training samples to obtain at least two modal information, and extracting a conversation context from a first original training sample; wherein the first original training sample is any one of the at least two original training samples;

[0104] Generate indication information, the indication information being used to indicate a sequence number of the modal information corresponding to the extracted conversation context in the at least two modal information;

[0105] Generate a spliced ​​training sample based on the extracted at least two modal information, the previous conversation context, and the instruction information.

[0106] In which, the at least two original training samples may contain content of multiple modalities, for example, both images and audio, or the at least two original training samples may contain content of the same modality, for example, both images but not video and audio. This embodiment does not impose any restrictions on this.

[0107] In at least two original training samples, there are at least two modal information and at least two conversation contexts. The at least two modal information can be randomly sorted as the modal information in the spliced ​​training sample, and one conversation context can be selected as the question in the spliced ​​training sample.

[0108] In order to adapt to questions about specific modal information, indication information can be added to the spliced ​​training samples. The indication information can be used to indicate which modal information the previous conversation is targeting, and can be specifically referred to in the form of the number from top to bottom.

[0109] In an optional implementation, generating a spliced ​​training sample based on at least two extracted modal information, conversation context, and instruction information may include: using the extracted at least two modal information as the modal information in the spliced ​​training sample; using the instruction information and the extracted conversation context as the conversation context in the spliced ​​training sample; and using the conversation context corresponding to the extracted conversation context as the conversation context in the spliced ​​training sample.

[0110] Exemplarily, a splicing operation is performed on three original training samples, wherein original training sample 1 includes: image 1, question 1, answer 1; original training sample 2 includes: image 2, question 2, answer 2; original training sample 3 includes: image 3, question 3, answer 3.

[0111] Three images are extracted from three original training samples, and the three images can be randomly ordered. At the same time, one question can be extracted from the three original training samples. The question can be a question in any original training sample, and the indication information is used to indicate which image the question is for.

[0112] For example, the spliced ​​training sample may include: Image 1, Image 2, Image 3, instruction information, Question 3 (e.g., "What age is this product suitable for?"), and Answer 3 (e.g., "This product is suitable for preschoolers"). The instruction information is specifically "the third image." In the spliced ​​training sample, the modal information includes Image 1, Image 2, and Image 3. The conversation context is a combination of the instruction information and the conversation context (Question 3) extracted from the original training sample, i.e., "What age is this product suitable for?" The conversation context is the conversation context corresponding to the extracted conversation context, i.e., Answer 3.

[0113] In another optional implementation, generating a spliced ​​training sample based on the extracted at least two modal information, the conversation context, and the instruction information may include:

[0114] Extracting conversation content from the remaining original training samples; wherein the remaining original training samples are any one or more original training samples other than the first original training sample among the at least two original training samples, and the conversation content includes a preceding conversation and a following conversation;

[0115] splicing the extracted at least two modal information, the conversation content in the remaining original training samples, the conversation context in the first original training sample, and the instruction information to obtain a spliced ​​training sample;

[0116] In the spliced ​​training samples, the conversation contexts from different original training samples are randomly ordered, and / or the modal information is randomly ordered.

[0117] Specifically, at least two modal information extracted can be used as modal information in the spliced ​​training sample; at least one set of dialogue content, instruction information and dialogue context extracted from the first original training sample can be used as dialogue context in the spliced ​​training sample; and the dialogue context in the first original training sample can be used as dialogue context in the spliced ​​training sample.

[0118] Optionally, if conversation contexts are extracted from at least two original training samples, the conversation contexts from the different original training samples can be randomly ordered in the concatenated training sample, or the modal information from the different original training samples can be randomly ordered. In this way, the sequence numbers of the conversation contexts and modal information of the same original training sample in the at least two conversation contexts can be inconsistent.

[0119] Exemplarily, three images are extracted from three original training samples, and the three images can be randomly sorted. At the same time, the last original training sample can be selected as the first original training sample, and the questions and answers extracted from the remaining two original training samples are spliced ​​with the questions and indication information in the first original training sample, wherein the indication information is used to indicate the serial number of the image corresponding to the question in the first original training sample.

[0120] In the spliced ​​training samples, both images and dialogues can be randomly ordered. For example, in the spliced ​​training samples, the order of the three images is: image 1, image 2, image 3, and the order of the questions is: question 2, question 3, question 1.

[0121] Accordingly, the spliced ​​training sample may include, in order: Image 1, Image 2, Image 3, Question 2, Answer 2, Question 3, Answer 3, Instruction information, Question 1 (e.g., "How is this product"), Answer 1 (e.g., "This product is very good"). The instruction information is specifically "Image 1", and the instruction information and Question 1 are spliced ​​together to form "Image 1, how is this product?" Images 1, 2, and 3 serve as the modal information of the spliced ​​training sample, Question 2, Answer 2, Question 3, Answer 3, Instruction information, and Question 1 serve as the preceding context of the spliced ​​training sample, and Answer 1 serves as the following context.

[0122] In actual applications, users may ask questions out of order. For example, after sending three images, they may directly ask a question about any one of them, or they may first ask questions about the second and third images and then the first image. By using the splicing method in this embodiment, adding instruction information to generate spliced ​​training samples, we can simulate the situation of users asking questions out of order in real scenarios, thereby improving the conversational effectiveness and adaptability of the model.

[0123] Moreover, when splicing, if there are multiple groups of dialogues, the ordering of the dialogues and the ordering of the images can be different, so that the images and dialogues can be randomly sorted and combined, further simulating the situation where users ask questions out of order in complex scenarios, thereby improving the dialogue effect of the model.

[0124] The above describes the generation of training samples using images as an example. When the modal information is audio or video, the method of generating training samples is similar to that of images. For example, questions and answers can be asked and answered for the video and audio to obtain the corresponding conversation context and conversation context. The video and audio can also be described in a fine-grained manner to obtain descriptive text of the video and audio, and the conversation context and conversation context can be self-generated based on the descriptive text. In addition, splicing operations can also be performed.

[0125] Step 203: construct a multimodal dialogue model based on the language model, and fine-tune the multimodal dialogue model based on the self-generated training samples.

[0126] Optionally, there are multiple ways to construct a multimodal dialogue model based on the language model.

[0127] For example, different channels can be set up in a multimodal dialogue model, where one channel is used to receive modal information, and the other channel is initialized based on a pre-trained language model and is used to receive the conversation context. The two channels process the modal information and the conversation context respectively, and then fuse them in the final output layer.

[0128] For example, a multimodal encoder can be added before the language model to construct a multimodal conversation model. The output of the multimodal encoder serves as part of the input to the language model. Alternatively, separate encoders can be used for images, audio, and video.

[0129] Figure 4 is a schematic diagram of the principles of a multimodal dialogue model provided by an embodiment of the present application. As shown in Figure 4, the multimodal dialogue model constructed by the pre-trained language model may include:

[0130] A multimodal encoder is used to obtain modal information and encode the modal information to obtain element information (token) corresponding to the modal information;

[0131] The pre-trained language model is used to obtain element information corresponding to the modal information and element information corresponding to the previous conversation, and output a predicted next conversation.

[0132] Specifically, since the conversation context is in textual form, the corresponding token can be directly obtained and fed into the language model for processing. However, modal information, such as images, cannot be directly fed into the language model. Therefore, a multimodal encoder can be added to encode the image, obtaining the corresponding encoded information, which is equivalent to the corresponding token. This information is then concatenated with the token corresponding to the conversation context and fed into the language model for processing. Separators can also be inserted during concatenation to distinguish between the two. This embodiment does not limit the specific form of the input.

[0133] Optionally, fine-tuning the multimodal dialogue model according to the self-generated training samples may include: fine-tuning the multimodal dialogue model according to the basic training samples, the spliced ​​training samples, and the self-generated training samples.

[0134] Through the above steps, basic training samples, spliced ​​training samples, and self-generated training samples can be constructed, and these training samples can be used to fine-tune the model.

[0135] Optionally, the model fine-tuning process may include: inputting the modal information and conversation context in the training sample into the model to obtain the predicted conversation context, calculating the loss value based on the predicted conversation context and the conversation context in the training sample, and adjusting the model parameters based on the loss value. The goal of fine-tuning is to make the model-predicted conversation context as close as possible to the actual conversation context in the training sample.

[0136] In the embodiments of the present application, the basic training samples are manually annotated and highly accurate training samples, which can be used to fine-tune the model in a supervised manner to improve the accuracy of the model; the spliced ​​training samples can expand the training samples and can cope with various complex and disordered dialogue scenarios; the self-generated training samples are generated by using the language capabilities of the pre-trained language model and combining them with fine-grained descriptions to generate more training samples, which can achieve automatic expansion of the training samples and obtain richer training samples based on a small amount of annotations. The comprehensive use of various types of training samples can effectively improve the fine-tuning effect of the model, and effectively improve the fine-tuning efficiency and applicability of the model while ensuring a certain degree of accuracy.

[0137] Optionally, in actual applications, to improve fine-tuning efficiency, one or two types of training samples may be selected for fine-tuning. For example, the model may be fine-tuned based solely on the basic training samples and the self-generated training samples. Of course, in addition to the aforementioned training samples, other training samples may be further added to fine-tune the model, and this embodiment does not impose any limitation thereto.

[0138] Step 204: Generate a conversation context and a conversation context based on the modal information using the fine-tuned multimodal dialogue model, and continue to fine-tune the multimodal dialogue model; wherein the fine-tuned multimodal dialogue model is used to determine the conversation context based on the modal information and the conversation context.

[0139] Optionally, generating a conversation context and a conversation follow-up based on modal information using a fine-tuned multimodal dialogue model may include: inputting the modal information and prompt words into the fine-tuned multimodal dialogue model to obtain a conversation context; and inputting the modal information and the obtained conversation context into the fine-tuned multimodal dialogue model to obtain a conversation follow-up. The prompt words used here can be the same as in the previous step, except that the descriptive text input into the model is replaced with the modal information.

[0140] Specifically, in step 202, a dialogue is generated based on fine-grained description text and prompt words. After the fine-tuning process in step 203, the model has the ability to process modal information, and the modal information can be input into the model to generate a dialogue.

[0141] The modal information used in step 204 may be the same as or different from the modal information in the previous steps. Even if unlabeled modal information is input into the model, the fine-tuned model is capable of understanding and processing it.

[0142] In this way, by first generating conversation content based on fine-grained descriptive text, and then combining the conversation content with the original modal information, the model is fine-tuned so that the fine-tuned model has the ability to process modal information. Then, the fine-tuned model can be directly used to generate more conversation content based on the modal information. In this way, the fine-tuned model's ability to understand modal information can be used to generate richer and more diverse training samples, providing sample support for subsequent iterative fine-tuning.

[0143] After obtaining the conversation content generated based on the modal information, the fine-tuned multimodal conversation model can be further fine-tuned based on the generated conversation content. After fine-tuning, the fine-tuned multimodal conversation model can be used to generate new conversation context and conversation context based on the modal information, and the multimodal conversation model can be fine-tuned based on the new conversation context and conversation context. The process of generating conversation content and fine-tuning can be iterated multiple times until a stopping condition is met.

[0144] Optionally, after obtaining the conversation context and conversation context generated based on the modal information, the generated conversation context and conversation context may be filtered or optimized before fine-tuning the model.

[0145] There are many ways to screen and optimize. For example, the generated conversation content can be input into a scoring model to screen out high-quality conversation content based on the score. Alternatively, it can be combined with the descriptive text of the modal information to determine whether the conversation context and the conversation context generated based on the modal information match based on the descriptive text, and then screen out the matching conversation content.

[0146] Alternatively, manual feedback can be used to screen and rewrite the dialogue content generated based on modal information and further fine-tune the model.

[0147] Optionally, generating a conversation context and a conversation context based on modal information using the fine-tuned multimodal dialogue model, and continuing to fine-tune the multimodal dialogue model may include:

[0148] Through the fine-tuned multimodal dialogue model, the preceding and following conversations are generated based on the modal information, and training samples generated based on the modal information are obtained;

[0149] Sending a screening instruction to a screening terminal, wherein the screening instruction is used to display at least one training sample generated based on the modality information to a screening person;

[0150] Obtaining a training sample returned by the screening terminal and obtained by the screening personnel after screening and / or rewriting the at least one training sample generated based on the modality information;

[0151] Fine-tuning the multimodal dialogue model based on the returned training samples;

[0152] The above process of generating training samples based on modal information, screening, rewriting and fine-tuning is repeated until the performance of the multimodal dialogue model meets the preset requirements.

[0153] Specifically, the fine-tuned model can generate the context of the conversation and the context of the conversation based on the modal information. The modal information, the context of the conversation and the context of the conversation here can be used as training samples generated based on the modal information. The training samples can be manually screened and rewritten. For example, high-quality training samples can be screened out from at least one training sample, or at least part of the training samples can be rewritten so that the rewritten context of the conversation and the context of the conversation are more in line with the actual conversation mode.

[0154] After the screening terminal returns the manually screened and rewritten training samples, the model can be fine-tuned based on the returned training samples. After fine-tuning, the above process can be repeated multiple times until the preset requirements are met. The preset requirements can be set according to actual needs.

[0155] In an optional implementation, a training sample set and a test sample set can be constructed, and the model can be fine-tuned based on the training sample set. After the fine-tuning is completed, the fine-tuned model can be tested based on the test sample set. If the performance of the model is determined to meet the requirements based on the test sample set, the fine-tuning operation is terminated, otherwise the fine-tuning continues.

[0156] In another optional implementation, after sending the training samples generated based on the modal information to the screening personnel, in addition to obtaining the training samples screened and rewritten by the screening personnel, the screening personnel's evaluation information on the training samples generated based on the modal information can be further obtained. The evaluation information is used to evaluate the quality of the training samples generated by the model. If the quality of the newly generated training samples basically meets the requirements, then the fine-tuning can be considered to be ended. Otherwise, the fine-tuning can be continued.

[0157] Alternatively, the quality of the generated training samples can be measured based on the difference between the training samples returned by the screener and the training samples generated by the model. If the difference before and after the screening rewrite is not large, it means that the model can already generate higher-quality answers, and you can consider ending fine-tuning at this time.

[0158] Through the above method, the conversation content generated by the fine-tuned model can be manually filtered to screen out high-quality questions and answers, and some answers can be modified as necessary to make them more in line with expectations. The model can then be fine-tuned based on the screened and rewritten conversations to make the new model have better performance, which can then be used for further sample generation, thus realizing a cycle of feedback and model updates and continuously optimizing model performance.

[0159] Optionally, a fourth display instruction may be output for displaying a modality list to the user, the modality list including images, audio, and video; and obtaining any one or more modalities selected by the user from the modality list. Accordingly, constructing a multimodal dialogue model based on the language model may include: adding encoders corresponding to the one or more modalities selected by the user to the language model to obtain the multimodal dialogue model.

[0160] In this embodiment of the present application, the user is allowed to select the modality to be trained. For example, the user can choose to train a model that can process images, a model that can process audio, or a model that can process both audio and images. Accordingly, when building a multimodal conversation model, encoders for the corresponding modalities can be added.

[0161] Specifically, a list of selectable modalities can be displayed to the user through the front-end interactive interface. The user selects one or more modalities in the interactive interface. For example, if image is selected, the modal information obtained during the training process can include content of the image modality.

[0162] Through the above method, users can independently select the processing capabilities of the multimodal dialogue model, so that the trained multimodal dialogue model can be applied to various actual needs, meet the personalized needs of users, and improve the user experience.

[0163] In an embodiment of the present application, various basic functions related to model training can be opened to users through the front-end interactive interface for adjusting the model training strategy, including but not limited to: model input, model output, model structure, model training process, etc.

[0164] For example, in terms of model input, users can adjust the input format and modality, and add, modify, or filter input information in training samples. In terms of model output, users can adjust the output format and modality, and modify and filter labels. In terms of model structure, users can adjust the model's specific structure and parameter count, and add, reduce, or modify one or more modules in the model. In terms of the model training process, users can plan the model's training stages, selecting the dataset and input / output strategies, loss function, and training stop conditions for each stage.

[0165] Optionally, the interactive interface can display relevant information to assist the user in making a selection, such as the various available options or the detailed content of each available option. Furthermore, the interactive interface can also display dynamic information about the model training process, such as intermediate results of model training, to facilitate timely updates to the strategy.

[0166] In summary, this embodiment can obtain modal information, determine the descriptive text corresponding to the modal information, generate a conversation context and a conversation aftercontext based on the descriptive text using a pretrained language model, obtain self-generated training samples, construct a multimodal dialogue model based on the language model, fine-tune the multimodal dialogue model based on the self-generated training samples, and use the fine-tuned multimodal dialogue model to generate a conversation context and a conversation aftercontext based on the modal information, and continue to fine-tune the multimodal dialogue model, thereby enabling the model to process multimodal information. The model's multimodal understanding ability can be used for iterative fine-tuning, requiring only a small amount of preliminary annotation to complete model training, reducing the time spent on manual annotation and improving overall training efficiency. Furthermore, by first generating training samples using descriptive text and then using the modal information itself, it is possible to mine the rich detailed features in the modal information and improve the accuracy of the model.

[0167] Based on the technical solutions provided in the above embodiments, after completing the fine-tuning operation, the multimodal dialogue model can also be optimized by combining contrastive learning and reinforcement learning. Figure 5 is a flow chart of an optimization model provided in an embodiment of the present application. As shown in Figure 5, after fine-tuning the model, the method can also include:

[0168] Step 501: construct a matching model based on the fine-tuned multimodal dialogue model, and train the matching model based on multiple training samples.

[0169] The matching model includes two branches. The input of the first branch is modal information and the previous conversation context, and the input of the second branch is the following conversation context. The matching model is used to determine the degree of matching between the inputs of the two branches.

[0170] In an embodiment of the present application, the matching model can be implemented based on contrastive learning, aiming to determine the context of the conversation that is more consistent with expectations, more consistent with the modal information and the context of the conversation.

[0171] Optionally, the two branches of the matching model can be initialized using the fine-tuned parameters of the multimodal dialogue model to improve the training effect of the matching model. Of course, the matching model can also be initialized using other parameters.

[0172] As described above, a multimodal conversation model can include a multimodal encoder and a language model. Specifically, the multimodal encoder can be used to obtain modal information and encode it to obtain element information corresponding to the modal information; the language model can be used to obtain element information corresponding to the modal information and element information corresponding to the previous conversation context, and output a predicted next conversation context.

[0173] In this embodiment, optionally, the language model may specifically include: a text processing module, used to obtain element information corresponding to the modal information and element information corresponding to the previous context of the conversation, and output corresponding embedding features (embedding); an output layer, used to output the predicted following context of the conversation based on the embedding features.

[0174] Accordingly, in this step, a matching model is constructed based on the fine-tuned multimodal dialogue model, and the matching model is trained according to a plurality of training samples, which may include:

[0175] Initializing the first branch of the matching model based on the parameters of the multimodal encoder and the text processing module in the fine-tuned multimodal dialogue model, and initializing the second branch of the matching model based on the parameters of the text processing module in the fine-tuned multimodal dialogue model;

[0176] During the training process, the modal information and conversation context in any training sample are input into the first branch, and the conversation context in any training sample is input into the other branch. Based on the embedded features output by the two branches, the parameters of the matching model are adjusted through comparative learning supervision.

[0177] Figure 6 is a schematic diagram of the principles of contrastive learning provided by an embodiment of the present application. As shown in Figure 6, the matching model includes two branches: the first branch includes a multimodal encoder and a text processing module for inputting modal information and the previous conversation context; the second branch includes a text processing module for inputting the next conversation context. Both branches are supervised by contrastive learning.

[0178] Optionally, in a multimodal dialogue model, an embedded feature can be obtained through autoregression based on the modal information and the token corresponding to the previous dialogue context. In the matching model, a special token can be added after the token corresponding to the modal information and the previous dialogue context, and the feature output after the special token is obtained as the embedded feature corresponding to the modal information and the previous dialogue context. The feature is then compared with the embedded feature output by another branch of the matching model, and the matching model is trained through comparative learning supervision.

[0179] The goal of training is to make the embedding features corresponding to the modal information and the previous conversation context, and the embedding features corresponding to the next conversation context in the same training sample as close as possible, and to make the embedding features corresponding to the modal information and the previous conversation context, and the embedding features corresponding to the next conversation context in different training samples as far apart as possible.

[0180] Exemplarily, a training sample may include an image, a question, and an answer. The image and question are input into the first branch of the matching model, and the answer is input into the second branch of the matching model. If the content input into the two branches matches, that is, they belong to the same training sample, then the embedded features output by the two branches will be closer. If the image, question, and answer do not match, that is, they do not belong to the same training sample, then the embedded features input into the two branches will be further apart.

[0181] Optionally, the loss function for contrastive learning can use InfoNCEloss. The loss value can be calculated using the following formula:

[0182] Among them, when InfoNCEloss is applied to the embodiment of the present application, the meanings of the relevant parameters are: K is the number of training samples, during training, q is the embedding feature corresponding to the input of the first branch (modal information and conversation context), k+ is the embedding feature corresponding to the conversation context matching the input, ki is the embedding feature corresponding to the conversation context in the i-th training sample, and τ is the temperature coefficient.

[0183] The trained matching model can be used to determine the matching degree of the inputs of the two branches. Specifically, the closer the embedded features of the two branches output, the higher the matching degree, and vice versa.

[0184] By comparing with existing labeled data, the matching model can learn and adjust according to the difference between the predicted results and the actual labels, thereby improving the accuracy of the matching model and the effect of subsequent optimization of the multimodal dialogue model based on the matching model.

[0185] The training samples used when training the matching model may be consistent with or different from the training samples used when fine-tuning the multimodal dialogue model.

[0186] Optionally, in addition to the aforementioned training samples, a massive amount of image-text pair data can also be used to train the matching model.

[0187] Specifically, we can crawl multiple sets of image-text pairs from the web. Each set of image-text pairs can include an image and its corresponding description, for example, an image and its corresponding title. The image and description information are fed into the two branches of the matching learning process for comparative learning.

[0188] Step 502: Use the fine-tuned multimodal dialogue model as a policy model for reinforcement learning, determine reward information in reinforcement learning based on the matching model, and optimize the multimodal dialogue model and the matching model through proximal policy optimization.

[0189] Among them, the reinforcement learning used in the embodiment of the present application can be reinforcement learning based on a strategy model. In reinforcement learning, state information (state) is input into the strategy model to obtain action information (action), and the corresponding reward information (reward) is determined, and the strategy model is iteratively optimized based on the reward information.

[0190] Optionally, this step may specifically include:

[0191] Inputting the modal information and the previous conversation context into a multimodal conversation model to obtain at least one predicted conversation context;

[0192] For any predicted conversation context, the modal information and the previous conversation context are input into the first branch of the matching model, and the predicted conversation context is input into the second branch of the matching model to obtain the matching degree as the reward information corresponding to the predicted conversation context;

[0193] According to the reward information corresponding to each predicted dialogue context, the multimodal dialogue model and the matching model are optimized through proximal strategy optimization.

[0194] In this embodiment, the reinforcement learning strategy model can be the multimodal dialogue model. The modal information and the preceding dialogue context can serve as state information. After being input into the strategy model, the resulting predicted dialogue context can serve as behavioral information. There can be one or more predicted dialogue contexts. When there are multiple predicted dialogue contexts, each predicted dialogue context can be associated with a reward. This reward information can be a score determined by a matching model, representing the degree of match between the behavioral information (predicted dialogue context) and the state information (modal information and preceding dialogue context).

[0195] After receiving the reward information, the policy model (i.e., the multimodal dialogue model) and the matching model can be optimized using Proximal Policy Optimization (PPO). PPO is a reinforcement learning algorithm that aims to maximize the expected cumulative reward by optimizing the model's policy.

[0196] Specifically, PPO can be trained through multiple iterations. In each iteration, the same state information can generate multiple predicted dialogue contexts. Higher-quality predicted dialogue contexts receive higher scores. The scores can then be used to update the parameters of the multimodal dialogue model and matching model, optimizing them for more accurate scoring and prediction.

[0197] Through repeated iterative optimization, the multimodal dialogue model and matching model will gradually learn more accurate and realistic prediction and scoring strategies, thereby improving the accuracy and robustness of the model. Furthermore, in contrastive learning and reinforcement learning, the global features of the input information are supervised. Compared to the token-by-token prediction during the fine-tuning phase of the multimodal dialogue model, this can better achieve globally consistent rewards or penalties, further improving the model's effectiveness and reducing hallucinations.

[0198] This embodiment of the present application also provides a method for generating a multimodal conversation model, integrating the fine-tuning and optimization processes of each of the above stages. Figure 7 is a schematic diagram of the principles of a method for generating a multimodal conversation model provided in an embodiment of the present application. As shown in Figure 7, in the first stage, basic data construction and data enhancement can be performed, and the model can be fine-tuned.

[0199] Specifically, by acquiring multimodal information from the network and manually annotating it to obtain basic training samples, the team then uses a pre-trained language model, such as the LLM, and a concatenation strategy to expand the training samples. The resulting training samples are then used for supervised fine-tuning of the multimodal dialogue model, which can include a multimodal encoder and a language model.

[0200] In the second stage, the model can be iteratively fine-tuned through human feedback self-instruction.

[0201] Specifically, we can use the fine-tuned multimodal dialogue model to input a large amount of modal data to generate corresponding dialogue content as training samples, manually filter out the poor dialogue content, and rewrite some dialogue content. The screened and rewritten training samples can be used to fine-tune the multimodal dialogue model, forming a flywheel continuous optimization model of feedback and model update.

[0202] In the third stage, in order to overcome the possible hallucination phenomenon of the model, the multimodal dialogue model can be optimized based on the matching model and reinforcement learning to help the multimodal dialogue model predict answers that are more in line with expectations and more consistent with the input information.

[0203] Specifically, the two branches of the matching model can be constructed from a multimodal encoder and a language model (specifically, the text processing module within the language model used to derive embedded features). In contrastive learning, the degree of match between the two branches of the matching model can be determined. One branch receives inputs from the modal information and the preceding conversation context, while the other receives inputs from the following conversation context. In reinforcement learning, the modal information and preceding conversation context can be input into the multimodal dialogue model to generate multiple predicted following conversation contexts. These are then scored based on the matching model, and the parameters of the multimodal dialogue model and the matching model are updated based on the scores.

[0204] This embodiment enables fine-tuning of multimodal dialogue model instructions. It fully leverages the textual capabilities of the language model, combining it with basic data and manual feedback iteration to enable the fine-tuned model to possess multimodal dialogue and comprehension capabilities. The introduction of manual feedback from instruction iteration can address inherent model capabilities and improve the model. This eliminates the need for extensive manual annotation of data, reducing annotation costs and, consequently, the overall production cost of fine-tuning data. This allows for more targeted model updates through model construction and data iteration. Furthermore, this embodiment does not require additional models to assist in understanding multimodal information. The system is relatively simple, effectively reducing required computing resources while better adapting to multimodal tasks. Furthermore, utilizing a matching model based on contrastive learning, combined with reinforcement learning, can maximize the incentive for the model to output accurate information, reduce model fantasies, and generate outputs that better meet requirements.

[0205] The multimodal conversation model provided in the embodiments of this application can be applied to a variety of scenarios, such as e-commerce, conferences, customer service, etc. Figure 8 is a flowchart of another method for generating a multimodal conversation model provided in the embodiments of this application. Figure 8 shows a specific training method for a multimodal conversation model in an e-commerce scenario.

[0206] As shown in FIG8 , the method includes:

[0207] Step 801: Acquire modal information of a product, where the modal information includes at least one of the following: an image, a video, or an audio of the product.

[0208] The product may be any product on the e-commerce platform, and the modal information of the product may be images, videos, audio, etc. of the product displayed on the e-commerce platform.

[0209] Step 802: Determine the description text corresponding to the product by extracting text from the product details page and / or manually annotating it. Based on the description text, generate a conversation context and a conversation context using a pre-trained language model to obtain a self-generated training sample. The self-generated training sample includes: the modal information, the conversation context, and the conversation context.

[0210] In this embodiment, the descriptive text corresponding to the modal information can be generated by manual annotation or directly extracted from the product details page. For example, the product details page may include images, videos, audio, etc. of the product, and may also include text for introducing the product. Using the text as the descriptive text corresponding to the modal information, or using the text as the corresponding descriptive text after manual optimization, can effectively improve the efficiency of determining the descriptive text and help to make the generated descriptive text richer and more comprehensive.

[0211] Step 803: construct a multimodal dialogue model based on the language model, and fine-tune the multimodal dialogue model based on the self-generated training samples.

[0212] Step 804: Generate a conversation context and a conversation subcontext based on the modal information using the fine-tuned multimodal dialogue model, and continue to fine-tune the multimodal dialogue model. The fine-tuned multimodal dialogue model is used to determine the conversation subcontext based on the modal information of the product and the conversation context.

[0213] The specific implementation principle and process of the method in this embodiment can be found in the above embodiments and will not be repeated here.

[0214] In summary, the method for generating a multimodal dialogue model provided in this embodiment can fine-tune the model according to the multimodal information of the product, so that the model has the ability to process multimodal information, and can use the model's multimodal understanding ability to perform iterative fine-tuning. Only a small amount of preliminary annotation is required to complete the model training, reducing the time spent on manual annotation and improving the overall training efficiency. In addition, by first using descriptive text to generate training samples and then using the modal information itself to generate training samples, it is possible to mine the rich detailed features in the product modal information, improve the accuracy of the model, meet the needs of intelligent customer service dialogue in e-commerce scenarios, and improve user experience.

[0215] Referring to the above method, the embodiment of the present application also provides a method for generating a multimodal model, which can be applied to dialogue and any other fields, for example, translation, summary generation, automatic question answering, dialogue robots and other downstream tasks. Figure 9 is a flow chart of a method for generating a multimodal model provided by an embodiment of the present application. As shown in Figure 9, the method includes:

[0216] Step 901: Acquire modal information, where the modal information includes any one or more modal contents except text.

[0217] Step 902: Determine the descriptive text corresponding to the modal information, and generate input information and output information based on the descriptive text through a pre-trained basic model to obtain a self-generated training sample. The self-generated training sample includes: the modal information, the input information, and the output information.

[0218] Step 903: construct a multimodal model based on the basic model, and fine-tune the multimodal model based on the self-generated training samples.

[0219] Step 904: Generate input information and output information based on the modal information using the fine-tuned multimodal model, and continue to fine-tune the multimodal model; wherein the fine-tuned multimodal model is used to determine output information based on the modal information and the input information.

[0220] The input information can be any information to be input into the base model, such as previous conversations, text to be translated, or requirement information. The output information can be any information output by the base model, such as subsequent conversations, translated text, or code information. The base model can be any pre-trained model, such as a language model used to generate text output based on text input. It can also be other base models, such as a model that generates images from text.

[0221] For example, in the field of translation, the translated text can be obtained through a multimodal model based on the modal information and the text to be translated. For example, given an image or video, the text to be translated is the content contained in the image or video described in the first language. The text is translated into the second language to obtain the translated text. When combining the image and video during translation, it is possible to refer to the richer content in the image and video, thereby improving the accuracy of the translation.

[0222] In the field of code generation, code information can be obtained through a model based on modal information and requirement information. For example, given a diagram, the requirement information is the corresponding analysis based on this diagram, and the code information is the code generated based on the diagram and requirement information. Therefore, multiple modal information can be referenced during code generation to meet the needs of different scenarios.

[0223] In addition to translation and code generation, it can also be applied to other fields. For example, the modal information can be a video, the input information can be an instruction to generate a corresponding text based on the video, and the text output information is the text obtained by the model. For another example, the modal information can be a chart, the input information can be a piece of code for analyzing the chart, and the code can be debugged through the model. The text output information obtained can be the debugged code, or problems in the code. In actual applications, the input and output information of the model can be determined according to the specific application field of the model, and this embodiment does not limit it.

[0224] The base model can also be a model that derives outputs from other modalities based on text. For example, if a model generates images from text, the output information can be an image. The base model can be further overlaid with the capabilities of other modalities, such as audio, to construct a multimodal model that derives images from audio and text.

[0225] Optionally, the method further includes:

[0226] According to the modal information, input information and output information are determined by manual labeling to obtain basic training samples; wherein the obtained basic training samples and self-generated training samples are both multiple;

[0227] Splicing at least two original training samples to obtain a spliced ​​training sample; wherein the original training sample is a training sample arbitrarily selected from a plurality of basic training samples and a plurality of self-generated training samples;

[0228] Accordingly, fine-tuning the multimodal model according to the self-generated training samples includes: fine-tuning the multimodal model according to the basic training samples, the spliced ​​training samples and the self-generated training samples.

[0229] Optionally, at least two original training samples are concatenated to obtain a concatenated training sample, including:

[0230] Extracting modal information of each original training sample from the at least two original training samples to obtain at least two modal information, and extracting input information from a first original training sample; wherein the first original training sample is any one of the at least two original training samples;

[0231] Generate indication information, where the indication information is used to indicate a sequence number of the modal information corresponding to the extracted input information in the at least two modal information;

[0232] Generate a spliced ​​training sample based on the extracted at least two modal information, input information and instruction information.

[0233] Optionally, determining input information and output information by manual labeling based on the modal information to obtain basic training samples includes:

[0234] Sending a first display instruction to the first terminal, where the first display instruction is used to display the modal information to the first annotator;

[0235] Obtaining input information returned by the first terminal and annotated by the first annotator according to the modal information;

[0236] Sending a second display instruction to the second terminal, where the second display instruction is used to display the modal information and the annotated input information to the second annotator;

[0237] Obtaining output information returned by the second terminal and annotated by the second annotator based on the modal information and the input information;

[0238] The determining of the description text corresponding to the modal information includes:

[0239] Sending a third display instruction to a third terminal, wherein the third display instruction is used to display the modal information to a third annotator;

[0240] Obtain the description text annotated by the third annotator based on the modal information and returned by the third terminal.

[0241] Optionally, generating input information and output information based on the modal information using the fine-tuned multimodal model, and further fine-tuning the multimodal model includes:

[0242] Through the fine-tuned multimodal model, input information and output information are generated according to the modal information, and training samples generated based on the modal information are obtained;

[0243] Sending a screening instruction to a screening terminal, wherein the screening instruction is used to display at least one training sample generated based on the modality information to a screening person;

[0244] Obtaining a training sample returned by the screening terminal and obtained by the screening personnel after screening and / or rewriting the at least one training sample generated based on the modality information;

[0245] Fine-tuning the multimodal model based on the returned training samples;

[0246] The above process of generating training samples based on modal information, screening, rewriting and fine-tuning is repeated until the performance of the multimodal model meets the preset requirements.

[0247] Optionally, generating input information and output information based on the description text through a pre-trained basic model includes:

[0248] Obtain prompt words, input the description text and the prompt words into a pre-trained basic model to obtain input information, and input the description text and the obtained input information into the basic model to obtain output information;

[0249] The fine-tuned multimodal model generates input information and output information based on modal information, including:

[0250] The modal information and the prompt word are input into the fine-tuned multimodal model to obtain input information, and the modal information and the obtained input information are input into the fine-tuned multimodal model to obtain output information.

[0251] Optionally, the method further includes:

[0252] Outputting a fourth display instruction, wherein the fourth display instruction is used to display a modality list to the user, wherein the modality list includes: image, audio, and video;

[0253] Obtaining any one or more modalities selected by the user from the modality list;

[0254] Accordingly, a multimodal model is constructed based on the basic model, including:

[0255] According to one or more modalities selected by the user, encoders corresponding to the one or more modalities are added to the basic model to obtain the multimodal model.

[0256] Optionally, the method further includes:

[0257] Based on the fine-tuned multimodal model, a matching model is constructed and trained based on a plurality of training samples; wherein the matching model includes two branches, the input of the first branch is the modal information and the input information, and the input of the second branch is the output information, and the matching model is used to determine the degree of matching between the inputs of the two branches;

[0258] The fine-tuned multimodal model is used as a policy model for reinforcement learning, and reward information in reinforcement learning is determined based on the matching model. The multimodal model and the matching model are optimized through proximal policy optimization.

[0259] Optional multimodal models built from pre-trained base models include:

[0260] A multimodal encoder is used to obtain modal information and encode the modal information to obtain element information corresponding to the modal information;

[0261] A pre-trained basic model, the basic model comprising: a text processing module for obtaining element information corresponding to the modal information and element information corresponding to the input information, and outputting corresponding embedded features; an output layer for outputting predicted output information based on the embedded features;

[0262] Accordingly, a matching model is constructed based on the fine-tuned multimodal model, and the matching model is trained based on multiple training samples, including:

[0263] Initializing the first branch of the matching model based on the parameters of the multimodal encoder and the text processing module in the fine-tuned multimodal model, and initializing the second branch of the matching model based on the parameters of the text processing module in the fine-tuned multimodal model;

[0264] During the training process, the modal information and input information in any training sample are input into the first branch, and the output information in any training sample is input into the other branch. According to the embedded features output by the two branches, the parameters of the matching model are adjusted through comparative learning supervision.

[0265] Optionally, the fine-tuned multimodal model is used as a policy model for reinforcement learning, and reward information in reinforcement learning is determined based on the matching model. The multimodal model and the matching model are optimized through proximal policy optimization, including:

[0266] Inputting modal information and input information into a multimodal model to obtain at least one predicted output information;

[0267] For any predicted output information, the modal information and input information are input into the first branch of the matching model, and the predicted output information is input into the second branch of the matching model to obtain the matching degree as the reward information corresponding to the predicted output information;

[0268] According to the reward information corresponding to each predicted output information, the multimodal model and the matching model are optimized through proximal strategy optimization.

[0269] The present application also provides a method for generating a multimodal translation model, which is applied to the translation field and includes:

[0270] Acquiring modal information, the modal information including content in any one or more modalities other than text; determining a description text corresponding to the modal information; and generating a text to be translated and a translated text based on the description text using a pre-trained language model to obtain a self-generated training sample, the self-generated training sample including: the modal information, the text to be translated, and the translated text;

[0271] Constructing a multimodal translation model based on the language model, and fine-tuning the multimodal translation model based on the self-generated training samples;

[0272] The fine-tuned multimodal translation model is used to generate a text to be translated and a translated text based on the modal information, and the multimodal translation model is further fine-tuned; wherein the fine-tuned multimodal translation model is used to determine the translated text based on the modal information and the text to be translated.

[0273] The present application also provides a method for generating a multimodal code model, which is applied to the field of code generation and includes:

[0274] Acquiring modal information, the modal information including content in any one or more modalities other than text; determining a description text corresponding to the modal information, and generating requirement information and code information based on the description text using a pre-trained language model to obtain a self-generated training sample, the self-generated training sample including: the modal information, the requirement information, and the code information;

[0275] constructing a multimodal code model based on the language model, and fine-tuning the multimodal code model based on the self-generated training samples;

[0276] The multimodal code model is fine-tuned, and requirement information and code information are generated based on the modal information, and the multimodal code model is further fine-tuned; wherein the fine-tuned multimodal code model is used to determine the code information according to the modal information and the requirement information.

[0277] In various fields, the model training method can refer to the solutions provided in the aforementioned embodiments, as long as the following are replaced in the aforementioned embodiments: the multimodal dialogue model is replaced with a corresponding model, such as a multimodal model, a multimodal translation model, a multimodal code model, etc.; the language model is replaced with a basic model; the conversation context is replaced with the actual input of the model, such as input information, text to be translated, requirement information, etc.; the conversation context is replaced with the actual output of the model, such as output information, translated text, code information, etc.

[0278] This embodiment can obtain modal information, determine the descriptive text corresponding to the modal information, generate input information and output information based on the descriptive text through a pre-trained basic model, obtain self-generated training samples, construct a multimodal model based on the basic model, fine-tune the multimodal model based on the self-generated training samples, and generate input information and output information based on the modal information through the fine-tuned multimodal model, and continue to fine-tune the multimodal model, so that the model has the ability to process multimodal information, and can use the model's multimodal understanding ability to perform iterative fine-tuning. Only a small amount of preliminary annotation is required to complete the model training, reducing the time spent on manual annotation and improving the overall training efficiency. In addition, by first using the descriptive text to generate training samples and then using the modal information itself to generate training samples, it is possible to mine the rich detailed features in the modal information and improve the accuracy of the model.

[0279] FIG10 is a flow chart of a multimodal processing method provided in an embodiment of the present application. As shown in FIG10 , the method may include:

[0280] Step 1001: Obtain information to be processed, where the information to be processed includes modal information and input information, and the modal information includes content of any one or more modalities other than text.

[0281] Step 1002: Determine output information corresponding to the information to be processed based on the multimodal model.

[0282] The multimodal model is generated by the method described in any of the above embodiments, and may be, for example, a multimodal dialogue model or a multimodal translation model.

[0283] Optionally, the method may be applied to a server or a terminal device, or may be performed jointly by a server and a terminal device.

[0284] The multimodal processing method provided in this embodiment can determine the output information in the interaction process based on the multimodal model, wherein the multimodal model has the ability to process multimodal information and can use the model's multimodal understanding ability to perform iterative fine-tuning. Only a small amount of preliminary annotation is required to complete the model training, reducing the time spent on manual annotation and improving the overall training efficiency. Moreover, in the fine-tuning process, the descriptive text is first used to generate training samples, and then the modal information itself is used to generate training samples, which can mine the rich detailed features in the modal information and improve the accuracy of the model.

[0285] Optionally, in the context of conversation, the input information and output information may be the preceding and following conversation context, respectively. The multimodal model is specifically a multimodal conversation model, comprising: a multimodal encoder for processing modal information to obtain element information corresponding to the modal information; and a language model for processing the element information corresponding to the modal information and the element information corresponding to the preceding conversation context. After determining the following conversation context corresponding to the input information based on the multimodal conversation model, the following conversation context may be presented to the user, and the user's next input may be obtained. If it is detected that the user's next input content still corresponds to the aforementioned modal information, the language model and the element information corresponding to the modal information are used to determine the next conversation context and present it to the user.

[0286] In this way, in multi-round dialogue interactions, if the user's questions in a certain round are directed to an image in a previous round, the element information corresponding to the image obtained in the previous round and the element information of the question in the current round can be directly spliced ​​and input into the language model for processing, omitting the step of inputting the image into the multimodal encoder for processing, effectively improving the processing efficiency in the actual application process, increasing the response speed, and enhancing the user experience.

[0287] The present application also provides an application system for a multimodal dialogue model, the multimodal dialogue model comprising: a multimodal encoder for processing modal information to obtain element information corresponding to the modal information; a language model for processing element information corresponding to the modal information and element information corresponding to the dialogue context;

[0288] The system includes: a terminal-side device and a cloud-side device; the terminal-side device is deployed with a language model, and the cloud-side device is deployed with a multimodal encoder and a language model;

[0289] The terminal-side device is configured to obtain information to be processed and send it to the cloud-side device, where the information to be processed includes modal information and conversation context, and at least part of the content of the information to be processed is provided by the user;

[0290] The cloud-side device is configured to obtain, based on the multimodal dialogue model, element information corresponding to the modal information and a dialogue context corresponding to the information to be processed, and send the information to the terminal-side device;

[0291] The terminal-side device is also used to: display the conversation context to the user and obtain the content input by the user in the next round; if the content input by the user in the next round is still for the aforementioned modal information, the locally deployed language model and the element information corresponding to the modal information are used to determine the conversation context for the next round and display it to the user; specifically, the element information corresponding to the modal information and the element information corresponding to the conversation context (including the content input in the next round) can be input into the language model to obtain the conversation context for the next round.

[0292] The multimodal dialogue model is generated by the method described in any of the aforementioned embodiments.

[0293] In this embodiment, the system can adopt a traditional server architecture, and the cloud-side device and the end-side device can be a server and a user terminal respectively. Alternatively, the system can adopt a distributed cloud architecture, and the cloud-side device and the end-side device can be a central cloud device and an edge cloud device respectively. All of the above devices can execute the method for generating a multimodal dialogue model provided in any embodiment of the present application.

[0294] In practical applications, a multimodal dialogue model can be deployed through end-cloud collaboration to assist in dialogue interaction. Specifically, the cloud-side device deploys a multimodal encoder and a language model. The multimodal encoder processes modal information to obtain modal information, such as element information corresponding to an image. The language model then processes the element information corresponding to the image and the element information corresponding to the previous conversation, such as the question, to obtain the next conversation, such as the answer.

[0295] The end-side device can obtain the element information corresponding to the image and the answer, and display the answer to the user. The end-side device is deployed with a language model. After obtaining the next round of user input questions, if the next round of user input questions are still related to the previous image, the end-side device uses the locally deployed language model, the element information corresponding to the image, and the next round of questions to determine the next round of answers and display them to the user.

[0296] It is understandable that the terminal device displays the answer to the user, either directly or by sending the answer to other terminals, which then display it to the user.

[0297] The system provided in this embodiment deploys models on the terminal-side device and the cloud-side device respectively, wherein the terminal-side device only deploys the language model. During the conversation with the user, the step of inputting the image into the multimodal encoder for processing can be appropriately omitted according to the conversation situation, thereby reducing the number of interactions with the cloud-side device, effectively improving the processing efficiency in the actual application process, increasing the response speed, and enhancing the user experience.

[0298] Corresponding to the above system, an embodiment of the present application further provides a multimodal dialogue method, which is applied to a terminal-side device, including:

[0299] Obtain information to be processed and send it to the cloud-side device, where the information to be processed includes modal information and conversation context, and at least part of the content of the information to be processed is provided by the user, so that the cloud-side device is used to obtain element information corresponding to the modal information and conversation context corresponding to the information to be processed according to the multimodal conversation model, and send it to the terminal-side device; display the conversation context to the user, and obtain the content input by the user in the next round; if the content input by the user in the next round is still for the aforementioned modal information, use the locally deployed language model and the element information corresponding to the modal information to determine the conversation context for the next round and display it to the user.

[0300] The present application also provides a multimodal conversation method, which is applied to a cloud-side device and includes:

[0301] Obtain information to be processed sent by the terminal device, where the information to be processed includes modal information and conversation context, and at least part of the content of the information to be processed is provided by the user; according to the multimodal dialogue model, obtain element information corresponding to the modal information and conversation context corresponding to the information to be processed, and send them to the terminal device, so that the terminal device displays the conversation context to the user and obtains the content input by the user in the next round. If the content input by the user in the next round is still for the aforementioned modal information, use the language model locally deployed on the terminal device and the element information corresponding to the modal information to determine the conversation context for the next round and display it to the user.

[0302] In other optional implementations, the language model may not be deployed on the terminal device, and only the multimodal dialogue model may be deployed on the server. The conversation context of each round of dialogue is determined by the cloud-side device, which can reduce the burden on the terminal device and has strong applicability.

[0303] In the technical solutions provided in the above embodiments, a model with text processing capabilities is used as the basic model, and based on this text processing capability, processing capabilities for other modalities are further expanded. Referring to the above principles, a multimodal model can be constructed based on the basic models of other modalities.

[0304] Specifically, an embodiment of the present application further provides a method for generating a multimodal model, which constructs a multimodal modality through a pre-trained basic model, wherein the pre-trained basic model has the ability to obtain information of a second modality based on information of a first modality; the method for generating a multimodal model may include:

[0305] Acquiring modal information, the modal information including content of any one or more modalities other than the first modality; obtaining corresponding description information based on the modal information, the description information being information of the first modality; generating input information and output information based on the description information using a pre-trained basic model to obtain a self-generated training sample, the self-generated training sample including: the modal information, the input information, and the output information; the input information including information of the first modality, and the output information including information of the second modality;

[0306] Constructing a multimodal model based on the basic model, and fine-tuning the multimodal model based on the self-generated training samples;

[0307] The multimodal model is fine-tuned to generate input information and output information based on the modal information, and the multimodal model is further fine-tuned; wherein the fine-tuned multimodal model is used to determine the output information based on the modal information and the input information.

[0308] The first mode and the second mode may be the same or different.

[0309] For example, the first modality is audio and the second modality is an image. The pre-trained base model can generate the corresponding image based on the audio input, that is, it has the ability to generate images from audio. Based on this base model, the capabilities of other modalities, such as text, can be further added to build a multimodal model that can generate images from audio and text.

[0310] Specifically, modal information can be obtained, where the modal information includes text; corresponding description information is obtained based on the text, where the description information is information of the first modality that the basic model can originally process, that is, the description audio corresponding to the text; and then, based on the description audio, input information and output information (corresponding to audio and image, respectively) are generated through the pre-trained basic model to obtain self-generated training samples, that is, the self-generated training samples include: text, input audio, and output image;

[0311] A multimodal model is constructed based on the basic model, and the multimodal model is fine-tuned based on the self-generated training samples, so that the multimodal model has the ability to obtain images based on text and audio; multiple iterations are performed on the fine-tuned multimodal model, and each iteration generates input audio and output images based on text, and the multimodal model is fine-tuned.

[0312] The solution in this embodiment is to replace the input capability of the basic model in the aforementioned embodiment from text to the first modality. The specific implementation principle, process and effect can be referred to the aforementioned embodiment and will not be repeated here.

[0313] Corresponding to the above method, the embodiments of the present application also provide corresponding devices for executing the method described in any of the above embodiments.

[0314] In one example, an embodiment of the present application provides a device for generating a multimodal model, including:

[0315] A first acquisition module is configured to acquire modal information, wherein the modal information includes content of any one or more modalities other than text;

[0316] A first determination module is configured to determine a description text corresponding to the modal information, and generate input information and output information based on the description text using a pre-trained basic model to obtain a self-generated training sample, wherein the self-generated training sample includes: the modal information, the input information, and the output information;

[0317] A first fine-tuning module, configured to construct a multimodal model based on the basic model and fine-tune the multimodal model based on the self-generated training samples;

[0318] The second fine-tuning module is used to generate input information and output information based on modal information through the fine-tuned multimodal model, and continue to fine-tune the multimodal model; wherein the fine-tuned multimodal model is used to determine output information based on the modal information and input information.

[0319] In another example, an embodiment of the present application provides a device for generating a multimodal dialogue model, including:

[0320] A second acquisition module is used to acquire modal information, where the modal information includes content of any one or more modalities other than text;

[0321] A second determination module is configured to determine a description text corresponding to the modal information, and generate a conversation context and a conversation context based on the description text using a pre-trained language model to obtain a self-generated training sample, wherein the self-generated training sample includes: the modal information, the conversation context, and the conversation context;

[0322] a third fine-tuning module, configured to construct a multimodal dialogue model based on the language model, and fine-tune the multimodal dialogue model based on the self-generated training samples;

[0323] The fourth fine-tuning module is used to generate a conversation context and a conversation context based on the modal information through the fine-tuned multimodal dialogue model, and continue to fine-tune the multimodal dialogue model; wherein the fine-tuned multimodal dialogue model is used to determine the conversation context based on the modal information and the conversation context.

[0324] In another example, an embodiment of the present application provides a multimodal processing device, including:

[0325] Acquire information to be processed, where the information to be processed includes modal information and input information, where the modal information includes content in any one or more modalities other than text;

[0326] Determining output information corresponding to the information to be processed based on the multimodal model;

[0327] The multimodal model is generated by the method described in any of the aforementioned embodiments.

[0328] The various devices provided in the embodiments of the present application can be used to implement the technical solutions of the aforementioned corresponding embodiments. Their implementation principles and technical effects are similar and will not be repeated here.

[0329] FIG11 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in FIG11 , the electronic device of this embodiment may include:

[0330] at least one processor 1101; and

[0331] a memory 1102 communicatively connected to the at least one processor;

[0332] The memory 1102 stores instructions that can be executed by the at least one processor 1101, and the instructions are executed by the at least one processor 1101 to enable the electronic device to execute the method as described in any of the above embodiments.

[0333] Optionally, the memory 1102 may be independent or integrated with the processor 1101 .

[0334] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0335] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the above embodiments is implemented.

[0336] An embodiment of the present application further provides a computer program product, including a computer program, which implements the method described in any of the aforementioned embodiments when executed by a processor.

[0337] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is merely a logical function division. In actual implementation, other division methods may be used. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not implemented.

[0338] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some of the steps of the methods described in various embodiments of the present application.

[0339] It should be understood that the above-mentioned processor can be a processing unit (Central Processing Unit, referred to as CPU), or it can be other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as DSP), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0340] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0341] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0342] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0343] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0344] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0345] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a multimodal model, characterized in that: include: Acquire modal information, where the modal information includes content of any one or more modalities other than text; Determine a description text corresponding to the modal information, and generate input information and output information according to the description text through a pre-trained basic model to obtain a self-generated training sample, wherein the self-generated training sample includes: the modal information, the input information, and the output information; Constructing a multimodal model based on the basic model, and fine-tuning the multimodal model based on the self-generated training samples; The multimodal model is fine-tuned to generate input information and output information based on the modal information, and the multimodal model is further fine-tuned; wherein the multimodal model is fine-tuned to determine the output information based on the modal information and the input information.

2. The method according to claim 1, characterized in that: Also includes: According to the modal information, input information and output information are determined by manual annotation to obtain basic training samples; wherein the obtained basic training samples and self-generated training samples are both multiple; Splicing at least two original training samples to obtain a spliced ​​training sample; wherein the original training sample is a training sample arbitrarily selected from a plurality of basic training samples and a plurality of self-generated training samples; Correspondingly, fine-tuning the multimodal model according to the self-generated training samples includes: fine-tuning the multimodal model according to the basic training samples, the concatenated training samples and the self-generated training samples.

3. The method according to claim 2, characterized in that At least two original training samples are concatenated to obtain concatenated training samples, including: Extracting modal information of each original training sample from the at least two original training samples to obtain at least two modal information, and extracting input information from the first original training sample; wherein the first original training sample is any one of the at least two original training samples; Generate indication information, where the indication information is used to indicate the sequence number of the modal information corresponding to the extracted input information in the at least two modal information; Generate a spliced ​​training sample based on the extracted at least two modal information, input information and indication information.

4. The method according to claim 2, characterized in that: The step of determining input information and output information by manual labeling according to the modal information to obtain basic training samples includes: Sending a first display instruction to the first terminal, where the first display instruction is used to display the modal information to the first annotator; Acquire input information returned by the first terminal and annotated by the first annotator according to the modal information; Sending a second display instruction to the second terminal, where the second display instruction is used to display the modal information and the annotated input information to the second annotator; Obtaining output information returned by the second terminal and annotated by the second annotator according to the modal information and the input information; The determining of the description text corresponding to the modal information includes: Sending a third display instruction to a third terminal, wherein the third display instruction is used to display the modal information to a third annotator; The description text annotated by the third annotator according to the modal information is obtained and returned by the third terminal.

5. The method according to claim 1, characterized in that Generating input information and output information based on modal information through the fine-tuned multimodal model, and further fine-tuning the multimodal model, including: Through the fine-tuned multimodal model, input information and output information are generated according to the modal information, and training samples generated based on the modal information are obtained; Sending a screening instruction to a screening terminal, wherein the screening instruction is used to display at least one training sample generated based on the modality information to a screening person; Acquire a training sample returned by the screening terminal and obtained by the screening personnel after screening and / or rewriting the at least one training sample generated based on the modal information; Fine-tuning the multimodal model according to the returned training samples; The above process of generating training samples based on modal information, screening, rewriting and fine-tuning is repeated until the performance of the multimodal model meets the preset requirements.

6. The method according to claim 1, characterized in that The step of generating input information and output information according to the description text through a pre-trained basic model includes: Obtain prompt words, input the description text and the prompt words into a pre-trained basic model to obtain input information, input the description text and the obtained input information into the basic model to obtain output information; The multimodal model after fine-tuning generates input information and output information according to the modal information, including: The modal information and the prompt word are input into the fine-tuned multimodal model to obtain input information, and the modal information and the obtained input information are input into the fine-tuned multimodal model to obtain output information.

7. The method according to claim 1, characterized in that Also includes: Outputting a fourth display instruction, wherein the fourth display instruction is used to display a modality list to the user, wherein the modality list includes: image, audio, and video; Obtaining any one or more modes selected by the user from the mode list; Accordingly, a multimodal model is constructed according to the basic model, including: According to one or more modalities selected by the user, encoders corresponding to the one or more modalities are added to the basic model to obtain the multimodal model.

8. The method according to any one of claims 1 to 7, characterized in that: Also includes: Based on the fine-tuned multimodal model, a matching model is constructed, and the matching model is trained according to a plurality of training samples; wherein the matching model includes two branches, the input of the first branch is the modal information and the input information, the input of the second branch is the output information, and the matching model is used to determine the matching degree of the inputs of the two branches; The fine-tuned multimodal model is used as a policy model for reinforcement learning, and reward information in reinforcement learning is determined based on the matching model. The multimodal model and the matching model are optimized by proximal strategy optimization.

9. The method according to claim 8, characterized in that The multimodal models built from pre-trained base models include: A multimodal encoder is used to obtain modal information and encode the modal information to obtain element information corresponding to the modal information; A pre-trained basic model, the basic model comprising: a text processing module, used to obtain element information corresponding to the modal information and element information corresponding to the input information, and output corresponding embedded features; an output layer, used to output predicted output information according to the embedded features; Accordingly, a matching model is constructed based on the fine-tuned multimodal model, and the matching model is trained according to a plurality of training samples, including: Initializing the first branch in the matching model based on the parameters of the multimodal encoder and the text processing module in the fine-tuned multimodal model, and initializing the second branch in the matching model based on the parameters of the text processing module in the fine-tuned multimodal model; During the training process, the modal information and input information in any training sample are input into the first branch, and the output information in any training sample is input into the other branch. According to the embedded features output by the two branches, the parameters of the matching model are adjusted through comparative learning supervision.

10. The method according to claim 8, characterized in that The fine-tuned multimodal model is used as a policy model for reinforcement learning, and reward information in reinforcement learning is determined based on the matching model. The multimodal model and the matching model are optimized by proximal policy optimization, including: Inputting the modal information and the input information into the multimodal model to obtain at least one predicted output information; For any predicted output information, the modal information and the input information are input into the first branch of the matching model, and the predicted output information is input into the second branch of the matching model to obtain the matching degree as the reward information corresponding to the predicted output information; According to the reward information corresponding to each predicted output information, the multimodal model and the matching model are optimized by proximal strategy optimization.

11. A method for generating a multimodal dialogue model, characterized in that: include: Acquire modal information, where the modal information includes content of any one or more modalities other than text; Determine a description text corresponding to the modal information, and generate a conversation context and a conversation context based on the description text by using a pre-trained language model to obtain a self-generated training sample, wherein the self-generated training sample includes: the modal information, the conversation context, and the conversation context; Constructing a multimodal dialogue model according to the language model, and fine-tuning the multimodal dialogue model according to the self-generated training samples; The multimodal dialogue model is fine-tuned to generate a preceding dialogue context and a following dialogue context based on the modal information, and the multimodal dialogue model is further fine-tuned; wherein the fine-tuned multimodal dialogue model is used to determine the following dialogue context based on the modal information and the preceding dialogue context.

12. The method according to claim 11, characterized in that The acquiring of modal information includes: acquiring modal information of the product, wherein the modal information includes at least one of the following: an image, a video, or an audio of the product; The determining the description text corresponding to the modal information includes: determining the description text corresponding to the product by extracting text from a product details page and / or manually annotating the text; Among them, the fine-tuned multimodal dialogue model is used to determine the following dialogue context based on the modal information of the product and the previous dialogue context.

13. A multimodal processing method, characterized in that: include: Acquire information to be processed, where the information to be processed includes modal information and input information, where the modal information includes content of any one or more modalities other than text; Based on the multimodal model, determining output information corresponding to the information to be processed; Wherein, the multimodal model is generated by the method described in any one of claims 1-12.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in any one of claims 1 to 13.

Images