Dialogue processing method, text generation method, and target dialogue model training method

Abstract

Embodiments of the present disclosure provide a dialogue processing method, a text generation method, and a target dialogue model training method. The dialogue processing method comprises: acquiring dialogue text and dialogue association data, the modality of the dialogue association data being different from that of the dialogue text; inputting the dialogue text and the dialogue association data into a target dialogue model to obtain a first modal feature corresponding to the dialogue association data and a text modal feature corresponding to the dialogue text, the target dialogue model being obtained by performing dialogue task training on a pre-trained dialogue model on the basis of a first sample set, and the pre-trained dialogue model being obtained by performing, on the basis of a second sample set, text prediction task training on first modal parameters in a first coding unit and an adaptive unit in an initial dialogue model; using the first modal parameters to perform processing on the first modal feature, and using a text modal parameter to perform processing on the text modal feature to obtain a multi-modal feature; and, on the basis of the multi-modal feature, generating reply text corresponding to the dialogue text.

Description

Dialogue processing, text generation, and target dialogue model training methods

[0001] This disclosure claims priority to Chinese patent application number 202311456162.1, filed with the China Patent Office on November 2, 2023, and entitled “Dialogue Processing, Text Generation, and Target Dialogue Model Training Method,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The embodiments of the present disclosure relate to the field of computer technology, and in particular to methods for dialog processing, text generation, and target dialog model training. Background Art

[0003] With the development of computer technology, large models have begun to shine. Their remarkable capabilities in language understanding, generation, interaction, and reasoning have led to their widespread application in natural language processing, including dialogue, translation, and code generation. However, with the increasing number of data modalities generated by businesses and users, how to use large models to process multimodal data has gradually become a research focus.

[0004] Currently, before using a large model for conversation processing, data from different modalities is first encoded, and then the encoded data features are sent to the large model for conversation processing. However, in this solution, the large model does not directly process multimodal data, resulting in low accuracy in multimodal data processing. Therefore, a highly accurate conversation processing solution is urgently needed.

[0005] Summary of the Invention

[0006] In light of this, embodiments of the present disclosure provide a method for processing conversations. One or more embodiments of the present disclosure also include a text generation method, a target conversation model training method, a conversation processing device, a text generation device, a target conversation model training device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0007] According to a first aspect of an embodiment of the present disclosure, a method for processing a conversation is provided, including:

[0008] Obtaining a conversation text and conversation-related data, wherein the conversation-related data and the conversation text have different modalities;

[0009] Inputting the conversation text and conversation-related data into a target conversation model to obtain first modal features corresponding to the conversation-related data and text modal features corresponding to the conversation text, wherein the target conversation model is obtained by training a pre-trained conversation model for a conversation task based on the first sample set, and the pre-trained conversation model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model for a text prediction task based on the second sample set;

[0010] Processing the first modal feature using the first modal parameter and processing the text modal feature using the text modal parameter to obtain a multimodal feature;

[0011] Generate reply text corresponding to the conversation text based on multimodal features.

[0012] According to a second aspect of an embodiment of the present disclosure, a text generation method is provided, comprising:

[0013] Obtain the target image and the task text corresponding to the target image;

[0014] Inputting a target image and a task text into a target dialogue model to obtain image modal features corresponding to the target image and text modal features corresponding to the task text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for the dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial dialogue model for the text prediction task based on the second sample set;

[0015] Processing the image modality feature using the first modality parameter and processing the text modality feature using the text modality parameter to obtain a multimodal feature;

[0016] Generate target text corresponding to the target image based on multimodal features.

[0017] According to a third aspect of an embodiment of the present disclosure, a target dialogue model training method is provided, which is applied to a cloud-side device and includes:

[0018] Obtaining a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation-related data, the sample conversation texts and the sample conversation-related data are in one-to-one correspondence, and the sample conversation texts carry sample conversation tags;

[0019] Inputting a plurality of sample conversation texts and a plurality of sample conversation association data into a pre-trained conversation model to obtain first prediction modal features corresponding to the plurality of sample conversation association data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is trained on a text prediction task by performing a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on a second sample set;

[0020] Processing the first prediction modal feature using the first modal parameter and processing the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0021] Generate predicted response texts corresponding to multiple sample conversation texts based on multimodal prediction features;

[0022] Based on the predicted reply text and sample conversation labels, adjust the model parameters of the pre-trained conversation model to obtain the trained target conversation model.

[0023] According to a fourth aspect of an embodiment of the present disclosure, there is provided a dialog processing device, including:

[0024] A first acquisition module is configured to acquire a conversation text and conversation-related data, wherein the conversation-related data is in a different modality than the conversation text;

[0025] a first input module configured to input the conversation text and conversation-related data into a target conversation model, and obtain first modal features corresponding to the conversation-related data and text modal features corresponding to the conversation text, wherein the target conversation model is obtained by training a pre-trained conversation model for a conversation task based on the first sample set, and the pre-trained conversation model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model for a text prediction task based on the second sample set;

[0026] A first processing module is configured to process the first modal feature using the first modal parameter and to process the text modal feature using the text modal parameter to obtain a multimodal feature;

[0027] The first generation module is configured to generate a reply text corresponding to the dialogue text based on the multimodal features.

[0028] According to a fifth aspect of an embodiment of the present disclosure, there is provided a text generation device, comprising:

[0029] A second acquisition module is configured to acquire a target image and a task text corresponding to the target image;

[0030] a second input module configured to input a target image and a task text into a target dialogue model to obtain image modal features corresponding to the target image and text modal features corresponding to the task text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial dialogue model for a text prediction task based on the second sample set;

[0031] a second processing module configured to process the image modality feature using the first modality parameter and to process the text modality feature using the text modality parameter to obtain a multimodal feature;

[0032] The second generation module is configured to generate a target text corresponding to the target image based on the multimodal features.

[0033] According to a sixth aspect of an embodiment of the present disclosure, a target dialogue model training apparatus is provided, which is applied to a cloud-side device and includes:

[0034] a third acquisition module configured to acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond one to one, and the sample conversation texts carry sample conversation tags;

[0035] a third input module configured to input a plurality of sample conversation texts and a plurality of sample conversation association data into a pre-trained conversation model, and obtain first prediction modal features corresponding to the plurality of sample conversation association data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is trained on a text prediction task by performing text prediction on a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on a second sample set;

[0036] a third processing module configured to process the first prediction modal feature using the first modal parameter and to process the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0037] A third generation module is configured to generate predicted reply texts corresponding to the plurality of sample conversation texts according to the multimodal prediction features;

[0038] The adjustment module is configured to adjust the model parameters of the pre-trained dialogue model according to the predicted reply text and the sample dialogue label to obtain a trained target dialogue model.

[0039] According to a seventh aspect of an embodiment of the present disclosure, there is provided a computing device, including:

[0040] memory and processor;

[0041] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0042] According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the method provided in the first aspect, the second aspect or the third aspect are implemented.

[0043] According to a ninth aspect of an embodiment of the present disclosure, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the method provided in the first aspect, the second aspect, or the third aspect above.

[0044] An embodiment of the present disclosure provides a conversation processing method, which obtains conversation text and conversation-related data, wherein the conversation-related data and the conversation text have different modalities; inputs the conversation text and the conversation-related data into a target conversation model to obtain first modal features corresponding to the conversation-related data and text modal features corresponding to the conversation text, wherein the target conversation model is obtained by performing conversation task training on a pre-trained conversation model based on a first sample set, and the pre-trained conversation model is obtained by performing text prediction task training on a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on a second sample set; the first modal features are processed using the first modal parameters, and the text modal features are processed using the text modal parameters to obtain multimodal features; and based on the multimodal features, a reply text corresponding to the conversation text is generated.

[0045] By encoding conversation-related data and conversation text separately in the target conversation model and utilizing different modal parameters for feature processing, we can distinguish data from different modalities, alleviate the friction between these modalities, and improve modal collaboration, ensuring the accuracy of conversation processing. Furthermore, since the pre-trained conversation model is trained on text prediction tasks, it can perform modal conversion on the data to obtain data in textual modality. Furthermore, since the target conversation model is trained on conversation tasks, it can handle conversation tasks based on modal conversion, thus improving the accuracy of the target conversation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG1 is an architecture diagram of a conversation processing system provided by one embodiment of the present disclosure;

[0047] FIG2 is an architecture diagram of another dialog processing system provided by an embodiment of the present disclosure;

[0048] FIG3 is a flow chart of a conversation processing method provided by an embodiment of the present disclosure;

[0049] FIG4 is a structural diagram of an adaptive unit in a target dialogue model provided by an embodiment of the present disclosure;

[0050] FIG5 is a schematic diagram of training a target dialogue model provided by an embodiment of the present disclosure;

[0051] FIG6 is a schematic diagram of training a pre-trained dialogue model provided by one embodiment of the present disclosure;

[0052] FIG7 is a flowchart of a text generation method provided by an embodiment of the present disclosure;

[0053] FIG8 is a flowchart of a target dialogue model training method provided by one embodiment of the present disclosure;

[0054] FIG9 is a flowchart of a method for training a pre-trained dialogue model according to an embodiment of the present disclosure;

[0055] FIG10 is a structural diagram of a target dialogue model provided by an embodiment of the present disclosure;

[0056] FIG11 is a structural diagram of a conversation processing device provided by an embodiment of the present disclosure;

[0057] FIG12 is a schematic structural diagram of a text generation device provided by an embodiment of the present disclosure;

[0058] FIG13 is a schematic structural diagram of a target dialogue model training device provided by one embodiment of the present disclosure;

[0059] FIG14 is a structural block diagram of a computing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] The following description sets forth many specific details to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below.

[0061] The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present disclosure. The singular forms "a", "the", and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and includes any or all possible combinations of one or more associated listed items.

[0062] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0063] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0064] In one or more embodiments of the present disclosure, a large model refers to a deep learning model with large-scale model parameters, which typically contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. The large model is pre-trained using large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large-scale language model (LLM), a multimodal pre-training model, etc.

[0065] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0066] First, the terms involved in one or more embodiments of the present disclosure are explained.

[0067] Large-scale language models: Large-scale language models, also known as large language models or large models, are language models that can process large corpora. These models typically have a very high number of parameters and complexity, and are able to learn the deep structure and complex relationships of language, thereby enabling natural language generation and understanding.

[0068] Multimodal Large Language Model: The Multimodal Large Language Model (MLLM) can understand and generate natural language and can process multiple types of input data, including text, images, audio, etc.

[0069] With the popularity of large models, how to use large models to process multimodal data has gradually become a research focus. Currently, large models can be used to process multimodal data in the following two ways. The first is to use a three-stage training method to pre-train the large model, pre-train multi-tasks, and fine-tune instructions (I Tuning) to obtain a model that can process multimodal data. The second is to encode data of different modalities, and then send the encoded data features to the large model for dialogue processing. However, the amount of training data and the scale of the model are both large in the first solution. In the second solution, the large model does not directly process multimodal data, resulting in the large model's multimodal data processing accuracy still being low.

[0070] To address the above issues, the embodiments of the present disclosure propose a modular and lightweight multimodal model system (mPLUG, multi-modal Pre-training framework for both vision-Language Understanding and Generation) that combines multimodal representations with large-scale language models to unlock the multimodal capabilities of large-scale language models. The target dialogue model is obtained through two-stage training (pre-training and instruction fine-tuning), utilizing less data and a smaller model size, and performs very well on various benchmark indicators. In addition, the dialogue processing method proposed in the embodiments of the present disclosure can be widely used in scenarios such as dialogue assistants, multimodal chatbots, writing robots, travel guides, creative copywriting, user guides, exhibition guides, etc., promoting the development of academia.

[0071] Specifically, an embodiment of the present disclosure proposes a dialogue processing method, which obtains dialogue text and dialogue-related data, wherein the dialogue-related data is of a different modality from the dialogue text; inputs the dialogue text and the dialogue-related data into a target dialogue model to obtain first modal features corresponding to the dialogue-related data and text modal features corresponding to the dialogue text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on a first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial dialogue model for a text prediction task based on a second sample set; the first modal features are processed using the first modal parameters, and the text modal features are processed using the text modal parameters to obtain multimodal features; and based on the multimodal features, a reply text corresponding to the dialogue text is generated.

[0072] By encoding conversation-related data and conversation text separately in the target conversation model and utilizing different modal parameters for feature processing, we can distinguish data from different modalities, alleviate the friction between these modalities, and improve modal collaboration, ensuring the accuracy of conversation processing. Furthermore, since the pre-trained conversation model is trained on text prediction tasks, it can perform modal conversion on the data to obtain data in textual modality. Furthermore, since the target conversation model is trained on conversation tasks, it can handle conversation tasks based on modal conversion, thus improving the accuracy of the target conversation model.

[0073] In the present disclosure, a dialogue processing method is provided. The present disclosure also involves a text generation method, a target dialogue model training method, a dialogue processing device, a text generation device, a target dialogue model training device, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0074] Referring to FIG1 , FIG1 shows an architecture diagram of a conversation processing system provided by an embodiment of the present disclosure. The conversation processing system may include a client 100 and a server 200;

[0075] The client 100 is configured to send a conversation text and conversation-related data to the server 200, wherein the conversation-related data is in a different modality than the conversation text;

[0076] The server 200 is configured to input the conversation text and conversation-related data into a target conversation model, obtain first modal features corresponding to the conversation-related data, and text modal features corresponding to the conversation text, wherein the target conversation model is obtained by training a pre-trained conversation model for a conversation task based on a first sample set, and the pre-trained conversation model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model for a text prediction task based on a second sample set; process the first modal features using the first modal parameters, and process the text modal features using the text modal parameters to obtain multimodal features; generate a reply text corresponding to the conversation text based on the multimodal features; and send the reply text to the client 100.

[0077] The client 100 is also used to receive the reply text sent by the server 200.

[0078] By applying the solution of the disclosed embodiments, conversation-related data and conversation text are separately encoded in the target conversation model, and feature processing is performed using different modal parameters. This allows for distinguishing data of different modalities, alleviating the friction between different modal data, while improving modal coordination and ensuring the accuracy of conversation processing. Furthermore, because the pre-trained conversation model is trained through text prediction tasks, it can perform modal conversion on the data to obtain data in text modality. Furthermore, because the target conversation model is trained through conversation tasks, it can handle conversation tasks based on modal conversion, thereby improving the accuracy of the target conversation model.

[0079] Referring to Figure 2, it shows an architectural diagram of another conversation processing system provided by one embodiment of the present disclosure. The conversation processing system may include multiple clients 100 and a server 200. The clients 100 may include end-side devices, and the server 200 may include cloud-side devices. Multiple clients 100 can establish communication connections through the server 200. In a conversation processing scenario, the server 200 provides conversation processing services between the multiple clients 100. The multiple clients 100 can act as either senders or receivers, communicating through the server 200.

[0080] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100. In a dialogue processing scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 can generate a reply text based on the data stream and push the reply text to other clients that have established communication.

[0081] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, or other processing before being released to the server 200.

[0082] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be based on the software development kit (SDK) of the corresponding service provided by the server 200, such as developed based on the real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0083] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server that is integrated with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0084] It is worth noting that the conversation processing methods provided in the embodiments of the present disclosure are generally executed by the server. However, in other embodiments of the present disclosure, the client may also have similar functions to the server and thus execute the conversation processing methods provided in the embodiments of the present disclosure. In other embodiments, the conversation processing methods provided in the embodiments of the present disclosure may also be executed jointly by the client and the server.

[0085] 3 , which shows a flow chart of a conversation processing method provided by an embodiment of the present disclosure, specifically comprising the following steps:

[0086] Step 302: Acquire the conversation text and conversation-related data, wherein the conversation-related data and the conversation text have different modalities.

[0087] In one or more embodiments of the present disclosure, a conversation text and conversation-related data may be obtained to perform conversation processing based on the conversation text and the conversation-related data to obtain a reply result corresponding to the conversation text.

[0088] Specifically, conversation-related data refers to data whose modality differs from that of text. Conversation-related data can be at least one of the following: conversation image data, conversation audio data, and conversation video data. Conversation text is used to express the purpose of the conversation. For example, conversation-related data could be a landscape image, and the conversation text could be, "Please answer the question: What season is the image?" Conversation-related data and conversation text can encompass a variety of scenarios, including but not limited to conferences, transportation navigation, and educational settings.

[0089] In practical applications, there are various ways to obtain conversation text and conversation-related data, and the method of selection depends on the actual situation. This disclosure does not impose any restrictions on this method. In one possible implementation of this disclosure, the user can receive the conversation text and conversation-related data sent by the client. In another possible implementation of this disclosure, the conversation text and conversation-related data can be read from other data acquisition devices or databases.

[0090] Step 304: Input the conversation text and conversation-related data into the target conversation model to obtain first modal features corresponding to the conversation-related data and text modal features corresponding to the conversation text, wherein the target conversation model is obtained by performing conversation task training on a pre-trained conversation model based on the first sample set, and the pre-trained conversation model is obtained by performing text prediction task training on a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on the second sample set.

[0091] Specifically, the target dialogue model is a natural language processing model. The target dialogue model is a two-stage training of the initial dialogue model: a pre-training stage and an instruction fine-tuning stage. During the pre-training stage, only the encoding parameters of the first encoding unit and the first modal parameters of the adaptive unit are trained. During the instruction fine-tuning stage, all model parameters are fine-tuned. Text modal features are valid feature representations of dialogue text. First modal features are valid feature representations of dialogue-related data. If the dialogue-related data is dialogue image data, the first modal features are image modal features, also known as visual features. If the dialogue-related data is dialogue audio data, the first modal features are audio modal features, also known as auditory features.

[0092] Step 306: Process the first modal feature using the first modal parameter, and process the text modal feature using the text modal parameter to obtain a multimodal feature.

[0093] Specifically, the first modal parameter is used to process the first modal feature, and the text modal parameter is used to process the text modal feature.

[0094] It should be noted that there are various ways to input the conversation text and conversation-related data into the target conversation model and obtain the first modal features corresponding to the conversation-related data and the text modal features corresponding to the conversation text. The method of selection depends on the specific circumstances and is not limited in this embodiment. In one possible implementation of the present disclosure, the conversation text and conversation-related data can be input into the target conversation model, and the target conversation model can use the same encoding unit to encode the conversation text and conversation-related data. In another possible implementation of the present disclosure, based on the different modalities of the conversation-related data and the conversation text data, the target conversation model can differentiate between different modalities using different encoding units to process data of different modalities.

[0095] In practical applications, the first modal feature is processed using the first modal parameter, and the text modal feature is processed using the text modal parameter. There are many ways to obtain multimodal features, and the specific selection is made according to the actual situation. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, the preset first modal parameter and text modal parameter can be obtained, and the first modal feature is directly processed using the first modal parameter, and the text modal feature is processed using the text modal parameter to obtain multimodal features. In another possible implementation of the present disclosure, an adaptive unit can be set in the target dialogue model, and the adaptive unit includes the first modal parameter and the text modal parameter, and the first modal parameter and the text modal parameter are updated as the target dialogue model is trained.

[0096] In an optional embodiment of the present disclosure, the target dialogue model includes a first encoding unit, a second encoding unit, and an adaptive unit, the adaptive unit including a first modal parameter and a text modal parameter. Inputting the dialogue text and dialogue-related data into the target dialogue model to obtain the first modal features corresponding to the dialogue-related data and the text modal features corresponding to the dialogue text may include the following steps:

[0097] Inputting the conversation-related data into a first encoding unit to obtain a first modal feature corresponding to the conversation-related data, and inputting the conversation text into a second encoding unit to obtain a text modal feature corresponding to the conversation text;

[0098] Processing the first modal feature using the first modal parameter and processing the text modal feature using the text modal parameter to obtain the multimodal feature may include the following steps:

[0099] The first modal feature and the text modal feature are input into the adaptive unit, the first modal feature is processed using the first modal parameter, and the text modal feature is processed using the text modal parameter to obtain a multimodal feature.

[0100] Specifically, the first encoding unit is used to encode the conversation-related data. The second encoding unit is used to encode the conversation text. The Modality-Adaptive Module can use different modality parameters to encode and map features of different modalities.

[0101] By applying the solution of the embodiment of the present disclosure, different encoding units are used to process data of different modalities, thereby ensuring the accuracy of the first modal features and the text modal features. In addition, an adaptive unit is set in the target dialogue model so that the first modal parameters and the text modal parameters can be trained and updated, thereby ensuring the accuracy of the first modal parameters and the text modal parameters.

[0102] In actual applications, the conversation-related data is input into the first encoding unit to obtain the first modal feature corresponding to the conversation-related data, and the conversation text is input into the second encoding unit. There are many ways to obtain the text modal feature corresponding to the conversation text, and the specific selection is made according to the actual situation. The embodiment of the present disclosure does not impose any restrictions on this.

[0103] In a possible implementation of the present disclosure, the first encoding unit may be used to directly encode the conversation-related data to obtain the first modal feature, and the second encoding unit may be used to encode the conversation text to obtain the text modal feature.

[0104] In another possible implementation of the present disclosure, since the length of the first modal feature may be relatively long, in order to improve the processing effect of the adaptive unit in the target dialogue model, a first summary subunit may be additionally introduced into the first encoding unit to reduce the length of the first modal feature. That is, the first encoding unit includes the first encoding subunit and the first summary subunit. The aforementioned step of inputting the dialogue-related data into the first encoding unit to obtain the first modal feature corresponding to the dialogue-related data may include the following steps:

[0105] Inputting the conversation-related data into the first encoding subunit to obtain initial modal features corresponding to the conversation-related data;

[0106] The initial modal features are input into the first summary subunit, and the target query vector is used to perform feature screening on the initial modal features to obtain the first modal features corresponding to the conversation-related data.

[0107] It should be noted that the first encoding subunit is used to convert the input conversation-related data into a series of vector representations, namely, initial modal features, which contain all the information of the conversation-related data.

[0108] Furthermore, when the first summary subunit is used to process the initial modal features, a learnable target query vector (Learnable Queries) can be used to perform feature screening and summary on the initial modal features to obtain the first modal features.

[0109] For example, taking the conversation-related data as conversation image data, the first encoding subunit is a visual encoder, and the first abstract subunit is a visual abstractor. The first encoding subunit and the first abstract subunit perform feature extraction on the conversation image data, and use a learnable target query vector to perform feature screening and summary on the visual information, thereby obtaining image modal features.

[0110] By applying the solution of the embodiment of the present disclosure, the conversation-related data is input into the first encoding subunit to obtain the initial modal features corresponding to the conversation-related data; the initial modal features are input into the first summary subunit, and the initial modal features are subjected to feature screening using the target query vector to obtain the first modal features corresponding to the conversation-related data. The first summary subunit and the learnable target query vector reduce the sequence length of the first modal features, thereby improving the processing efficiency of the adaptive unit.

[0111] In an optional embodiment of the present disclosure, in the adaptive unit, different normalization layers (Norm) can also be used to perform feature normalization on features of different modalities, that is, the adaptive unit includes a first normalization layer, a self-attention layer, and a second normalization layer, the first normalization layer includes a first modal parameter and a text modal parameter, and the second normalization layer includes a first modal parameter and a text modal parameter; the above-mentioned inputting the first modal feature and the text modal feature into the adaptive unit, processing the first modal feature using the first modal parameter, and processing the text modal feature using the text modal parameter to obtain a multimodal feature, can include the following steps:

[0112] The modal identifier, the first modal feature and the text modal feature are sequentially input into the first normalization layer, the self-attention layer and the second normalization layer to obtain multimodal features, wherein the modal identifier is used to identify the first modal feature and the text modal feature respectively.

[0113] It should be noted that, after the first normalization layer, the first modal feature can be normalized using the modal identifier and the first modal parameter, and the text modal feature can be normalized using the modal identifier and the text modal parameter to obtain the first attention parameter and the text attention parameter; after the self-attention layer, the first attention parameter and the text attention parameter can be attention-processed to obtain the self-attention feature; after the second normalization layer, the first modal feature and the self-attention feature can be normalized using the modal identifier and the first modal parameter, and the text modal feature and the self-attention feature can be normalized using the modal identifier and the text modal parameter to obtain multimodal features.

[0114] Specifically, the modality indicator is used to identify feature data of different modalities. For example, a feature with a subscript of 0 is a text modal feature, and a feature with a subscript of 1 is a first modal feature. Similarly, modal parameters in the adaptive unit can also be distinguished using the modality indicator. For example, a parameter with a subscript of 0 is a text modal parameter, and a parameter with a subscript of 1 is a first modal parameter.

[0115] Refer to Figure 4, which shows a structural diagram of an adaptive unit in a target dialogue model provided by an embodiment of the present disclosure. As shown in Figure 4, the adaptive unit includes a first normalization layer, a self-attention layer and a second normalization layer. The first normalization layer includes Norm0 and Norm1; the self-attention layer includes a matrix multiplication and scaling processing layer (MatMul&Scale), a normalization layer (Softmax), a matrix multiplication processing layer and a linear layer (Linear); the second normalization layer includes Norm0 and Norm1, wherein Norm0 in the first normalization layer is used to normalize the input text modal features using the modal identifier and text modal parameters to obtain text attention parameters, and Norm1 is used to normalize the input first modal features using the modal identifier and the first modal parameters to obtain first attention parameters. Furthermore, the first query vector (Q1), first value vector (V1), and first key vector (K1) included in the first attention parameter and the text attention parameter, as well as the text query vector (Q0), text value vector (V0), and text key vector (K0) included in the text attention parameter, are input into the self-attention layer. At the same time, the weights WQ corresponding to Q1 and Q0, the weight WV1 corresponding to V1, the weight WV0 corresponding to V0, the weight WK1 corresponding to K1, and the weight WK0 corresponding to K0 are also input into the self-attention layer. The self-attention layer performs attention processing on the first attention parameter and the text attention parameter to obtain self-attention features. Finally, the first modality features, the text modality features, and the self-attention features are added and input into the second normalization layer. After normalization processing by the second normalization layer, multimodal features are obtained, where multimodal features can be referred to as cross-modality features.

[0116] By applying the solution of the embodiment of the present disclosure, different modal parameters are set in the adaptive unit according to the characteristics of different modalities, thereby realizing modular differentiation of the characteristics of different modalities. The idea of ​​modal collaboration in which modalities can help each other is adopted. According to the characteristics of different modalities, the data processing of different modalities is modularized and modeled with different parameters, thereby alleviating the problem of modal pull and improving modal collaboration.

[0117] Step 308: Generate a reply text corresponding to the conversation text based on the multimodal features.

[0118] In one or more embodiments of the present disclosure, a conversation text and conversation-related data are obtained, wherein the conversation-related data and the conversation text have different modalities; the conversation text and the conversation-related data are input into a target conversation model to obtain a first modal feature corresponding to the conversation-related data and a text modal feature corresponding to the conversation text; the first modal feature is processed using a first modal parameter, and the text modal feature is processed using a text modal parameter. After obtaining multimodal features, a reply text corresponding to the conversation text can be further generated based on the multimodal features.

[0119] For example, assuming the dialogue text is “Please answer what season is in the picture”, the corresponding reply text of the dialogue text is “It is winter in the picture”.

[0120] By applying the solution of the disclosed embodiments, conversation-related data and conversation text are separately encoded in the target conversation model, and feature processing is performed using different modal parameters. This allows for distinguishing data of different modalities, alleviating the friction between different modal data, while improving modal coordination and ensuring the accuracy of conversation processing. Furthermore, because the pre-trained conversation model is trained through text prediction tasks, it can perform modal conversion on the data to obtain data in text modality. Furthermore, because the target conversation model is trained through conversation tasks, it can handle conversation tasks based on modal conversion, thereby improving the accuracy of the target conversation model.

[0121] In an optional embodiment of the present disclosure, the target dialogue model includes a shared feedforward unit; and generating a reply text corresponding to the dialogue text based on the multimodal features may include the following steps:

[0122] The multimodal features are input into the shared feedforward unit, the multimodal features are decoded, and the reply text corresponding to the dialogue text is generated.

[0123] Specifically, in a feedforward neural network (FFN), each node has independent weights and biases, which means a large number of parameters are required to train the network. To reduce model complexity, the disclosed embodiments propose a shared feedforward unit (FFN). All nodes in the same stage can share the same FFN, requiring only a single weight and bias matrix to complete the calculation, significantly reducing the number of parameters required for the model.

[0124] In practical applications, a shared feedforward unit consists of an input layer, a first hidden layer, an activation layer, a second hidden layer, and an output layer. The input layer receives the output from the previous layer and passes it to the next node. The first hidden layer is a linear transformation layer that fully connects the input layer using shared weights and bias matrices. The activation layer includes an activation function that converts the output of the hidden layer into a nonlinear activation signal. The second hidden layer is also a linear transformation layer that fully connects the output of the first hidden layer using shared weights and bias matrices. The output layer scales and translates the output of the second hidden layer, then returns it to the previous layer to generate the response text corresponding to the conversation text.

[0125] Using the solution of the disclosed embodiments, multimodal features are fed into a shared feedforward unit, which decodes them and generates a corresponding response text. This shared feedforward unit efficiently extracts multimodal features and converts them into a higher-level abstract representation. Furthermore, by sharing weights and bias matrices, model complexity can be significantly reduced and accuracy improved.

[0126] In an optional embodiment of the present disclosure, after generating a reply text corresponding to the conversation text based on the multimodal features, the following steps may also be included:

[0127] Sending a reply text to the client, so that the client displays the reply text to the user based on the display requirement information;

[0128] Receive adjustment information sent by the user based on the reply text, and adjust parameters of the target dialogue model according to the adjustment information.

[0129] In actual applications, there are many ways to send a reply text to the client, and the specific selection is based on the actual situation. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, the reply text can be sent directly to the client. In another possible implementation of the present disclosure, the reply text can be sent to the client based on the user's display demand information. Among them. The display demand information represents the user's demand for viewing the reply text. The display demand information includes but is not limited to displaying only the reply text, displaying the conversation-related data and the reply text and identifying and distinguishing the two, displaying multiple reply texts for the user to choose, and displaying the link corresponding to the reply text. The display demand information is specifically set according to the actual needs of the user, and the embodiments of the present disclosure do not impose any restrictions on this.

[0130] Furthermore, after receiving a reply text, if the user is dissatisfied with the reply, they can send adjustment information based on the reply text to retrain the target dialogue model and proceed with the second dialogue. The implementation method for "adjusting the parameters of the target dialogue model based on the adjustment information" is similar to the training method of the target dialogue model and will not be further described in this embodiment.

[0131] By applying the solution of the embodiment of the present disclosure, a reply text is sent to the client, so that the client displays the reply text to the user based on the display requirement information; adjustment information sent by the user based on the reply text is received, and parameters of the target dialogue model are adjusted according to the adjustment information, thereby increasing interaction with the user and improving user satisfaction.

[0132] In an optional embodiment of the present disclosure, before inputting the dialogue text and dialogue-related data into the target dialogue model, the following steps may be further included:

[0133] Obtaining a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation-related data, the sample conversation texts and the sample conversation-related data are in one-to-one correspondence, and the sample conversation texts carry sample conversation tags;

[0134] Inputting a plurality of sample conversation texts and a plurality of sample conversation association data into a pre-trained conversation model to obtain first prediction modal features corresponding to the plurality of sample conversation association data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is trained on a text prediction task by performing a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on a second sample set;

[0135] Processing the first prediction modal feature using the first modal parameter and processing the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0136] Generate predicted response texts corresponding to multiple sample conversation texts based on multimodal prediction features;

[0137] Based on the predicted reply text and sample conversation labels, adjust the model parameters of the pre-trained conversation model to obtain the trained target conversation model.

[0138] Specifically, the first sample set can be referred to as a text instruction sample set, the multiple sample conversation texts in the first sample set can be referred to as a text instruction sample set, and the multiple sample conversation-associated data can be referred to as a multimodal instruction sample set. The initial conversation model can be an untrained natural language processing model or a model directly loaded with parameters from a large-scale language model. The pre-trained conversation model is trained using supervised training, meaning that the multiple sample conversation texts included in the first sample set carry real-world sample conversation labels. These sample conversation labels serve as generation targets for the pre-trained conversation model and guide the model training process.

[0139] Referring to FIG5 , FIG5 shows a training diagram of a target dialogue model provided by an embodiment of the present disclosure. During the training process of the target dialogue model, the pre-trained dialogue model can be fine-tuned using a text instruction sample set and a multimodal instruction sample set, thereby achieving the effect of modal collaboration. That is, multiple sample dialogue texts and multiple sample dialogue-related data are input into the first encoding subunit and the first summary subunit, and the sample dialogue labels carried by the sample dialogue texts are input into the decoding unit for instruction fine-tuning to obtain the trained target dialogue model.

[0140] In practical applications, there are multiple ways to obtain the first sample set, and the method to be used depends on the actual situation. The embodiments of this disclosure do not impose any restrictions on this. In one possible implementation of this disclosure, the first sample set can be read from other data acquisition devices or databases. In another possible implementation of this disclosure, the first sample set can be received from a user.

[0141] It should be noted that the implementation method of "inputting multiple sample conversation texts and multiple sample conversation association data into the pre-trained conversation model to obtain first prediction modal features corresponding to the multiple sample conversation association data and text prediction modal features corresponding to the multiple sample conversation texts" is the same as the implementation method of the above-mentioned step 304; the implementation method of "processing the first prediction modal features using the first modal parameters and processing the text prediction modal features using the text modal parameters to obtain multimodal prediction features" is the same as the implementation method of the above-mentioned step 306; the implementation method of "generating predicted reply texts corresponding to the multiple sample conversation texts according to the multimodal prediction features" is the same as the implementation method of the above-mentioned step 308, and the present disclosed embodiment will not repeat them again.

[0142] In actual applications, when adjusting the model parameters of the pre-trained dialogue model based on the predicted reply text and sample dialogue labels, a first loss value can be calculated based on the predicted reply text and sample dialogue labels, and the model parameters of the pre-trained dialogue model can be adjusted based on the first loss value until the first preset stopping condition is reached, thereby obtaining a trained target dialogue model.

[0143] It should be noted that there are many functions for calculating the first loss value, such as cross entropy loss function, L1 norm loss function, maximum loss function, mean square error loss function, logarithmic loss function, etc. The specific selection is based on the actual situation, and the embodiments of this disclosure do not impose any restrictions on this.

[0144] In one possible implementation of the present disclosure, the first preset stopping condition includes a first loss value being less than or equal to a first preset threshold value. After calculating the first loss value based on the predicted reply text and the sample conversation tag, the first loss value is compared with the first preset threshold value.

[0145] Specifically, if the first loss value is greater than the first preset threshold, it means that the difference between the predicted reply text and the sample conversation label is large, and the pre-trained conversation model has poor prediction ability for the reply text. At this time, the model parameters of the pre-trained conversation model can be adjusted, and the pre-trained conversation model can continue to be trained until the first loss value is less than or equal to the first preset threshold, indicating that the difference between the predicted reply text and the sample conversation label is small, and the first preset stop condition is met, and the target conversation model that has completed the training is obtained.

[0146] In another possible implementation of the present disclosure, in addition to comparing the first loss value and the first preset threshold, the number of iterations may be combined to determine whether the current pre-trained dialogue model is trained.

[0147] Specifically, if the first loss value is greater than a first preset threshold, the model parameters of the pre-trained dialogue model are adjusted, and the pre-trained dialogue model is continued to be trained until a first preset number of iterations is reached, and the iteration is stopped to obtain a target dialogue model that has completed the training. The first preset threshold and the first preset number of iterations are selected according to actual conditions, and the embodiments of the present disclosure do not impose any limitations on this.

[0148] Using the solution of the disclosed embodiments, the model parameters of the pre-trained dialogue model are adjusted based on the predicted response text and sample conversation labels. If the first preset stopping condition is not met, the pre-trained dialogue model is trained continuously until the first preset stopping condition is met, completing the training and obtaining the target dialogue model. By continuously adjusting the model parameters of the pre-trained dialogue model, the resulting target dialogue model can be made more accurate.

[0149] In an optional embodiment of the present disclosure, the pre-trained dialogue model includes a first encoding unit and a second encoding unit, the first encoding unit including a first encoding sub-unit and a first summarization sub-unit; the above-mentioned inputting a plurality of sample dialogue texts and a plurality of sample dialogue association data into the pre-trained dialogue model to obtain first prediction modal features corresponding to the plurality of sample dialogue association data and text prediction modal features corresponding to the plurality of sample dialogue texts may include the following steps:

[0150] Inputting the plurality of sample conversation association data into the first encoding subunit to obtain initial predicted modal features corresponding to the plurality of sample conversation association data respectively;

[0151] Inputting the initial predicted modal features into the first summary subunit, and using the target query vector to perform feature screening on the initial predicted modal features to obtain first predicted modal features corresponding to the plurality of sample conversation association data;

[0152] The plurality of sample dialogue texts are input into the second encoding unit to obtain text prediction modal features corresponding to the plurality of sample dialogue texts respectively.

[0153] It should be noted that the implementation method of the above-mentioned "inputting multiple sample conversation association data into the first encoding subunit to obtain initial predicted modal features corresponding to the multiple sample conversation association data respectively; inputting the initial predicted modal features into the first summary subunit, using the target query vector to perform feature screening on the initial predicted modal features, and obtaining first predicted modal features corresponding to the multiple sample conversation association data respectively" is the same as the implementation method of the above-mentioned "inputting conversation association data into the first encoding subunit to obtain initial modal features corresponding to the conversation association data; inputting the initial modal features into the first summary subunit, using the target query vector to perform feature screening on the initial modal features, and obtaining first modal features corresponding to the conversation association data", and the present disclosed embodiment will not repeat this.

[0154] By applying the solution of the embodiment of the present disclosure, the sequence length of the first predicted modal feature is reduced through the first summary subunit and the learnable target query vector, thereby improving the processing efficiency of the adaptive unit.

[0155] In an optional embodiment of the present disclosure, before inputting the plurality of sample conversation texts and the plurality of sample conversation associated data into the pre-trained conversation model, the following steps may also be included:

[0156] Acquire a second sample set, wherein the second sample set includes a plurality of sample-related data, and the sample-related data carries a sample text label;

[0157] Inputting a plurality of sample association data into the initial dialogue model, and obtaining second prediction modal features corresponding to the plurality of sample association data respectively through the first encoding unit;

[0158] The adaptive unit of the initial dialogue model processes the second predicted modal features using the first modal parameters to obtain target predicted features;

[0159] Generate prediction texts corresponding to multiple sample association data according to the target prediction features;

[0160] According to the predicted text and the sample text label, the first modal parameter and the encoding parameter of the first encoding unit are adjusted to obtain a pre-trained dialogue model that has been trained.

[0161] Specifically, the second sample set can be called a modality-aligned sample set. For example, if the sample-related data is sample image data, the training process of the initial dialogue model using the second sample set can be understood as an image-text alignment process. The initial dialogue model is trained using supervised training, meaning that the multiple sample-related data included in the second sample set carry real sample text labels. These sample text labels serve as the generation targets for the initial dialogue model and guide the model training process.

[0162] Refer to Figure 6, which shows a training diagram of a pre-trained dialogue model provided by an embodiment of the present disclosure. During the training process of the pre-trained dialogue model, multiple sample-related data are input into the first encoding subunit and the first summary subunit, and the sample text labels carried by the sample-related data are input into the decoding unit for partial parameter adjustment, that is, the first encoding subunit and the first summary subunit are trained, and the first modal parameters in the adaptive unit are trained at the same time, and the encoding parameters of the second encoding unit and the text modal parameters in the adaptive unit are kept unchanged during the training process.

[0163] In practical applications, the method for obtaining the first sample set may refer to the method for obtaining the second sample set described above. The embodiment of the present disclosure does not impose any limitation on the method for obtaining the first sample set.

[0164] It should be noted that the implementation method of "inputting multiple sample association data into the initial dialogue model, and obtaining the second prediction modal features corresponding to the multiple sample association data respectively through the first encoding unit" is the same as the implementation method of "inputting the dialogue association data into the first encoding unit, and obtaining the first modal features corresponding to the dialogue association data"; the implementation method of "processing the second prediction modal features using the first modal parameters through the adaptive unit of the initial dialogue model to obtain the target prediction features" is the same as the implementation method of "inputting the first modal features and text modal features into the adaptive unit, processing the first modal features using the first modal parameters, and processing the text modal features using the text modal parameters" The implementation method of the above-mentioned "processing the features to obtain multimodal features" is the same as the implementation method of the above-mentioned "generating predicted texts corresponding to multiple sample associated data according to the target prediction features" is the same as the implementation method of the above-mentioned "generating reply texts corresponding to the dialogue text according to the multimodal features"; the implementation method of "adjusting the first modal parameters and the encoding parameters of the first encoding unit according to the predicted text and the sample text label to obtain a trained pre-trained dialogue model" is the same as the implementation method of the above-mentioned "adjusting the model parameters of the pre-trained dialogue model according to the predicted reply text and the sample dialogue label to obtain a trained target dialogue model", which will not be repeated in the present embodiment.

[0165] Using the solution of the embodiments of the present disclosure, the first modal parameters of the initial dialogue model and the encoding parameters of the first encoding unit are adjusted based on the predicted text and sample text labels. If a second preset stopping condition is not met, the initial dialogue model is trained continuously until the second preset stopping condition is met, completing the training and obtaining a pre-trained dialogue model. By continuously adjusting the first modal parameters of the initial dialogue model and the encoding parameters of the first encoding unit, the resulting pre-trained dialogue model can be made more accurate.

[0166] In an optional embodiment of the present disclosure, the initial dialogue model further includes a second encoding unit, which includes text encoding parameters. Before inputting the plurality of sample-related data into the initial dialogue model and obtaining the second predicted modal features corresponding to the plurality of sample-related data respectively through the first encoding unit, the following steps may also be included:

[0167] Based on the text encoding parameters, encoding parameters of the first encoding unit are set.

[0168] In practical applications, there are multiple ways to set the encoding parameters of the first encoding unit based on the text encoding parameters, and the specific selection is based on the actual situation. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, the text encoding parameters can be copied, and the copied text encoding parameters are set as the encoding parameters of the first encoding unit. In another possible implementation of the present disclosure, the text encoding parameters can be copied, the copied text encoding parameters are adjusted, and the adjusted text encoding parameters are set as the encoding parameters of the first encoding unit, wherein the adjustment method includes but is not limited to increasing the preset parameters and decreasing the preset parameters.

[0169] By applying the solution of the embodiment of the present disclosure, the encoding parameters of the first encoding unit are set based on the text encoding parameters, so that the encoding parameters of the first encoding unit can be trained separately, thereby setting different encoding units for different modal data, thereby alleviating the problem of modal pull and improving modal collaboration.

[0170] In an optional embodiment of the present disclosure, the adaptive unit includes text modal parameters; and the adaptive unit of the initial dialogue model may further include the following steps before processing the second predicted modal features using the first modal parameters to obtain the target predicted features:

[0171] Based on the text modal parameter, a first modal parameter in the adaptive unit is set.

[0172] In practical applications, there are multiple ways to set the first modal parameter in the adaptive unit based on the text modal parameter, and the specific selection is made according to the actual situation. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, the text modal parameter can be copied, and the copied text modal parameter is set as the first modal parameter in the adaptive unit. In another possible implementation of the present disclosure, the text modal parameter can be copied, the copied text modal parameter is adjusted, and the adjusted text modal parameter is set as the first modal parameter in the adaptive unit, wherein the adjustment method includes but is not limited to increasing the preset parameter and reducing the preset parameter.

[0173] By applying the solution of the embodiment of the present disclosure, the first modal parameter in the adaptive unit is set based on the text modal parameter, so that the first modal parameter in the adaptive unit can be trained separately, thereby setting different modal parameters for different modal data, thereby alleviating the problem of modal pulling and improving modal collaboration.

[0174] The following further illustrates the dialog processing method provided by the present disclosure using the application of the dialog processing method in a text generation scenario as an example, in conjunction with FIG7 . FIG7 shows a flowchart of a text generation method provided by an embodiment of the present disclosure, which specifically includes the following steps:

[0175] Step 702: Obtain a target image and a task text corresponding to the target image.

[0176] Step 704: Input the target image and the task text into the target dialogue model to obtain image modal features corresponding to the target image and text modal features corresponding to the task text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for the dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in the initial dialogue model for the text prediction task based on the second sample set.

[0177] Step 706: Process the image modal feature using the first modal parameter and process the text modal feature using the text modal parameter to obtain a multimodal feature.

[0178] Step 708: Generate target text corresponding to the target image based on the multimodal features.

[0179] It should be noted that the implementation of steps 702 to 708 is the same as the implementation of steps 302 to 308 described above, and will not be described in detail in this embodiment of the present disclosure.

[0180] For example, assuming that the target image is a girl standing happily on the podium, the task text corresponding to the target image is "Why is the girl happy?", and the target text generated by the above text generation method is "The girl is happy because she is standing on the podium."

[0181] By applying the solution of the disclosed embodiments, the target image and the corresponding task text are separately encoded in the target dialogue model, and feature processing is performed using different modal parameters. This allows for distinguishing data of different modalities, alleviating the friction between data of different modalities, while improving modal coordination and ensuring the accuracy of text generation. Furthermore, because the pre-trained dialogue model is trained on a text prediction task, the pre-trained dialogue model can perform modal conversion on the data to obtain data in text modality. Furthermore, because the target dialogue model is trained on a dialogue task, it can handle dialogue tasks based on modal conversion, thereby improving the accuracy of the target dialogue model.

[0182] 8 , which shows a flow chart of a target dialogue model training method provided by one embodiment of the present disclosure. The target dialogue model training method is applied to a cloud-side device and specifically includes the following steps:

[0183] Step 802: Obtain a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data are in one-to-one correspondence, and the sample conversation texts carry sample conversation tags.

[0184] Step 804: Input the plurality of sample conversation texts and the plurality of sample conversation association data into a pre-trained conversation model to obtain first prediction modal features corresponding to the plurality of sample conversation association data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is obtained by training the first encoding unit and the first modal parameters in the adaptive unit in the initial conversation model for a text prediction task based on the second sample set.

[0185] Step 806: Process the first prediction modal feature using the first modal parameter, and process the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature.

[0186] Step 808: Generate predicted reply texts corresponding to multiple sample conversation texts based on the multimodal prediction features.

[0187] Step 810: Adjust the model parameters of the pre-trained dialogue model based on the predicted reply text and the sample dialogue label to obtain a trained target dialogue model.

[0188] It should be noted that the implementation method of steps 802 to 810 can refer to the training method of the target dialogue model in the above-mentioned dialogue processing method, and will not be described in detail in this embodiment of the present disclosure.

[0189] In actual applications, after the cloud-side device obtains the trained target dialogue model, it can send the model parameters of the target dialogue model to the terminal-side device, so that the user can locally build the target dialogue model based on the model parameters of the target dialogue model and further perform dialogue processing tasks.

[0190] Using the solution of the disclosed embodiments, the model parameters of the pre-trained dialogue model are adjusted based on the predicted response text and sample conversation labels. If the first preset stopping condition is not met, the pre-trained dialogue model is trained continuously until the first preset stopping condition is met, completing the training and obtaining the target dialogue model. By continuously adjusting the model parameters of the pre-trained dialogue model, the resulting target dialogue model can be made more accurate.

[0191] 9 , which shows a flow chart of a pre-trained dialogue model training method provided by one embodiment of the present disclosure. The pre-trained dialogue model training method is applied to a cloud-side device and specifically includes the following steps:

[0192] Step 902: Acquire a second sample set, wherein the second sample set includes a plurality of sample-related data, and the sample-related data carries sample text labels.

[0193] Step 904: Input the plurality of sample association data into the initial dialogue model, and obtain second predicted modal features corresponding to the plurality of sample association data respectively through the first encoding unit.

[0194] Step 906: The adaptive unit of the initial dialogue model processes the second predicted modal features using the first modal parameters to obtain target predicted features.

[0195] Step 908: Generate predicted texts corresponding to the plurality of sample-related data respectively according to the target prediction features.

[0196] Step 910: According to the predicted text and the sample text label, adjust the first modal parameters and the encoding parameters of the first encoding unit to obtain a trained pre-trained dialogue model.

[0197] It should be noted that the implementation method of steps 902 to 910 can refer to the training method of the pre-trained dialogue model in the above-mentioned dialogue processing method, and will not be described in detail in the embodiment of the present disclosure.

[0198] In actual applications, after the cloud-side device obtains the trained pre-trained dialogue model, it can send the model parameters of the pre-trained dialogue model to the terminal-side device, so that the user can build a pre-trained dialogue model locally based on the model parameters of the pre-trained dialogue model and further perform dialogue processing tasks.

[0199] Using the solution of the embodiments of the present disclosure, the first modal parameters of the initial dialogue model and the encoding parameters of the first encoding unit are adjusted based on the predicted text and sample text labels. If a second preset stopping condition is not met, the initial dialogue model is trained continuously until the second preset stopping condition is met, completing the training and obtaining a pre-trained dialogue model. By continuously adjusting the first modal parameters of the initial dialogue model and the encoding parameters of the first encoding unit, the resulting pre-trained dialogue model can be made more accurate.

[0200] Referring to FIG10 , FIG10 shows a structural diagram of a target dialogue model provided by one embodiment of the present disclosure. The target dialogue model includes a first encoding unit, a second encoding unit, and a decoding unit. The first encoding unit includes a first encoding subunit and a first summarization subunit. The decoding unit includes an adaptation unit and a shared feedforward unit.

[0201] As shown in Figure 10, the conversation association data is input into the first encoding subunit to obtain the initial modal features; the initial modal features and the target query vector are input into the first summary subunit to obtain the first modal features; the conversation text is input into the second encoding unit to obtain the text modal features; the first modal features and the text modal features are input into the adaptive unit, and the first modal features are processed using the first modal parameters in the adaptive unit, and the text modal features are processed using the text modal parameters to obtain multimodal features; the multimodal features are input into the shared feedforward unit to generate the reply text corresponding to the conversation text.

[0202] By applying the solutions of the embodiments of the present disclosure, during the conversation processing process, the target conversation model proposed in the embodiments of the present disclosure can unify the features of different modalities into the same semantic space for modeling, while preserving the characteristics of the modalities. This balances the specificity between modalities and leverages the synergy between different modalities to achieve high accuracy. Furthermore, the target conversation model uses a smaller model size and less data, achieving both improved accuracy across multiple multimodal datasets and improved metrics for plain text on plain text datasets, achieving excellent model performance.

[0203] Corresponding to the above-mentioned dialogue processing method embodiment, the present disclosure also provides a dialogue processing device embodiment. FIG11 shows a schematic diagram of the structure of a dialogue processing device provided by one embodiment of the present disclosure. As shown in FIG11 , the device includes:

[0204] A first acquisition module 1102 is configured to acquire a conversation text and conversation-related data, wherein the conversation-related data is in a different modality than the conversation text;

[0205] A first input module 1104 is configured to input the conversation text and conversation-related data into a target conversation model to obtain first modal features corresponding to the conversation-related data and text modal features corresponding to the conversation text, wherein the target conversation model is obtained by training a pre-trained conversation model for a conversation task based on the first sample set, and the pre-trained conversation model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model for a text prediction task based on the second sample set;

[0206] A first processing module 1108 is configured to process the first modal feature using the first modal parameter and to process the text modal feature using the text modal parameter to obtain a multimodal feature;

[0207] The first generating module 1110 is configured to generate a reply text corresponding to the conversation text according to the multimodal features.

[0208] Optionally, the target dialogue model includes a first encoding unit, a second encoding unit and an adaptive unit, and the adaptive unit includes a first modal parameter and a text modal parameter; the first input module 1104 is further configured to input the dialogue-related data into the first encoding unit to obtain the first modal feature corresponding to the dialogue-related data, and input the dialogue text into the second encoding unit to obtain the text modal feature corresponding to the dialogue text; the first processing module 1108 is further configured to input the first modal feature and the text modal feature into the adaptive unit, process the first modal feature using the first modal parameter, and process the text modal feature using the text modal parameter to obtain a multimodal feature.

[0209] Optionally, the first encoding unit includes a first encoding subunit and a first summary subunit; the first input module 1104 is further configured to input the conversation-related data into the first encoding subunit to obtain initial modal features corresponding to the conversation-related data; input the initial modal features into the first summary subunit, use the target query vector to perform feature screening on the initial modal features, and obtain first modal features corresponding to the conversation-related data.

[0210] Optionally, the adaptive unit includes a first normalization layer, a self-attention layer and a second normalization layer, the first normalization layer includes first modal parameters and text modal parameters, and the second normalization layer includes first modal parameters and text modal parameters; the first processing module 1108 is further configured to input the modal identification, first modal features and text modal features into the first normalization layer, the self-attention layer and the second normalization layer in sequence to obtain multimodal features, wherein the modal identification is used to identify the first modal features and text modal features respectively.

[0211] Optionally, the target dialogue model includes a shared feedforward unit; the first generation module 1110 is further configured to input the multimodal features into the shared feedforward unit, decode the multimodal features, and generate a reply text corresponding to the dialogue text.

[0212] Optionally, the device also includes: a sending module, configured to send a reply text to the client, so that the client displays the reply text to the user based on the display requirement information; receive adjustment information sent by the user based on the reply text, and adjust the parameters of the target dialogue model according to the adjustment information.

[0213] Optionally, the apparatus further includes: a first generation module configured to obtain a first sample set, wherein the first sample set includes multiple sample conversation texts and multiple sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond one-to-one, and the sample conversation texts carry sample conversation labels; input the multiple sample conversation texts and the multiple sample conversation associated data into a pre-trained conversation model to obtain first prediction modal features corresponding to the multiple sample conversation associated data and text prediction modal features corresponding to the multiple sample conversation texts, wherein the pre-trained conversation model is trained for a text prediction task on a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on a second sample set; process the first prediction modal feature using the first modal parameter and process the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature; generate predicted reply texts corresponding to the multiple sample conversation texts based on the multimodal prediction feature; adjust model parameters of the pre-trained conversation model based on the predicted reply text and the sample conversation labels to obtain a trained target conversation model.

[0214] Optionally, the pre-trained dialogue model includes a first encoding unit and a second encoding unit, the first encoding unit includes a first encoding subunit and a first summary subunit; the first generation module is further configured to input multiple sample dialogue association data into the first encoding subunit to obtain initial prediction modal features corresponding to the multiple sample dialogue association data respectively; input the initial prediction modal features into the first summary subunit, use the target query vector to perform feature screening on the initial prediction modal features, and obtain first prediction modal features corresponding to the multiple sample dialogue association data respectively; input multiple sample dialogue texts into the second encoding unit to obtain text prediction modal features corresponding to the multiple sample dialogue texts respectively.

[0215] Optionally, the device also includes: a second generation module, configured to obtain a second sample set, wherein the second sample set includes multiple sample association data, and the sample association data carries a sample text label; inputting the multiple sample association data into the initial dialogue model, and obtaining second prediction modal features corresponding to the multiple sample association data respectively through the first encoding unit; processing the second prediction modal features using the first modal parameters through the adaptive unit of the initial dialogue model to obtain target prediction features; generating predicted texts corresponding to the multiple sample association data respectively according to the target prediction features; adjusting the first modal parameters and the encoding parameters of the first encoding unit according to the predicted text and the sample text label to obtain a pre-trained dialogue model that has been trained.

[0216] Optionally, the adaptive unit includes a text modal parameter; the device further includes: a first setting module configured to set a first modal parameter in the adaptive unit based on the text modal parameter.

[0217] By applying the solution of the disclosed embodiments, conversation-related data and conversation text are separately encoded in the target conversation model, and feature processing is performed using different modal parameters. This allows for differentiation of data from different modalities, alleviating the friction between these modalities while improving modal coordination and ensuring the accuracy of conversation processing. Furthermore, because the pre-trained conversation model is trained on text prediction tasks, it can convert data modally to text modality. Furthermore, because the target conversation model is trained on conversation tasks, it can handle conversation tasks based on modal conversion, thereby improving the accuracy of the target conversation model.

[0218] The above is a schematic diagram of a conversation processing device according to this embodiment. It should be noted that the technical solution of this conversation processing device and the technical solution of the conversation processing method described above are based on the same concept. For details not described in detail in the technical solution of the conversation processing device, please refer to the description of the technical solution of the conversation processing method described above.

[0219] Corresponding to the above-mentioned text generation method embodiment, the present disclosure also provides a text generation device embodiment. FIG12 shows a schematic structural diagram of a text generation device provided by an embodiment of the present disclosure. As shown in FIG12 , the device includes:

[0220] The second acquisition module 1202 is configured to acquire a target image and a task text corresponding to the target image;

[0221] A second input module 1204 is configured to input a target image and a task text into a target dialogue model to obtain image modal features corresponding to the target image and text modal features corresponding to the task text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit and a first modal parameter in an adaptive unit in an initial dialogue model for a text prediction task based on the second sample set;

[0222] The second processing module 1206 is configured to process the image modality feature using the first modality parameter and process the text modality feature using the text modality parameter to obtain a multimodal feature;

[0223] The second generating module 1208 is configured to generate a target text corresponding to the target image according to the multimodal features.

[0224] Applying the solution of the embodiments of the present disclosure, by separately encoding the target image and task text in the target dialogue model and performing feature processing using different modal parameters, it is possible to distinguish data of different modalities, alleviate the problem of pulling between different modal data, and simultaneously improve modal collaboration and ensure the accuracy of text generation. Furthermore, because the pre-trained dialogue model is trained through text prediction tasks, the pre-trained dialogue model can perform modal conversion on the data to obtain data in text modality. Furthermore, because the target dialogue model is trained through dialogue tasks, it can handle dialogue tasks based on modal conversion, thereby improving the accuracy of the target dialogue model.

[0225] The above is a schematic diagram of a text generation device according to this embodiment. It should be noted that the technical solution of the text generation device and the technical solution of the above-mentioned text generation method are based on the same concept. For details not described in detail in the technical solution of the text generation device, please refer to the description of the technical solution of the above-mentioned text generation method.

[0226] Corresponding to the above-mentioned target dialogue model training method embodiment, the present disclosure also provides a target dialogue model training device embodiment. Figure 13 shows a schematic diagram of the structure of a target dialogue model training device provided by one embodiment of the present disclosure. As shown in Figure 13, the device is applied to a cloud-side device and includes:

[0227] A third acquisition module 1302 is configured to acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation-related data, the sample conversation texts and the sample conversation-related data correspond one to one, and the sample conversation texts carry sample conversation tags;

[0228] The third input module 1304 is configured to input a plurality of sample conversation texts and a plurality of sample conversation association data into a pre-trained conversation model to obtain first prediction modal features corresponding to the plurality of sample conversation association data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is trained on a text prediction task by performing text prediction on a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on the second sample set;

[0229] The third processing module 1306 is configured to process the first prediction modal feature using the first modal parameter and process the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0230] The third generating module 1308 is configured to generate predicted reply texts corresponding to the plurality of sample conversation texts according to the multimodal prediction features;

[0231] The adjustment module 1310 is configured to adjust the model parameters of the pre-trained dialogue model according to the predicted reply text and the sample dialogue label to obtain a trained target dialogue model.

[0232] Using the solution of the disclosed embodiments, the model parameters of the pre-trained dialogue model are adjusted based on the predicted response text and sample conversation labels. If the first preset stopping condition is not met, the pre-trained dialogue model is trained continuously until the first preset stopping condition is met, completing the training and obtaining the target dialogue model. By continuously adjusting the model parameters of the pre-trained dialogue model, the resulting target dialogue model can be made more accurate.

[0233] The above is a schematic diagram of a target dialogue model training device according to this embodiment. It should be noted that the technical solution of this target dialogue model training device and the technical solution of the target dialogue model training method described above are based on the same concept. For details not described in detail in the technical solution of the target dialogue model training device, please refer to the description of the technical solution of the target dialogue model training method described above.

[0234] Figure 14 shows a block diagram of a computing device according to an embodiment of the present disclosure. Components of the computing device 1400 include, but are not limited to, a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 via a bus 1430, and a database 1450 is used to store data.

[0235] Computing device 1400 also includes an access device 1440 that enables computing device 1400 to communicate via one or more networks 1460. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. Access device 1440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0236] In one embodiment of the present disclosure, the aforementioned components of computing device 1400 and other components not shown in FIG14 may also be connected to each other, for example, via a bus. It should be understood that the block diagram of the computing device structure shown in FIG14 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.

[0237] Computing device 1400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1400 may also be a mobile or stationary server.

[0238] The processor 1420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned dialogue processing method, text generation method, or target dialogue model training method.

[0239] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solutions of the aforementioned dialogue processing method, text generation method, and target dialogue model training method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the aforementioned dialogue processing method, text generation method, or target dialogue model training method.

[0240] An embodiment of the present disclosure further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned dialogue processing method, text generation method, or target dialogue model training method.

[0241] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solutions of the aforementioned dialogue processing method, text generation method, and target dialogue model training method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the aforementioned dialogue processing method, text generation method, or target dialogue model training method.

[0242] An embodiment of the present disclosure further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned dialogue processing method or text generation method or target dialogue model training method.

[0243] The above is an illustrative embodiment of a computer program. It should be noted that the technical solution of this computer program shares the same concept as the technical solutions of the aforementioned dialogue processing method, text generation method, and target dialogue model training method. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solutions of the aforementioned dialogue processing method, text generation method, or target dialogue model training method.

[0244] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0245] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0246] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present disclosure are not limited by the described order of actions, because according to the embodiments of the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present disclosure. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0247] The preferred embodiments of the present disclosure disclosed above are only used to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to only the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of the present disclosure. The present disclosure selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. A conversation processing method, comprising: Acquire a dialogue text and dialogue-related data, wherein the dialogue-related data is in a different modality from the dialogue text; Inputting the dialogue text and the dialogue-related data into a target dialogue model to obtain a first modal feature corresponding to the dialogue-related data and a text modal feature corresponding to the dialogue text, wherein the target dialogue model is obtained by performing dialogue task training on a pre-trained dialogue model based on a first sample set, and the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit and a first modal parameter in an adaptive unit in an initial dialogue model based on a second sample set; Processing the first modal feature using a first modal parameter and processing the text modal feature using a text modal parameter to obtain a multimodal feature; Based on the multimodal features, a reply text corresponding to the conversation text is generated.

2. According to the method of claim 1, the target dialogue model comprises a first encoding unit, a second encoding unit and an adaptive unit, and the adaptive unit comprises a first modal parameter and a text modal parameter; The step of inputting the dialogue text and the dialogue-related data into a target dialogue model to obtain a first modal feature corresponding to the dialogue-related data and a text modal feature corresponding to the dialogue text includes: Inputting the conversation-related data into the first encoding unit to obtain a first modal feature corresponding to the conversation-related data, and inputting the conversation text into the second encoding unit to obtain a text modal feature corresponding to the conversation text; The step of processing the first modal feature using the first modal parameter and processing the text modal feature using the text modal parameter to obtain a multimodal feature includes: The first modal feature and the text modal feature are input into the adaptive unit, the first modal feature is processed using a first modal parameter, and the text modal feature is processed using a text modal parameter to obtain a multimodal feature.

3. The method according to claim 2, wherein the first encoding unit comprises a first encoding subunit and a first summary subunit; The step of inputting the conversation-related data into the first encoding unit to obtain a first modal feature corresponding to the conversation-related data includes: Inputting the conversation-related data into the first encoding subunit to obtain initial modal features corresponding to the conversation-related data; The initial modal features are input into the first summary subunit, and the initial modal features are screened using the target query vector to obtain the first modal features corresponding to the conversation-related data.

4. The method according to claim 2, wherein the adaptive unit comprises a first normalization layer, a self-attention layer, and a second normalization layer, wherein the first normalization layer comprises a first modality parameter and a text modality parameter, and the second normalization layer comprises a first modality parameter and a text modality parameter; The step of inputting the first modal feature and the text modal feature into the adaptive unit, processing the first modal feature using the first modal parameter, and processing the text modal feature using the text modal parameter to obtain a multimodal feature includes: The modal identification, the first modal feature and the text modal feature are sequentially input into the first normalization layer, the self-attention layer and the second normalization layer to obtain multimodal features, wherein the modal identification is used to respectively identify the first modal feature and the text modal feature.

5. The method according to any one of claims 1 to 4, wherein the target dialogue model comprises a shared feedforward unit; Generating a reply text corresponding to the dialogue text according to the multimodal feature includes: The multimodal features are input into the shared feedforward unit, the multimodal features are decoded, and the pair The reply text corresponding to the conversation text.

6. The method according to any one of claims 1 to 4, after generating a reply text corresponding to the conversation text according to the multimodal features, further comprising: Sending the reply text to the client, so that the client displays the reply text to the user based on the display requirement information; Receive adjustment information sent by the user based on the reply text, and adjust parameters of the target dialogue model according to the adjustment information.

7. The method according to any one of claims 1 to 4, before inputting the dialogue text and the dialogue-related data into the target dialogue model, further comprising: Acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond to each other one-to-one, and the sample conversation texts carry sample conversation tags; Inputting the plurality of sample dialogue texts and the plurality of sample dialogue association data into a pre-trained dialogue model, obtaining first prediction modal features respectively corresponding to the plurality of sample dialogue association data and text prediction modal features respectively corresponding to the plurality of sample dialogue texts, wherein the pre-trained dialogue model is obtained by performing text prediction task training on the first encoding unit in the initial dialogue model and the first modal parameters in the adaptive unit based on the second sample set; Processing the first prediction modal feature using a first modal parameter and processing the text prediction modal feature using a text modal parameter to obtain a multimodal prediction feature; Generating predicted reply texts corresponding to the plurality of sample dialogue texts respectively according to the multimodal prediction features; According to the predicted reply text and the sample dialogue label, the model parameters of the pre-trained dialogue model are adjusted to obtain a trained target dialogue model.

8. The method according to claim 7, wherein the pre-trained dialogue model comprises a first encoding unit and a second encoding unit, wherein the first encoding unit comprises a first encoding subunit and a first summary subunit; The step of inputting the plurality of sample conversation texts and the plurality of sample conversation associated data into a pre-trained conversation model to obtain first prediction modal features respectively corresponding to the plurality of sample conversation associated data and text prediction modal features respectively corresponding to the plurality of sample conversation texts includes: Inputting the plurality of sample conversation associated data into the first encoding subunit to obtain initial predicted modal features respectively corresponding to the plurality of sample conversation associated data; Inputting the initial predicted modal features into the first summarization subunit, and performing feature screening on the initial predicted modal features using the target query vector to obtain first predicted modal features corresponding to the plurality of sample conversation associated data respectively; The multiple sample dialogue texts are input into the second encoding unit to obtain text prediction modal features respectively corresponding to the multiple sample dialogue texts.

9. The method according to claim 7, before inputting the plurality of sample conversation texts and the plurality of sample conversation associated data into the pre-trained conversation model, further comprising: Acquire a second sample set, wherein the second sample set includes a plurality of sample-related data, and the sample-related data carries a sample text label; Inputting the plurality of sample-related data into an initial dialogue model, and obtaining second prediction modal features respectively corresponding to the plurality of sample-related data through a first encoding unit; The adaptive unit of the initial dialogue model processes the second predicted modal feature using the first modal parameter to obtain a target predicted feature; Generate predicted texts corresponding to the plurality of sample-related data respectively according to the target prediction features; According to the predicted text and the sample text label, the first modal parameter and the encoding parameter of the first encoding unit are adjusted to obtain a trained pre-trained dialogue model.

10. The method according to claim 9, wherein the adaptive unit comprises a text modality parameter; Before the adaptive unit of the initial dialogue model processes the second predicted modal feature using the first modal parameter to obtain the target predicted feature, the method further includes: Based on the text modal parameter, a first modal parameter in the adaptive unit is set.

11. A text generation method, comprising: Acquire a target image and a task text corresponding to the target image; Inputting the target image and the task text into a target dialogue model to obtain image modality features corresponding to the target image and text modality features corresponding to the task text, wherein the target dialogue model is obtained by performing dialogue task training on a pre-trained dialogue model based on a first sample set, and the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit and a first modality parameter in an adaptive unit in an initial dialogue model based on a second sample set; Processing the image modality feature using a first modality parameter and processing the text modality feature using a text modality parameter to obtain a multimodal feature; A target text corresponding to the target image is generated according to the multimodal features.

12. A target dialogue model training method, applied to a cloud-side device, comprising: Acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond to each other one-to-one, and the sample conversation texts carry sample conversation tags; Inputting the plurality of sample dialogue texts and the plurality of sample dialogue association data into a pre-trained dialogue model, obtaining first prediction modal features respectively corresponding to the plurality of sample dialogue association data and text prediction modal features respectively corresponding to the plurality of sample dialogue texts, wherein the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit in an initial dialogue model and a first modal parameter in an adaptive unit based on a second sample set; Processing the first prediction modal feature using a first modal parameter and processing the text prediction modal feature using a text modal parameter to obtain a multimodal prediction feature; Generating predicted reply texts corresponding to the plurality of sample dialogue texts respectively according to the multimodal prediction features; According to the predicted reply text and the sample dialogue label, the model parameters of the pre-trained dialogue model are adjusted to obtain a trained target dialogue model.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 10 or claim 11 or claim 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 10 or claim 11 or claim 12.

15. A computer program, when executed in a computer, causes the computer to execute the steps of the method according to any one of claims 1 to 10 or claim 11 or claim 12.

Images