Dialogue processing method, text generation method and target dialogue model training method

Abstract

The embodiment of the invention provides a dialogue processing method, a text generation method and a target dialogue model training method, and the dialogue processing method comprises the steps that a dialogue text and dialogue associated data are obtained, and the mode of the dialogue associated data is different from that of the dialogue text; the dialogue text and the dialogue associated data are input into a target dialogue model, first modal features corresponding to the dialogue associated data and text modal features corresponding to the dialogue text are obtained, and the target dialogue model is obtained by conducting dialogue task training on a pre-trained dialogue model based on a first sample set; the pre-trained dialogue model is obtained by performing text prediction task training on a first coding unit in the initial dialogue model and a first modal parameter in an adaptive unit based on a second sample set; processing the first modal feature by using the first modal parameter, and processing the text modal feature by using the text modal parameter to obtain a multi-modal feature; and according to the multi-modal features, generating a reply text corresponding to the dialogue text.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to dialogue processing, text generation, and target dialogue model training methods. Background Art

[0002] With the development of computer technology, large models have begun to shine, showing extraordinary capabilities in language understanding, generation, interaction and reasoning, and are widely used in natural language processing fields such as dialogue, translation, and code generation. However, as the data modalities generated by enterprises and users increase, how to use large models to process multimodal data has gradually become a research focus.

[0003] Currently, before using the big model for dialogue processing, the data of different modalities are first encoded, and then the encoded data features are sent to the big model for dialogue processing. However, in the above solution, the big model does not directly process multimodal data, resulting in the low accuracy of multimodal data processing of the big model. Therefore, a dialogue processing solution with high accuracy is urgently needed. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a method for processing a dialogue. One or more embodiments of this specification also relate to a text generation method, a target dialogue model training method, a dialogue processing device, a text generation device, a target dialogue model training device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a dialog processing method is provided, including:

[0006] Acquire a dialogue text and dialogue-related data, wherein the dialogue-related data is in a different modality from the dialogue text;

[0007] Inputting the dialogue text and the dialogue-related data into the target dialogue model, obtaining a first modal feature corresponding to the dialogue-related data and a text modal feature corresponding to the dialogue text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit in the initial dialogue model and a first modal parameter in the adaptive unit based on the second sample set for a text prediction task;

[0008] Processing the first modal feature using the first modal parameter and processing the text modal feature using the text modal parameter to obtain a multimodal feature;

[0009] Generate reply text corresponding to the conversation text based on multimodal features.

[0010] According to a second aspect of the embodiments of this specification, a text generation method is provided, including:

[0011] Obtain the target image and the task text corresponding to the target image;

[0012] Inputting a target image and a task text into a target dialogue model to obtain an image modality feature corresponding to the target image and a text modality feature corresponding to the task text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on a first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit in an initial dialogue model and a first modality parameter in an adaptive unit based on a second sample set for a text prediction task;

[0013] The image modality feature is processed using the first modality parameter, and the text modality feature is processed using the text modality parameter to obtain a multimodal feature;

[0014] According to the multimodal features, the target text corresponding to the target image is generated.

[0015] According to a third aspect of an embodiment of this specification, a target dialogue model training method is provided, which is applied to a cloud-side device, including:

[0016] Acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond one to one, and the sample conversation texts carry sample conversation tags;

[0017] Inputting a plurality of sample conversation texts and a plurality of sample conversation associated data into a pre-trained conversation model, obtaining first prediction modal features corresponding to the plurality of sample conversation associated data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is obtained by performing text prediction task training on a first encoding unit in an initial conversation model and a first modal parameter in an adaptive unit based on a second sample set;

[0018] Processing the first prediction modal feature using the first modal parameter and processing the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0019] Generate predicted reply texts corresponding to multiple sample conversation texts according to the multimodal prediction features;

[0020] According to the predicted reply text and sample conversation labels, the model parameters of the pre-trained conversation model are adjusted to obtain the trained target conversation model.

[0021] According to a fourth aspect of an embodiment of this specification, a dialog processing device is provided, including:

[0022] A first acquisition module is configured to acquire a dialogue text and dialogue-related data, wherein the dialogue-related data is in a different modality from the dialogue text;

[0023] A first input module is configured to input the dialogue text and the dialogue-related data into a target dialogue model to obtain a first modal feature corresponding to the dialogue-related data and a text modal feature corresponding to the dialogue text, wherein the target dialogue model is obtained by performing dialogue task training on a pre-trained dialogue model based on a first sample set, and the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit in an initial dialogue model and a first modal parameter in an adaptive unit based on a second sample set;

[0024] A first processing module is configured to process the first modal feature using the first modal parameter and process the text modal feature using the text modal parameter to obtain a multimodal feature;

[0025] The first generation module is configured to generate a reply text corresponding to the dialogue text based on the multimodal features.

[0026] According to a fifth aspect of the embodiments of this specification, there is provided a text generation device, including:

[0027] A second acquisition module is configured to acquire a target image and a task text corresponding to the target image;

[0028] a second input module configured to input a target image and a task text into a target dialogue model to obtain an image modality feature corresponding to the target image and a text modality feature corresponding to the task text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit in an initial dialogue model and a first modality parameter in an adaptive unit based on the second sample set for a text prediction task;

[0029] A second processing module is configured to process the image modality feature using the first modality parameter and process the text modality feature using the text modality parameter to obtain a multimodal feature;

[0030] The second generation module is configured to generate a target text corresponding to the target image according to the multimodal features.

[0031] According to a sixth aspect of an embodiment of this specification, a target dialogue model training device is provided, which is applied to a cloud-side device, including:

[0032] A third acquisition module is configured to acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond to each other one by one, and the sample conversation texts carry sample conversation tags;

[0033] A third input module is configured to input a plurality of sample dialogue texts and a plurality of sample dialogue association data into a pre-trained dialogue model, and obtain first prediction modal features corresponding to the plurality of sample dialogue association data and text prediction modal features corresponding to the plurality of sample dialogue texts, wherein the pre-trained dialogue model is obtained by training a first encoding unit in an initial dialogue model and a first modal parameter in an adaptive unit for a text prediction task based on the second sample set;

[0034] A third processing module is configured to process the first prediction modal feature using the first modal parameter and process the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0035] A third generation module is configured to generate predicted reply texts corresponding to a plurality of sample dialogue texts respectively according to the multimodal prediction features;

[0036] The adjustment module is configured to adjust the model parameters of the pre-trained dialogue model according to the predicted reply text and the sample dialogue label to obtain the trained target dialogue model.

[0037] According to a seventh aspect of an embodiment of this specification, there is provided a computing device, including:

[0038] Memory and processor;

[0039] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0040] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0041] According to a ninth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the method provided in the first aspect, the second aspect, or the third aspect above.

[0042] A conversation processing method provided by an embodiment of the present specification obtains conversation text and conversation-related data, wherein the conversation-related data is of a different modality from the conversation text; inputs the conversation text and the conversation-related data into a target conversation model to obtain a first modal feature corresponding to the conversation-related data and a text modal feature corresponding to the conversation text, wherein the target conversation model is obtained by performing conversation task training on a pre-trained conversation model based on a first sample set, and the pre-trained conversation model is obtained by performing text prediction task training on a first encoding unit and a first modal parameter in an adaptive unit in an initial conversation model based on a second sample set; the first modal feature is processed using the first modal parameter, and the text modal feature is processed using the text modal parameter to obtain a multimodal feature; and a reply text corresponding to the conversation text is generated based on the multimodal feature.

[0043] By encoding the dialogue-related data and dialogue text separately in the target dialogue model and using different modal parameters for feature processing, it is possible to distinguish data of different modalities, alleviate the problem of pulling between different modal data, and improve modal coordination, ensuring the accuracy of dialogue processing. In addition, since the pre-trained dialogue model is trained through text prediction tasks, the pre-trained dialogue model can perform modal conversion on the data to obtain text modal data. Furthermore, since the target dialogue model is trained through dialogue tasks, the target dialogue model can process dialogue tasks based on modal conversion, improving the accuracy of the target dialogue model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is an architecture diagram of a conversation processing system provided by an embodiment of this specification;

[0045] Figure 2 is an architecture diagram of another dialog processing system provided by an embodiment of this specification;

[0046] Figure 3 is a flow chart of a conversation processing method provided by an embodiment of this specification;

[0047] Figure 4 is a structural diagram of an adaptive unit in a target dialogue model provided by an embodiment of this specification;

[0048] Figure 5 is a training diagram of a target dialogue model provided by an embodiment of this specification;

[0049] Figure 6 is a training diagram of a pre-trained dialogue model provided by an embodiment of this specification;

[0050] Figure 7 is a flow chart of a text generation method provided by an embodiment of this specification;

[0051] Figure 8 is a flow chart of a target dialogue model training method provided by an embodiment of this specification;

[0052] Fig. 9 is a flowchart of a pre-trained dialogue model training method provided by an embodiment of this specification;

[0053] Fig.10 is a structural diagram of a target dialogue model provided by an embodiment of this specification;

[0054] Fig.11 is a structural diagram of a conversation processing device provided by an embodiment of this specification;

[0055] Fig.12 It is a structural schematic diagram of a text generation device provided by an embodiment of this specification;

[0056] Fig.13 It is a structural diagram of a target dialogue model training device provided by an embodiment of this specification;

[0057] Fig.14 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0058] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0059] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0060] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0061] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0062] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multimodal pre-training model, etc.

[0063] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0064] First, the terms involved in one or more embodiments of this specification are explained.

[0065] Large-scale language model: Large-scale language model can be called large language model or large model, which refers to a language model that can process large-scale corpus. Such models usually have very high parameter volume and complexity, and can learn the deep structure and complex relationships of language, thereby realizing the natural generation and understanding of language.

[0066] Multimodal Large Language Model: The Multimodal Large Language Model (MLLM) can understand and generate natural language and can process multiple types of input data, including text, images, audio, etc.

[0067] With the popularity of big models, how to use big models to process multimodal data has gradually become a research focus. Currently, big models can be used to process multimodal data in the following two ways. The first is to use a three-stage training method to pre-train the big model, pre-train multi-tasks, and fine-tune instructions (I Tuning, Instruction Tuning) to obtain a model that can process multimodal data. The second is to encode data of different modalities, and then send the encoded data features to the big model for dialogue processing. However, the amount of training data and the model scale are both large in the first solution. In the second solution, the big model does not directly process multimodal data, resulting in the low accuracy of multimodal data processing of the big model.

[0068] In order to solve the above problems, the embodiments of this specification propose a modular and lightweight multimodal model system (mPLUG, multi-modal Pre-training framework for both vision-Language Understanding and Generation) that combines multimodal representation with a large-scale language model to unlock the multimodal capabilities of a large-scale language model. The target dialogue model is obtained through two-stage training (pre-training and instruction fine-tuning), which uses less data and a smaller model scale, and performs very well on various benchmark indicators. In addition, the dialogue processing method proposed in the embodiments of this specification can be widely used in scenarios such as dialogue assistants, multimodal chat robots, writing robots, travel guides, creative copywriting, user guides, exhibition guides, etc., which has promoted the development of academia.

[0069] Specifically, an embodiment of the present specification proposes a dialogue processing method, which obtains dialogue text and dialogue-related data, wherein the dialogue-related data is different from the dialogue text in modality; inputs the dialogue text and the dialogue-related data into a target dialogue model to obtain a first modal feature corresponding to the dialogue-related data and a text modal feature corresponding to the dialogue text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on a first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit in an initial dialogue model and a first modal parameter in an adaptive unit based on a second sample set for a text prediction task; the first modal feature is processed using the first modal parameter, and the text modal feature is processed using the text modal parameter to obtain a multimodal feature; and a reply text corresponding to the dialogue text is generated based on the multimodal feature.

[0070] By encoding the dialogue-related data and dialogue text separately in the target dialogue model and using different modal parameters for feature processing, it is possible to distinguish data of different modalities, alleviate the problem of pulling between different modal data, and improve modal coordination, ensuring the accuracy of dialogue processing. In addition, since the pre-trained dialogue model is trained through text prediction tasks, the pre-trained dialogue model can perform modal conversion on the data to obtain text modal data. Furthermore, since the target dialogue model is trained through dialogue tasks, the target dialogue model can process dialogue tasks based on modal conversion, improving the accuracy of the target dialogue model.

[0071] In this specification, a dialogue processing method is provided. This specification also involves a text generation method, a target dialogue model training method, a dialogue processing device, a text generation device, a target dialogue model training device, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0072] See also Figure 1 , Figure 1 The diagram shows an architecture diagram of a dialog processing system provided by an embodiment of the present specification. The dialog processing system may include a client 100 and a server 200;

[0073] The client 100 is used to send a conversation text and conversation-related data to the server 200, wherein the conversation-related data is in a different mode from the conversation text;

[0074] The server 200 is used to input the dialogue text and the dialogue-related data into the target dialogue model, and obtain the first modal feature corresponding to the dialogue-related data and the text modal feature corresponding to the dialogue text, wherein the target dialogue model is obtained by training the pre-trained dialogue model for dialogue tasks based on the first sample set, and the pre-trained dialogue model is obtained by training the first encoding unit and the first modal parameter in the adaptive unit in the initial dialogue model for text prediction tasks based on the second sample set; the first modal feature is processed using the first modal parameter, and the text modal feature is processed using the text modal parameter to obtain a multi-modal feature; a reply text corresponding to the dialogue text is generated according to the multi-modal feature; and the reply text is sent to the client 100;

[0075] The client 100 is also used to receive the reply text sent by the server 200.

[0076] By applying the solution of the embodiment of this specification, the dialogue-related data and the dialogue text are encoded separately in the target dialogue model, and different modal parameters are used for feature processing, so that data of different modalities can be distinguished, the pulling problem between different modal data is alleviated, and modal coordination is improved, and the accuracy of dialogue processing is ensured. In addition, since the pre-trained dialogue model is obtained by training the text prediction task, the pre-trained dialogue model can perform modal conversion on the data to obtain data of text modality. Furthermore, since the target dialogue model is obtained by training the dialogue task, the target dialogue model can process the dialogue task on the basis of modal conversion, thereby improving the accuracy of the target dialogue model.

[0077] See also Figure 2 , Figure 2 The diagram shows an architecture of another dialog processing system provided by an embodiment of the present specification. The dialog processing system may include multiple clients 100 and a server 200, wherein the client 100 may include a terminal-side device and the server 200 may include a cloud-side device. A communication connection may be established between multiple clients 100 through the server 200. In the dialog processing scenario, the server 200 is used to provide dialog processing services between multiple clients 100. Multiple clients 100 may serve as a sending end or a receiving end respectively, and realize communication through the server 200.

[0078] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the dialogue processing scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a reply text according to the data stream and pushes the reply text to other clients that have established communication.

[0079] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0080] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language Version 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 can be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The client 100 can be deployed in an electronic device, and needs to rely on the device to run or some APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0081] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers for background training that provide support for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0082] It is worth noting that the conversation processing method provided in the embodiments of this specification is generally executed by the server, but in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the conversation processing method provided in the embodiments of this specification. In other embodiments, the conversation processing method provided in the embodiments of this specification may also be jointly executed by the client and the server.

[0083] See also Figure 3 , Figure 3 A flow chart of a conversation processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0084] Step 302: Acquire the dialogue text and dialogue-related data, wherein the dialogue-related data is in a different mode from the dialogue text.

[0085] In one or more embodiments of the present specification, a conversation text and conversation-related data may be obtained to perform conversation processing based on the conversation text and the conversation-related data to obtain a reply result corresponding to the conversation text.

[0086] Specifically, dialogue-related data refers to data with different data modalities and text modalities. Dialogue-related data can be at least one of dialogue image data, dialogue audio data, dialogue video data, and the like. Dialogue text is used to express the purpose of the dialogue. For example, dialogue-related data can be a landscape picture, and dialogue text can be "Please answer what season is in the picture". Dialogue-related data and dialogue text can be data in a variety of scenarios, including but not limited to conference scenarios, traffic navigation scenarios, teaching scenarios, and the like.

[0087] In actual applications, there are many ways to obtain the conversation text and the conversation-related data, which can be selected according to the actual situation, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the conversation text and the conversation-related data sent by the user through the client can be received. In another possible implementation of this specification, the conversation text and the conversation-related data can be read from other data acquisition devices or databases.

[0088] Step 304: Input the dialogue text and dialogue-related data into the target dialogue model to obtain the first modal features corresponding to the dialogue-related data and the text modal features corresponding to the dialogue text, wherein the target dialogue model is obtained by performing dialogue task training on a pre-trained dialogue model based on the first sample set, and the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit in the initial dialogue model and a first modal parameter in an adaptive unit based on the second sample set.

[0089] Specifically, the target dialogue model is a natural language processing model. The target dialogue model is a model obtained by training the initial dialogue model in two stages, namely, a pre-training stage and an instruction fine-tuning stage. In the pre-training stage, only the encoding parameters of the first encoding unit and the first modal parameters in the adaptive unit are trained, and in the instruction fine-tuning stage, all parameters of the model are fine-tuned. The text modal feature is an effective feature representation of the dialogue text. The first modal feature is an effective feature representation of the dialogue-related data. If the dialogue-related data is dialogue image data, the first modal feature is an image modal feature, which can also be called a visual feature; if the dialogue-related data is dialogue audio data, the first modal feature is an audio modal feature, which can also be called an auditory feature.

[0090] Step 306: Process the first modal feature using the first modal parameter, and process the text modal feature using the text modal parameter to obtain a multimodal feature.

[0091] Specifically, the first modal parameter is used to process the first modal feature, and the text modal parameter is used to process the text modal feature.

[0092] It should be noted that there are multiple ways to input the dialogue text and dialogue-related data into the target dialogue model to obtain the first modal features corresponding to the dialogue-related data and the text modal features corresponding to the dialogue text. The selection is made according to the actual situation, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the dialogue text and dialogue-related data can be input into the target dialogue model, and the dialogue text and dialogue-related data can be encoded using the same encoding unit in the target dialogue model. In another possible implementation of this specification, in view of the different modal characteristics of the dialogue-related data and the dialogue text data, different modalities can be distinguished by modules in the target dialogue model, and different encoding units can be used to process data of different modalities.

[0093] In practical applications, the first modal feature is processed using the first modal parameter, and the text modal feature is processed using the text modal parameter. There are many ways to obtain multimodal features, which are selected according to actual conditions. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the preset first modal parameter and text modal parameter can be obtained, and the first modal feature is directly processed using the first modal parameter, and the text modal feature is processed using the text modal parameter to obtain multimodal features. In another possible implementation of this specification, an adaptive unit can be set in the target dialogue model, and the adaptive unit includes the first modal parameter and the text modal parameter, and the first modal parameter and the text modal parameter are updated as the target dialogue model is trained.

[0094] In an optional embodiment of the present specification, the target dialogue model includes a first encoding unit, a second encoding unit and an adaptive unit, and the adaptive unit includes a first modal parameter and a text modal parameter; the above-mentioned inputting the dialogue text and the dialogue-related data into the target dialogue model to obtain the first modal feature corresponding to the dialogue-related data and the text modal feature corresponding to the dialogue text may include the following steps:

[0095] Inputting the conversation-related data into a first encoding unit to obtain a first modal feature corresponding to the conversation-related data, and inputting the conversation text into a second encoding unit to obtain a text modal feature corresponding to the conversation text;

[0096] Processing the first modal feature using the first modal parameter and processing the text modal feature using the text modal parameter to obtain the multimodal feature may include the following steps:

[0097] The first modal feature and the text modal feature are input into the adaptive unit, the first modal feature is processed using the first modal parameter, and the text modal feature is processed using the text modal parameter to obtain a multimodal feature.

[0098] Specifically, the first encoding unit is used to encode the conversation-related data. The second encoding unit is used to encode the conversation text. The Modality-Adaptive Module can use different modality parameters to encode and map the features of different modalities.

[0099] By applying the solution of the embodiments of this specification, different encoding units are used to process data of different modalities, thereby ensuring the accuracy of the first modal features and the text modal features. In addition, an adaptive unit is set in the target dialogue model so that the first modal parameters and the text modal parameters can be trained and updated, thereby ensuring the accuracy of the first modal parameters and the text modal parameters.

[0100] In actual applications, the conversation-related data is input into the first encoding unit to obtain the first modal feature corresponding to the conversation-related data, and the conversation text is input into the second encoding unit. There are many ways to obtain the text modal feature corresponding to the conversation text, and the specific method is selected according to the actual situation. The embodiments of this specification do not impose any limitations on this.

[0101] In a possible implementation of the present specification, the conversation-related data may be directly encoded using a first encoding unit to obtain a first modal feature, and the conversation text may be encoded using a second encoding unit to obtain a text modal feature.

[0102] In another possible implementation of the present specification, since the length of the first modal feature may be relatively long, in order to improve the processing effect of the adaptive unit in the target dialogue model, a first summary subunit may be additionally introduced into the first encoding unit, and the length of the first modal feature may be reduced by using the first summary subunit, that is, the first encoding unit includes the first encoding subunit and the first summary subunit; the above-mentioned inputting the dialogue-related data into the first encoding unit to obtain the first modal feature corresponding to the dialogue-related data may include the following steps:

[0103] Inputting the dialogue-related data into the first encoding subunit to obtain initial modal features corresponding to the dialogue-related data;

[0104] The initial modal features are input into the first summary subunit, and the target query vector is used to perform feature screening on the initial modal features to obtain the first modal features corresponding to the conversation-related data.

[0105] It should be noted that the first encoding subunit is used to convert the input conversation-related data into a series of vector representations, namely, initial modal features, which contain all the information of the conversation-related data.

[0106] Furthermore, when the initial modal features are processed using the first summary subunit, a learnable target query vector (Learnable Queries) can be used to perform feature screening and summary on the initial modal features to obtain the first modal features.

[0107] Exemplarily, taking the conversation-related data as conversation image data, the first encoding subunit is a visual encoder (Visual Encoder), and the first summary subunit is a visual abstractor (Visual Abstractor). The first encoding subunit and the first summary subunit perform feature extraction on the conversation image data, and use a learnable target query vector to perform feature screening and summary on the visual information, so as to obtain image modal features.

[0108] By applying the solution of the embodiments of the present specification, the conversation-related data is input into the first encoding subunit to obtain initial modal features corresponding to the conversation-related data; the initial modal features are input into the first summary subunit, and the initial modal features are screened using the target query vector to obtain first modal features corresponding to the conversation-related data. The sequence length of the first modal features is reduced by the first summary subunit and the learnable target query vector, thereby improving the processing efficiency of the adaptive unit.

[0109] In an optional embodiment of the present specification, in the adaptive unit, different normalization layers (Norm) can also be used to normalize the features of different modalities, that is, the adaptive unit includes a first normalization layer, a self-attention layer and a second normalization layer, the first normalization layer includes a first modal parameter and a text modal parameter, and the second normalization layer includes a first modal parameter and a text modal parameter; the above-mentioned inputting the first modal feature and the text modal feature into the adaptive unit, processing the first modal feature using the first modal parameter, and processing the text modal feature using the text modal parameter to obtain the multimodal feature, can include the following steps:

[0110] The modal identification, the first modal features and the text modal features are sequentially input into the first normalization layer, the self-attention layer and the second normalization layer to obtain multimodal features, wherein the modal identification is used to identify the first modal features and the text modal features respectively.

[0111] It should be noted that, after the first normalization layer, the first modal feature can be normalized using the modal identification and the first modal parameter, and the text modal feature can be normalized using the modal identification and the text modal parameter to obtain the first attention parameter and the text attention parameter; after the self-attention layer, the first attention parameter and the text attention parameter can be attention-processed to obtain the self-attention feature; after the second normalization layer, the first modal feature and the self-attention feature can be normalized using the modal identification and the first modal parameter, and the text modal feature and the self-attention feature can be normalized using the modal identification and the first modal parameter to obtain the multimodal feature.

[0112] Specifically, the modality indicator is used to identify feature data of different modalities, for example, the feature with subscript 0 is a text modality feature, and the feature with subscript 1 is a first modality feature. Similarly, the modality parameters in the adaptive unit can also be distinguished by the modality indicator, for example, the parameter with subscript 0 is a text modality parameter, and the parameter with subscript 1 is a first modality parameter.

[0113] See also Figure 4 , Figure 4 FIG. 1 shows a structural diagram of an adaptive unit in a target dialogue model provided by an embodiment of the present specification, such as Figure 4 As shown, the adaptive unit includes a first normalization layer, a self-attention layer, and a second normalization layer. The first normalization layer includes Norm 0 and Norm 1 ; The self-attention layer includes a matrix multiplication and scaling layer (MatMul&Scale), a normalization layer (Softmax), a matrix multiplication layer, and a linear layer (Linear); The second normalization layer includes Norm 0 and Norm1 , where the Norm in the first normalization layer 0 It is used to normalize the input text modality features using the modality identifier and text modality parameters to obtain the text attention parameters, Norm 1 The first attention parameter is used to normalize the input first modal feature using the modal identifier and the first modal parameter to obtain the first attention parameter. Further, the first query vector (Q 1 ), the first value vector (V 1 ) and the first key vector (K 1 ) and the text query vector (Q 0 ), text value vector (V 0 ) and the text key vector (K 0 ) is input into the self-attention layer, and Q 1 and Q 0 The corresponding weight W Q 、V 1 The corresponding weight W V1 、V 0 The corresponding weight W V0 , K 1 The corresponding weight W K1 , K 0 The corresponding weight W K0 The self-attention layer is also input, and the first attention parameter and the text attention parameter are processed by the self-attention layer to obtain the self-attention feature. Finally, the first modality feature, the text modality feature, and the self-attention feature are added and input into the second normalization layer, and the multi-modal feature is obtained after the normalization processing of the second normalization layer. The multi-modal feature can be called cross-modality feature.

[0114] By applying the solution of the embodiments of this specification, different modal parameters are set in the adaptive unit according to the characteristics of different modes, thereby realizing module differentiation of the characteristics of different modes. The idea of ​​modal collaboration in which the modalities can help each other is adopted. According to the characteristics of different modalities, the data processing of different modalities is module differentiated, and different parameters are used for modeling, thereby alleviating the problem of modal pull and improving modal collaboration.

[0115] Step 308: Generate a reply text corresponding to the conversation text based on the multimodal features.

[0116] In one or more embodiments of the present specification, a conversation text and conversation-related data are obtained, wherein the conversation-related data and the conversation text have different modalities; the conversation text and the conversation-related data are input into a target conversation model to obtain a first modal feature corresponding to the conversation-related data and a text modal feature corresponding to the conversation text; the first modal feature is processed using a first modal parameter, and the text modal feature is processed using a text modal parameter, and after obtaining multimodal features, a reply text corresponding to the conversation text can be further generated based on the multimodal features.

[0117] For example, assuming the dialogue text is "Please answer what season is in the picture", the reply text corresponding to the dialogue text is "It is winter in the picture".

[0118] By applying the solution of the embodiment of this specification, the dialogue-related data and the dialogue text are encoded separately in the target dialogue model, and different modal parameters are used for feature processing, so that data of different modalities can be distinguished, the pulling problem between different modal data is alleviated, and modal coordination is improved, and the accuracy of dialogue processing is ensured. In addition, since the pre-trained dialogue model is obtained by training the text prediction task, the pre-trained dialogue model can perform modal conversion on the data to obtain data of text modality. Furthermore, since the target dialogue model is obtained by training the dialogue task, the target dialogue model can process the dialogue task on the basis of modal conversion, thereby improving the accuracy of the target dialogue model.

[0119] In an optional embodiment of the present specification, the target dialogue model includes a shared feedforward unit; the above-mentioned generating a reply text corresponding to the dialogue text according to the multimodal features may include the following steps:

[0120] The multimodal features are input into the shared feedforward unit, the multimodal features are decoded, and the reply text corresponding to the dialogue text is generated.

[0121] Specifically, in a feedforward neural network (FFN), each node has independent weights and biases, which means that a large number of parameters are required to train the network. In order to reduce the complexity of the model, the embodiments of this specification propose a shared feedforward unit (Shared FFN, Shared Feed Forward Network), where all nodes in the same stage can share the same FFN, and only a single weight and bias matrix is ​​needed to complete the calculation, thereby greatly reducing the number of parameters required for the model.

[0122] In practical applications, the shared feedforward unit includes an input layer, a first hidden layer, an activation layer, a second hidden layer, and an output layer. The input layer is used to receive the output from the previous layer and pass it to the next node. The first hidden layer is a linear transformation layer, which is used to perform a full connection operation on the input layer using shared weights and bias matrices. The activation layer includes an activation function, which is used to convert the output of the hidden layer into a nonlinear activation signal. The second hidden layer is also a linear transformation layer, which is used to perform a full connection operation on the output of the first hidden layer using shared weights and bias matrices. The output layer is used to scale and translate the output of the second hidden layer, and then return it to the previous layer, and finally generate a reply text corresponding to the conversation text.

[0123] By applying the solution of the embodiment of this specification, the multimodal features are input into the shared feedforward unit, the multimodal features are decoded, and the reply text corresponding to the dialogue text is generated. The shared feedforward unit can effectively extract the multimodal features and convert them into a higher level of abstract representation. At the same time, by sharing the weight and bias matrix, the complexity of the model can be significantly reduced and the accuracy of the model can be improved.

[0124] In an optional embodiment of the present specification, after generating the reply text corresponding to the dialogue text according to the multimodal features, the following steps may also be included:

[0125] Sending a reply text to the client, so that the client displays the reply text to the user based on the display requirement information;

[0126] Receive adjustment information sent by the user based on the reply text, and adjust parameters of the target dialogue model according to the adjustment information.

[0127] In actual applications, there are many ways to send a reply text to the client, which can be selected based on actual conditions, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the reply text can be sent directly to the client. In another possible implementation of this specification, the reply text can be sent to the client based on the user's display demand information. Among them. The display demand information represents the user's demand for viewing the reply text. The display demand information includes, but is not limited to, displaying only the reply text, displaying the conversation-related data and the reply text and identifying and distinguishing the two, displaying multiple reply texts for the user to choose, and displaying the link corresponding to the reply text. The display demand information is specifically set according to the actual needs of the user, and the embodiments of this specification do not impose any restrictions on this.

[0128] Furthermore, after the user obtains the reply text, if he is not satisfied with the reply text, he can send adjustment information based on the reply text to retrain the target dialogue model and perform secondary dialogue processing. The implementation method of "adjusting the parameters of the target dialogue model according to the adjustment information" is the same as the training method of the target dialogue model, and will not be described in detail in this embodiment of the specification.

[0129] By applying the scheme of the embodiments of this specification, a reply text is sent to the client, so that the client displays the reply text to the user based on the display demand information; adjustment information sent by the user based on the reply text is received, and parameters of the target dialogue model are adjusted according to the adjustment information, thereby increasing the interaction with the user and improving user satisfaction.

[0130] In an optional embodiment of the present specification, before the dialog text and dialog-related data are input into the target dialog model, the following steps may also be included:

[0131] Acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond one to one, and the sample conversation texts carry sample conversation tags;

[0132] Inputting a plurality of sample conversation texts and a plurality of sample conversation associated data into a pre-trained conversation model, obtaining first prediction modal features corresponding to the plurality of sample conversation associated data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is obtained by performing text prediction task training on a first encoding unit in an initial conversation model and a first modal parameter in an adaptive unit based on a second sample set;

[0133] Processing the first prediction modal feature using the first modal parameter and processing the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0134] Generate predicted reply texts corresponding to multiple sample conversation texts according to the multimodal prediction features;

[0135] According to the predicted reply text and sample conversation labels, the model parameters of the pre-trained conversation model are adjusted to obtain the trained target conversation model.

[0136] Specifically, the first sample set can be called a text instruction sample set, the multiple sample dialogue texts in the first sample set can be called a text instruction sample set, and the multiple sample dialogue associated data can be called a multimodal instruction sample set. The initial dialogue model can be an untrained natural language processing model, or a model that directly loads large-scale language model parameters. The training method for the pre-trained dialogue model is supervised training, that is, the multiple sample dialogue texts included in the first sample set carry real sample dialogue labels, and the sample dialogue labels are the generation targets of the pre-trained dialogue model, which are used to guide the model training process.

[0137] See also Figure 5 , Figure 5 A training schematic diagram of a target dialogue model provided by an embodiment of the present specification is shown. During the training process of the target dialogue model, the pre-trained dialogue model can be fine-tuned using a text instruction sample set and a multimodal instruction sample set, thereby achieving a modal collaboration effect, that is, multiple sample dialogue texts and multiple sample dialogue-related data are input into a first encoding subunit and a first summary subunit, and sample dialogue tags carried by the sample dialogue texts are input into a decoding unit for instruction fine-tuning to obtain a trained target dialogue model.

[0138] In practical applications, there are many ways to obtain the first sample set, which are selected according to actual conditions, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the first sample set can be read from other data acquisition devices or databases. In another possible implementation of this specification, the first sample set sent by the user can be received.

[0139] It should be noted that the implementation method of "inputting multiple sample conversation texts and multiple sample conversation associated data into the pre-trained conversation model to obtain first prediction modal features corresponding to the multiple sample conversation associated data and text prediction modal features corresponding to the multiple sample conversation texts" is the same as the implementation method of the above step 304; the implementation method of "processing the first prediction modal features using the first modal parameters, and processing the text prediction modal features using the text modal parameters to obtain multimodal prediction features" is the same as the implementation method of the above step 306; the implementation method of "generating predicted reply texts corresponding to multiple sample conversation texts according to the multimodal prediction features" is the same as the implementation method of the above step 308, and this embodiment of the specification will not be repeated here.

[0140] In actual applications, when adjusting the model parameters of the pre-trained dialogue model according to the predicted reply text and the sample dialogue labels, the first loss value can be calculated according to the predicted reply text and the sample dialogue labels, and the model parameters of the pre-trained dialogue model can be adjusted according to the first loss value until the first preset stopping condition is reached, thereby obtaining a trained target dialogue model.

[0141] It should be noted that there are many functions for calculating the first loss value, such as cross entropy loss function, L1 norm loss function, maximum loss function, mean square error loss function, logarithmic loss function, etc. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0142] In a possible implementation of the present specification, the first preset stop condition includes that the first loss value is less than or equal to a first preset threshold value. After calculating the first loss value based on the predicted reply text and the sample conversation tag, the first loss value is compared with the first preset threshold value.

[0143] Specifically, if the first loss value is greater than the first preset threshold, it means that the difference between the predicted reply text and the sample conversation label is large, and the pre-trained conversation model has poor prediction ability for the reply text. At this time, the model parameters of the pre-trained conversation model can be adjusted, and the pre-trained conversation model can continue to be trained until the first loss value is less than or equal to the first preset threshold, indicating that the difference between the predicted reply text and the sample conversation label is small, and the first preset stop condition is met, and the target conversation model that has completed the training is obtained.

[0144] In another possible implementation of the present specification, in addition to comparing the first loss value and the first preset threshold, it is also possible to determine whether the current pre-trained dialogue model is trained in combination with the number of iterations.

[0145] Specifically, if the first loss value is greater than a first preset threshold, the model parameters of the pre-trained dialogue model are adjusted, and the pre-trained dialogue model is continuously trained until a first preset number of iterations is reached, and then the iteration is stopped to obtain a target dialogue model that has completed the training, wherein the first preset threshold and the first preset number of iterations are selected according to actual conditions, and the embodiments of this specification do not impose any limitation on this.

[0146] By applying the solution of the embodiment of this specification, the model parameters of the pre-trained dialogue model are adjusted according to the predicted reply text and the sample dialogue label, and the pre-trained dialogue model is continuously trained when the first preset stop condition is not met, until the first preset stop condition is met, and the training is completed to obtain the target dialogue model. By continuously adjusting the model parameters of the pre-trained dialogue model, the target dialogue model finally obtained can be made more accurate.

[0147] In an optional embodiment of the present specification, the pre-trained dialogue model includes a first encoding unit and a second encoding unit, and the first encoding unit includes a first encoding sub-unit and a first summary sub-unit; the above-mentioned inputting a plurality of sample dialogue texts and a plurality of sample dialogue association data into the pre-trained dialogue model to obtain the first prediction modal features corresponding to the plurality of sample dialogue association data and the text prediction modal features corresponding to the plurality of sample dialogue texts may include the following steps:

[0148] Inputting the plurality of sample conversation associated data into the first encoding subunit to obtain initial predicted modal features corresponding to the plurality of sample conversation associated data respectively;

[0149] Inputting the initial predicted modal features into the first summary subunit, and using the target query vector to perform feature screening on the initial predicted modal features to obtain first predicted modal features corresponding to the plurality of sample conversation associated data respectively;

[0150] The plurality of sample dialogue texts are input into the second encoding unit to obtain text prediction modal features respectively corresponding to the plurality of sample dialogue texts.

[0151] It should be noted that the implementation method of the above-mentioned "inputting multiple sample conversation association data into the first encoding subunit to obtain initial predicted modal features corresponding to the multiple sample conversation association data respectively; inputting the initial predicted modal features into the first summary subunit, using the target query vector to perform feature screening on the initial predicted modal features, and obtaining the first predicted modal features corresponding to the multiple sample conversation association data respectively" is the same as the implementation method of the above-mentioned "inputting the conversation association data into the first encoding subunit to obtain the initial modal features corresponding to the conversation association data; inputting the initial modal features into the first summary subunit, using the target query vector to perform feature screening on the initial modal features, and obtaining the first modal features corresponding to the conversation association data", and this embodiment of the specification will not be repeated in detail.

[0152] By applying the solution of the embodiment of this specification, the sequence length of the first prediction modal feature is reduced through the first summary sub-unit and the learnable target query vector, thereby improving the processing efficiency of the adaptive unit.

[0153] In an optional embodiment of the present specification, before the multiple sample conversation texts and the multiple sample conversation associated data are input into the pre-trained conversation model, the following steps may also be included:

[0154] Acquire a second sample set, wherein the second sample set includes a plurality of sample-related data, and the sample-related data carries a sample text label;

[0155] Inputting a plurality of sample-related data into an initial dialogue model, and obtaining second prediction modal features respectively corresponding to the plurality of sample-related data through a first encoding unit;

[0156] The adaptive unit of the initial dialogue model processes the second prediction modal feature using the first modal parameter to obtain a target prediction feature;

[0157] Generate prediction texts corresponding to multiple sample associated data according to the target prediction features;

[0158] According to the predicted text and the sample text label, the first modal parameter and the encoding parameter of the first encoding unit are adjusted to obtain a pre-trained dialogue model that has been trained.

[0159] Specifically, the second sample set can be called a modality-aligned sample set. Taking the sample-associated data as sample image data as an example, the training process of the initial dialogue model using the second sample set can be understood as a process of image-text alignment. The training method for the initial dialogue model is supervised training, that is, the multiple sample-associated data included in the second sample set carry real sample text labels. The sample text labels are the generation targets of the initial dialogue model and are used to guide the model training process.

[0160] See also Figure 6 , Figure 6 A training schematic diagram of a pre-trained dialogue model provided by an embodiment of the present specification is shown. During the training process of the pre-trained dialogue model, multiple sample-related data are input into the first encoding subunit and the first summary subunit, and the sample text labels carried by the sample-related data are input into the decoding unit for partial parameter adjustment, that is, the first encoding subunit and the first summary subunit are trained, and the first modal parameters in the adaptive unit are trained at the same time, and the encoding parameters of the second encoding unit and the text modal parameters in the adaptive unit are kept unchanged during the training process.

[0161] In practical applications, the method for obtaining the first sample set may refer to the method for obtaining the second sample set described above, and the embodiments of this specification do not limit the method for obtaining the first sample set.

[0162] It should be noted that the implementation method of “inputting multiple sample-related data into the initial dialogue model, and obtaining the second prediction modal features corresponding to the multiple sample-related data respectively through the first encoding unit” is the same as the implementation method of “inputting the dialogue-related data into the first encoding unit, and obtaining the first modal features corresponding to the dialogue-related data”; the implementation method of “processing the second prediction modal features using the first modal parameters through the adaptive unit of the initial dialogue model to obtain the target prediction features” is the same as the implementation method of “inputting the first modal features and the text modal features into the adaptive unit, processing the first modal features using the first modal parameters, and processing the text modal features using the text modal parameters”. The implementation method of the above-mentioned "processing the features to obtain multimodal features" is the same as the implementation method of the above-mentioned "generating predicted texts corresponding to multiple sample associated data respectively according to the target prediction features" is the same as the implementation method of the above-mentioned "generating a reply text corresponding to the dialogue text according to the multimodal features"; the implementation method of "adjusting the first modal parameters and the encoding parameters of the first encoding unit according to the predicted text and the sample text label to obtain a trained pre-trained dialogue model" is the same as the implementation method of the above-mentioned "adjusting the model parameters of the pre-trained dialogue model according to the predicted reply text and the sample dialogue label to obtain a trained target dialogue model", which will not be repeated in the embodiments of this specification.

[0163] By applying the solution of the embodiment of this specification, the first modal parameter of the initial dialogue model and the encoding parameter of the first encoding unit are adjusted according to the predicted text and the sample text label, and the initial dialogue model is continuously trained when the second preset stop condition is not met until the second preset stop condition is met, and the training is completed to obtain the pre-trained dialogue model. By continuously adjusting the first modal parameter of the initial dialogue model and the encoding parameter of the first encoding unit, the final pre-trained dialogue model can be made more accurate.

[0164] In an optional embodiment of the present specification, the initial dialogue model further includes a second encoding unit, and the second encoding unit includes text encoding parameters; before the plurality of sample-related data are input into the initial dialogue model and the second prediction modal features respectively corresponding to the plurality of sample-related data are obtained by the first encoding unit, the following steps may also be included:

[0165] Based on the text encoding parameters, encoding parameters of the first encoding unit are set.

[0166] In practical applications, there are multiple ways to set the encoding parameters of the first encoding unit based on the text encoding parameters, which are selected according to the actual situation, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the text encoding parameters can be copied, and the copied text encoding parameters are set as the encoding parameters of the first encoding unit. In another possible implementation of this specification, the text encoding parameters can be copied, the copied text encoding parameters are adjusted, and the adjusted text encoding parameters are set as the encoding parameters of the first encoding unit, wherein the adjustment method includes but is not limited to increasing the preset parameters and reducing the preset parameters.

[0167] By applying the solution of the embodiments of this specification, the encoding parameters of the first encoding unit are set based on the text encoding parameters, so that the encoding parameters of the first encoding unit can be trained separately, thereby setting different encoding units for different modal data, thereby alleviating the problem of modal pull and improving modal coordination.

[0168] In an optional embodiment of the present specification, the adaptive unit includes text modal parameters; the adaptive unit of the initial dialogue model may further include the following steps before processing the second predicted modal features using the first modal parameters to obtain the target predicted features:

[0169] Based on the text modal parameter, a first modal parameter in the adaptive unit is set.

[0170] In practical applications, there are many ways to set the first modal parameter in the adaptive unit based on the text modal parameter, which can be selected according to the actual situation, and the embodiments of this specification do not limit this. In one possible implementation of this specification, the text modal parameter can be copied, and the copied text modal parameter is set as the first modal parameter in the adaptive unit. In another possible implementation of this specification, the text modal parameter can be copied, the copied text modal parameter is adjusted, and the adjusted text modal parameter is set as the first modal parameter in the adaptive unit, wherein the adjustment method includes but is not limited to increasing the preset parameter and reducing the preset parameter.

[0171] By applying the solution of the embodiment of this specification, the first modal parameter in the adaptive unit is set based on the text modal parameter, so that the first modal parameter in the adaptive unit can be trained separately, thereby setting different modal parameters for different modal data, thereby alleviating the problem of modal pull and improving modal coordination.

[0172] The following combination Figure 7 , taking the application of the dialogue processing method provided in this specification in the text generation scenario as an example, the dialogue processing method is further explained. Among them, Figure 7A flowchart of a text generation method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0173] Step 702: Obtain a target image and a task text corresponding to the target image.

[0174] Step 704: Input the target image and the task text into the target dialogue model to obtain image modal features corresponding to the target image and text modal features corresponding to the task text, wherein the target dialogue model is obtained by training a pre-trained dialogue model for a dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training a first encoding unit in the initial dialogue model and a first modal parameter in the adaptive unit based on the second sample set for a text prediction task.

[0175] Step 706: Process the image modal feature using the first modal parameter, and process the text modal feature using the text modal parameter to obtain a multimodal feature.

[0176] Step 708: Generate target text corresponding to the target image based on the multimodal features.

[0177] It should be noted that the implementation of step 702 to step 708 is the same as the implementation of step 302 to step 308 described above, and will not be described in detail in the embodiments of this specification.

[0178] For example, assuming that the target image is a girl standing happily on the podium, the task text corresponding to the target image is "Why is the girl happy?", and the target text generated by the above text generation method is "The girl is happy because she is standing on the podium."

[0179] By applying the solution of the embodiment of this specification, the target image and the task text corresponding to the target image are encoded in the target dialogue model respectively, and different modal parameters are used for feature processing, so as to achieve the distinction of data of different modalities, alleviate the pulling problem between different modal data, and improve modal coordination, and ensure the accuracy of text generation. In addition, since the pre-trained dialogue model is obtained through training of text prediction tasks, the pre-trained dialogue model can perform modal conversion on the data to obtain data of text modality. Furthermore, since the target dialogue model is obtained through training of dialogue tasks, the target dialogue model can process dialogue tasks on the basis of modal conversion, thereby improving the accuracy of the target dialogue model.

[0180] See also Figure 8 , Figure 8 A flow chart of a target dialogue model training method provided by an embodiment of the present specification is shown. The target dialogue model training method is applied to a cloud-side device and specifically includes the following steps:

[0181] Step 802: Obtain a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond one to one, and the sample conversation texts carry sample conversation tags.

[0182] Step 804: Input a plurality of sample conversation texts and a plurality of sample conversation associated data into a pre-trained conversation model to obtain first prediction modal features corresponding to the plurality of sample conversation associated data and text prediction modal features corresponding to the plurality of sample conversation texts, wherein the pre-trained conversation model is obtained by performing text prediction task training on a first encoding unit in an initial conversation model and a first modal parameter in an adaptive unit based on a second sample set.

[0183] Step 806: Process the first prediction modal feature using the first modal parameter, and process the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature.

[0184] Step 808: Generate predicted reply texts corresponding to a plurality of sample conversation texts according to the multimodal prediction features.

[0185] Step 810: According to the predicted reply text and the sample dialogue label, the model parameters of the pre-trained dialogue model are adjusted to obtain a trained target dialogue model.

[0186] It should be noted that the implementation method of steps 802 to 810 can refer to the training method of the target dialogue model in the above-mentioned dialogue processing method, and this embodiment of the specification will not be described in detail.

[0187] In actual applications, after the cloud-side device obtains the trained target dialogue model, it can send the model parameters of the target dialogue model to the terminal-side device, so that the user can locally build the target dialogue model based on the model parameters of the target dialogue model and further perform dialogue processing tasks.

[0188] By applying the solution of the embodiment of this specification, the model parameters of the pre-trained dialogue model are adjusted according to the predicted reply text and the sample dialogue label, and the pre-trained dialogue model is continuously trained when the first preset stop condition is not met, until the first preset stop condition is met, and the training is completed to obtain the target dialogue model. By continuously adjusting the model parameters of the pre-trained dialogue model, the target dialogue model finally obtained can be made more accurate.

[0189] See also Fig. 9 , Fig. 9 A flowchart of a pre-trained dialogue model training method provided by an embodiment of the present specification is shown. The pre-trained dialogue model training method is applied to a cloud-side device and specifically includes the following steps:

[0190] Step 902: Acquire a second sample set, wherein the second sample set includes a plurality of sample-related data, and the sample-related data carries sample text labels.

[0191] Step 904: input the plurality of sample-related data into the initial dialogue model, and obtain second predicted modal features respectively corresponding to the plurality of sample-related data through a first encoding unit.

[0192] Step 906: The adaptive unit of the initial dialogue model processes the second predicted modal feature using the first modal parameter to obtain a target predicted feature.

[0193] Step 908: Generate predicted texts corresponding to the plurality of sample-related data respectively according to the target prediction features.

[0194] Step 910: According to the predicted text and the sample text label, the first modal parameter and the encoding parameter of the first encoding unit are adjusted to obtain a pre-trained dialogue model.

[0195] It should be noted that the implementation method of steps 902 to 910 can refer to the training method of the pre-trained dialogue model in the above-mentioned dialogue processing method, and this embodiment of the specification will not be described in detail.

[0196] In actual applications, after the cloud-side device obtains the pre-trained dialogue model, it can send the model parameters of the pre-trained dialogue model to the terminal-side device, so that the user can locally build the pre-trained dialogue model based on the model parameters of the pre-trained dialogue model to further perform dialogue processing tasks.

[0197] By applying the solution of the embodiment of this specification, the first modal parameter of the initial dialogue model and the encoding parameter of the first encoding unit are adjusted according to the predicted text and the sample text label, and the initial dialogue model is continuously trained when the second preset stop condition is not met until the second preset stop condition is met, and the training is completed to obtain the pre-trained dialogue model. By continuously adjusting the first modal parameter of the initial dialogue model and the encoding parameter of the first encoding unit, the final pre-trained dialogue model can be made more accurate.

[0198] See also Fig.10 , Fig.10 A structural diagram of a target dialogue model provided by an embodiment of the present specification is shown, wherein the target dialogue model includes a first encoding unit, a second encoding unit, and a decoding unit, wherein the first encoding unit includes a first encoding subunit and a first summary subunit, and the decoding unit includes an adaptive unit and a shared feedforward unit;

[0199] like Fig.10As shown, the conversation associated data is input into the first encoding subunit to obtain the initial modal features; the initial modal features and the target query vector are input into the first summary subunit to obtain the first modal features; the conversation text is input into the second encoding unit to obtain the text modal features; the first modal features and the text modal features are input into the adaptive unit, in which the first modal features are processed using the first modal parameters and the text modal features are processed using the text modal parameters to obtain the multimodal features; the multimodal features are input into the shared feedforward unit to generate the reply text corresponding to the conversation text.

[0200] By applying the solution of the embodiments of this specification, during the dialogue processing, the target dialogue model proposed in the embodiments of this specification can unify the features of different modalities into the same semantic space for modeling, and retain the characteristics of the modalities, thereby balancing the specificity between the modalities and utilizing the mutual synergy of different modalities to achieve higher accuracy. In addition, the target dialogue model uses a smaller model size and less data, which can not only improve the accuracy of multiple multimodal data sets, but also improve the indicators of plain text on plain text data sets, achieving very good model performance.

[0201] Corresponding to the above-mentioned dialogue processing method embodiment, this specification also provides a dialogue processing device embodiment, Fig.11 FIG. 1 is a schematic diagram showing the structure of a dialog processing device provided by an embodiment of the present specification. Fig.11 As shown, the device comprises:

[0202] A first acquisition module 1102 is configured to acquire a conversation text and conversation-related data, wherein the conversation-related data is in a different modality from the conversation text;

[0203] The first input module 1104 is configured to input the dialogue text and the dialogue-related data into the target dialogue model, and obtain the first modal feature corresponding to the dialogue-related data and the text modal feature corresponding to the dialogue text, wherein the target dialogue model is obtained by performing dialogue task training on the pre-trained dialogue model based on the first sample set, and the pre-trained dialogue model is obtained by performing text prediction task training on the first encoding unit and the first modal parameter in the adaptive unit in the initial dialogue model based on the second sample set;

[0204] A first processing module 1108 is configured to process the first modal feature using the first modal parameter and process the text modal feature using the text modal parameter to obtain a multimodal feature;

[0205] The first generating module 1110 is configured to generate a reply text corresponding to the conversation text according to the multimodal features.

[0206] Optionally, the target dialogue model includes a first encoding unit, a second encoding unit and an adaptive unit, the adaptive unit includes a first modal parameter and a text modal parameter; the first input module 1104 is further configured to input dialogue-related data into the first encoding unit to obtain first modal features corresponding to the dialogue-related data, and input the dialogue text into the second encoding unit to obtain text modal features corresponding to the dialogue text; the first processing module 1108 is further configured to input the first modal features and the text modal features into the adaptive unit, process the first modal features using the first modal parameters, and process the text modal features using the text modal parameters to obtain multimodal features.

[0207] Optionally, the first encoding unit includes a first encoding subunit and a first summary subunit; the first input module 1104 is further configured to input the conversation-related data into the first encoding subunit to obtain initial modal features corresponding to the conversation-related data; input the initial modal features into the first summary subunit, and use the target query vector to perform feature screening on the initial modal features to obtain first modal features corresponding to the conversation-related data.

[0208] Optionally, the adaptive unit includes a first normalization layer, a self-attention layer and a second normalization layer, the first normalization layer includes first modal parameters and text modal parameters, and the second normalization layer includes first modal parameters and text modal parameters; the first processing module 1108 is further configured to input the modal identification, first modal features and text modal features into the first normalization layer, the self-attention layer and the second normalization layer in sequence to obtain multimodal features, wherein the modal identification is used to identify the first modal features and the text modal features, respectively.

[0209] Optionally, the target dialogue model includes a shared feedforward unit; the first generation module 1110 is further configured to input the multimodal features into the shared feedforward unit, decode the multimodal features, and generate a reply text corresponding to the dialogue text.

[0210] Optionally, the device also includes: a sending module, configured to send a reply text to the client, so that the client displays the reply text to the user based on the display requirement information; receive adjustment information sent by the user based on the reply text, and adjust parameters of the target dialogue model according to the adjustment information.

[0211] Optionally, the device also includes: a first generation module, configured to obtain a first sample set, wherein the first sample set includes multiple sample dialogue texts and multiple sample dialogue associated data, the sample dialogue texts and the sample dialogue associated data correspond one to one, and the sample dialogue texts carry sample dialogue labels; input the multiple sample dialogue texts and the multiple sample dialogue associated data into a pre-trained dialogue model to obtain first prediction modal features corresponding to the multiple sample dialogue associated data and text prediction modal features corresponding to the multiple sample dialogue texts, wherein the pre-trained dialogue model is obtained by training the first encoding unit in the initial dialogue model and the first modal parameters in the adaptive unit for a text prediction task based on the second sample set; the first prediction modal features are processed using the first modal parameters, and the text prediction modal features are processed using the text modal parameters to obtain multimodal prediction features; based on the multimodal prediction features, predicted reply texts corresponding to the multiple sample dialogue texts are generated; based on the predicted reply texts and the sample dialogue labels, the model parameters of the pre-trained dialogue model are adjusted to obtain a trained target dialogue model.

[0212] Optionally, the pre-trained dialogue model includes a first encoding unit and a second encoding unit, the first encoding unit includes a first encoding subunit and a first summary subunit; the first generation module is further configured to input multiple sample dialogue association data into the first encoding subunit to obtain initial prediction modal features corresponding to the multiple sample dialogue association data respectively; input the initial prediction modal features into the first summary subunit, use the target query vector to perform feature screening on the initial prediction modal features, and obtain first prediction modal features corresponding to the multiple sample dialogue association data respectively; input multiple sample dialogue texts into the second encoding unit to obtain text prediction modal features corresponding to the multiple sample dialogue texts respectively.

[0213] Optionally, the device also includes: a second generation module, configured to obtain a second sample set, wherein the second sample set includes multiple sample-related data, and the sample-related data carries a sample text label; inputting the multiple sample-related data into the initial dialogue model, and obtaining second prediction modal features corresponding to the multiple sample-related data respectively through the first encoding unit; processing the second prediction modal features using the first modal parameters through the adaptive unit of the initial dialogue model to obtain target prediction features; generating predicted texts corresponding to the multiple sample-related data respectively according to the target prediction features; adjusting the first modal parameters and the encoding parameters of the first encoding unit according to the predicted texts and the sample text labels to obtain a pre-trained dialogue model that has been trained.

[0214] Optionally, the adaptive unit includes a text modal parameter; the device further includes: a first setting module configured to set a first modal parameter in the adaptive unit based on the text modal parameter.

[0215] By applying the solution of the embodiment of this specification, the dialogue-related data and the dialogue text are encoded separately in the target dialogue model, and different modal parameters are used for feature processing, so that data of different modalities can be distinguished, the pulling problem between different modal data is alleviated, and modal coordination is improved, and the accuracy of dialogue processing is ensured. In addition, since the pre-trained dialogue model is obtained by training the text prediction task, the pre-trained dialogue model can perform modal conversion on the data to obtain data of text modality. Furthermore, since the target dialogue model is obtained by training the dialogue task, the target dialogue model can process the dialogue task on the basis of modal conversion, thereby improving the accuracy of the target dialogue model.

[0216] The above is a schematic scheme of a dialogue processing device of this embodiment. It should be noted that the technical scheme of the dialogue processing device and the technical scheme of the dialogue processing method described above are of the same concept, and the details not described in detail in the technical scheme of the dialogue processing device can be found in the description of the technical scheme of the dialogue processing method described above.

[0217] Corresponding to the above-mentioned text generation method embodiment, this specification also provides a text generation device embodiment, Fig.12 FIG. 1 shows a schematic diagram of a text generation device provided by an embodiment of the present specification. Fig.12 As shown, the device comprises:

[0218] The second acquisition module 1202 is configured to acquire a target image and a task text corresponding to the target image;

[0219] The second input module 1204 is configured to input the target image and the task text into the target dialogue model to obtain the image modality feature corresponding to the target image and the text modality feature corresponding to the task text, wherein the target dialogue model is obtained by training the pre-trained dialogue model for the dialogue task based on the first sample set, and the pre-trained dialogue model is obtained by training the first encoding unit in the initial dialogue model and the first modality parameter in the adaptive unit based on the second sample set for the text prediction task;

[0220] The second processing module 1206 is configured to process the image modality feature using the first modality parameter and process the text modality feature using the text modality parameter to obtain a multimodal feature;

[0221] The second generating module 1208 is configured to generate a target text corresponding to the target image according to the multimodal features.

[0222] The solution of the embodiment of this specification is applied. The solution of the embodiment of this specification is applied. By encoding the target image and the task text separately in the target dialogue model and using different modal parameters for feature processing, it is possible to distinguish data of different modalities, alleviate the pull problem between different modal data, and improve modal coordination, thereby ensuring the accuracy of text generation. In addition, since the pre-trained dialogue model is obtained by training the text prediction task, the pre-trained dialogue model can perform modal conversion on the data to obtain data of text modality. Furthermore, since the target dialogue model is obtained by training the dialogue task, the target dialogue model can process the dialogue task on the basis of modal conversion, thereby improving the accuracy of the target dialogue model.

[0223] The above is a schematic scheme of a text generation device of this embodiment. It should be noted that the technical scheme of the text generation device and the technical scheme of the above text generation method belong to the same concept, and the details not described in detail in the technical scheme of the text generation device can be referred to the description of the technical scheme of the above text generation method.

[0224] Corresponding to the above-mentioned target dialogue model training method embodiment, this specification also provides a target dialogue model training device embodiment, Fig.13 FIG. 1 is a schematic diagram showing the structure of a target dialogue model training device provided by an embodiment of the present specification. Fig.13 As shown, the device is applied to a cloud-side device, including:

[0225] The third acquisition module 1302 is configured to acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond to each other one by one, and the sample conversation texts carry sample conversation tags;

[0226] The third input module 1304 is configured to input a plurality of sample dialogue texts and a plurality of sample dialogue association data into a pre-trained dialogue model, and obtain first prediction modal features corresponding to the plurality of sample dialogue association data and text prediction modal features corresponding to the plurality of sample dialogue texts, wherein the pre-trained dialogue model is obtained by training the first encoding unit in the initial dialogue model and the first modal parameters in the adaptive unit for a text prediction task based on the second sample set;

[0227] The third processing module 1306 is configured to process the first prediction modal feature using the first modal parameter and process the text prediction modal feature using the text modal parameter to obtain a multimodal prediction feature;

[0228] The third generating module 1308 is configured to generate predicted reply texts corresponding to the plurality of sample dialogue texts respectively according to the multimodal prediction feature;

[0229] The adjustment module 1310 is configured to adjust the model parameters of the pre-trained dialogue model according to the predicted reply text and the sample dialogue label to obtain a trained target dialogue model.

[0230] By applying the solution of the embodiment of this specification, the model parameters of the pre-trained dialogue model are adjusted according to the predicted reply text and the sample dialogue label, and the pre-trained dialogue model is continuously trained when the first preset stop condition is not met, until the first preset stop condition is met, and the training is completed to obtain the target dialogue model. By continuously adjusting the model parameters of the pre-trained dialogue model, the target dialogue model finally obtained can be made more accurate.

[0231] The above is a schematic scheme of a target dialogue model training device of this embodiment. It should be noted that the technical scheme of the target dialogue model training device and the technical scheme of the target dialogue model training method described above belong to the same concept, and the details not described in detail in the technical scheme of the target dialogue model training device can be found in the description of the technical scheme of the target dialogue model training method described above.

[0232] Fig.14 14 shows a block diagram of a computing device provided by an embodiment of the present specification. The components of the computing device 1400 include but are not limited to a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 via a bus 1430, and the database 1450 is used to store data.

[0233] The computing device 1400 also includes an access device 1440 that enables the computing device 1400 to communicate via one or more networks 1460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world microwave interconnection access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0234] In one embodiment of the present specification, the above components of the computing device 1400 and Fig.14 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.14 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0235] The computing device 1400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1400 may also be a mobile or stationary server.

[0236] Among them, the processor 1420 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned dialogue processing method or text generation method or target dialogue model training method.

[0237] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the above-mentioned dialogue processing method, text generation method and target dialogue model training method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned dialogue processing method or text generation method or target dialogue model training method.

[0238] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned dialogue processing method or text generation method or target dialogue model training method.

[0239] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the above-mentioned dialogue processing method, text generation method and target dialogue model training method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned dialogue processing method or text generation method or target dialogue model training method.

[0240] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned dialogue processing method or text generation method or target dialogue model training method.

[0241] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical schemes of the above-mentioned dialogue processing method, text generation method and target dialogue model training method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned dialogue processing method or text generation method or target dialogue model training method.

[0242] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0243] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0244] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0245] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A conversation processing method, comprising: Acquire a dialogue text and dialogue-related data, wherein the dialogue-related data is in a different modality from the dialogue text; Inputting the dialogue text and the dialogue-related data into a target dialogue model to obtain a first modal feature corresponding to the dialogue-related data and a text modal feature corresponding to the dialogue text, wherein the target dialogue model is obtained by performing dialogue task training on a pre-trained dialogue model based on a first sample set, and the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit and a first modal parameter in an adaptive unit in an initial dialogue model based on a second sample set; Processing the first modal feature using a first modal parameter and processing the text modal feature using a text modal parameter to obtain a multimodal feature; Based on the multimodal features, a reply text corresponding to the conversation text is generated.

2. According to the method of claim 1, the target dialogue model comprises a first encoding unit, a second encoding unit and an adaptive unit, and the adaptive unit comprises a first modal parameter and a text modal parameter; The step of inputting the dialogue text and the dialogue-related data into a target dialogue model to obtain a first modal feature corresponding to the dialogue-related data and a text modal feature corresponding to the dialogue text includes: Inputting the conversation-related data into the first encoding unit to obtain a first modal feature corresponding to the conversation-related data, and inputting the conversation text into the second encoding unit to obtain a text modal feature corresponding to the conversation text; The step of processing the first modal feature using the first modal parameter and processing the text modal feature using the text modal parameter to obtain a multimodal feature includes: The first modal feature and the text modal feature are input into the adaptive unit, the first modal feature is processed using a first modal parameter, and the text modal feature is processed using a text modal parameter to obtain a multimodal feature.

3. The method according to claim 2, wherein the first encoding unit comprises a first encoding subunit and a first summary subunit; The step of inputting the conversation-related data into the first encoding unit to obtain a first modal feature corresponding to the conversation-related data includes: Inputting the conversation-related data into the first encoding subunit to obtain initial modal features corresponding to the conversation-related data; The initial modal features are input into the first summary subunit, and the initial modal features are screened using the target query vector to obtain the first modal features corresponding to the conversation-related data.

4. The method according to claim 2, wherein the adaptive unit comprises a first normalization layer, a self-attention layer, and a second normalization layer, wherein the first normalization layer comprises a first modality parameter and a text modality parameter, and the second normalization layer comprises a first modality parameter and a text modality parameter; The step of inputting the first modal feature and the text modal feature into the adaptive unit, processing the first modal feature using the first modal parameter, and processing the text modal feature using the text modal parameter to obtain a multimodal feature includes: The modal identification, the first modal feature and the text modal feature are sequentially input into the first normalization layer, the self-attention layer and the second normalization layer to obtain multimodal features, wherein the modal identification is used to respectively identify the first modal feature and the text modal feature.

5. The method according to claim 1, wherein the target dialogue model comprises a shared feed-forward unit; Generating a reply text corresponding to the dialogue text according to the multimodal feature includes: The multimodal features are input into the shared feedforward unit, the multimodal features are decoded, and a reply text corresponding to the dialogue text is generated.

6. The method according to claim 1, after generating a reply text corresponding to the dialogue text according to the multimodal features, further comprising: Sending the reply text to the client, so that the client displays the reply text to the user based on the display requirement information; Receive adjustment information sent by the user based on the reply text, and adjust parameters of the target dialogue model according to the adjustment information.

7. The method according to claim 1, before inputting the dialogue text and the dialogue-related data into the target dialogue model, further comprising: Acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond to each other one-to-one, and the sample conversation texts carry sample conversation tags; Inputting the plurality of sample dialogue texts and the plurality of sample dialogue association data into a pre-trained dialogue model, obtaining first prediction modal features respectively corresponding to the plurality of sample dialogue association data and text prediction modal features respectively corresponding to the plurality of sample dialogue texts, wherein the pre-trained dialogue model is obtained by performing text prediction task training on the first encoding unit in the initial dialogue model and the first modal parameters in the adaptive unit based on the second sample set; Processing the first prediction modal feature using a first modal parameter and processing the text prediction modal feature using a text modal parameter to obtain a multimodal prediction feature; Generating predicted reply texts corresponding to the plurality of sample dialogue texts respectively according to the multimodal prediction features; According to the predicted reply text and the sample dialogue label, the model parameters of the pre-trained dialogue model are adjusted to obtain a trained target dialogue model.

8. The method according to claim 7, wherein the pre-trained dialogue model comprises a first encoding unit and a second encoding unit, wherein the first encoding unit comprises a first encoding subunit and a first summary subunit; The step of inputting the plurality of sample conversation texts and the plurality of sample conversation associated data into a pre-trained conversation model to obtain first prediction modal features respectively corresponding to the plurality of sample conversation associated data and text prediction modal features respectively corresponding to the plurality of sample conversation texts includes: Inputting the plurality of sample conversation associated data into the first encoding subunit to obtain initial predicted modal features respectively corresponding to the plurality of sample conversation associated data; Inputting the initial predicted modal features into the first summarization subunit, and performing feature screening on the initial predicted modal features using the target query vector to obtain first predicted modal features corresponding to the plurality of sample conversation associated data respectively; The multiple sample dialogue texts are input into the second encoding unit to obtain text prediction modal features respectively corresponding to the multiple sample dialogue texts.

9. The method according to claim 7, before inputting the plurality of sample conversation texts and the plurality of sample conversation associated data into the pre-trained conversation model, further comprising: Acquire a second sample set, wherein the second sample set includes a plurality of sample-related data, and the sample-related data carries a sample text label; Inputting the plurality of sample-related data into an initial dialogue model, and obtaining second prediction modal features respectively corresponding to the plurality of sample-related data through a first encoding unit; The adaptive unit of the initial dialogue model processes the second predicted modal feature using the first modal parameter to obtain a target predicted feature; Generate predicted texts corresponding to the plurality of sample-related data respectively according to the target prediction features; According to the predicted text and the sample text label, the first modal parameter and the encoding parameter of the first encoding unit are adjusted to obtain a trained pre-trained dialogue model.

10. The method according to claim 9, wherein the adaptive unit comprises a text modality parameter; Before the adaptive unit of the initial dialogue model processes the second predicted modal feature using the first modal parameter to obtain the target predicted feature, the method further includes: Based on the text modal parameter, a first modal parameter in the adaptive unit is set.

11. A text generation method, comprising: Acquire a target image and a task text corresponding to the target image; Inputting the target image and the task text into a target dialogue model to obtain image modality features corresponding to the target image and text modality features corresponding to the task text, wherein the target dialogue model is obtained by performing dialogue task training on a pre-trained dialogue model based on a first sample set, and the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit and a first modality parameter in an adaptive unit in an initial dialogue model based on a second sample set; Processing the image modality feature using a first modality parameter and processing the text modality feature using a text modality parameter to obtain a multimodal feature; A target text corresponding to the target image is generated according to the multimodal features.

12. A target dialogue model training method, applied to a cloud-side device, comprising: Acquire a first sample set, wherein the first sample set includes a plurality of sample conversation texts and a plurality of sample conversation associated data, the sample conversation texts and the sample conversation associated data correspond to each other one-to-one, and the sample conversation texts carry sample conversation tags; Inputting the plurality of sample dialogue texts and the plurality of sample dialogue association data into a pre-trained dialogue model, obtaining first prediction modal features respectively corresponding to the plurality of sample dialogue association data and text prediction modal features respectively corresponding to the plurality of sample dialogue texts, wherein the pre-trained dialogue model is obtained by performing text prediction task training on a first encoding unit in an initial dialogue model and a first modal parameter in an adaptive unit based on a second sample set; Processing the first prediction modal feature using a first modal parameter and processing the text prediction modal feature using a text modal parameter to obtain a multimodal prediction feature; Generating predicted reply texts corresponding to the plurality of sample dialogue texts respectively according to the multimodal prediction features; According to the predicted reply text and the sample dialogue label, the model parameters of the pre-trained dialogue model are adjusted to obtain a trained target dialogue model.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 10 or claim 11 or claim 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 10 or claim 11 or claim 12.

Images