Training method and device of dialogue generation model, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a method and device for training a dialogue generation model, an electronic device and a storage medium. The method comprises: obtaining a training sample; the training sample comprises a question in a dialogue text, a reply to the question, and multi-modal data of a target object corresponding to the dialogue text; inputting the question and the multi-modal data into an encoding layer of the dialogue generation model to obtain multi-modal feature representation perceived by the question; inputting the reply to the question and the multi-modal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of the reply to the question; and training the dialogue generation model according to the prediction result of the reply to the question. This technical solution can avoid the situation that the dialogue generation model cannot give an accurate reply due to the missing attribute value of the target object to which the question is directed, thereby improving the prediction accuracy of the dialogue generation model and improving the user experience.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to a method and device for training a dialogue generation model, an electronic device and a storage medium. BACKGROUND

[0002] With the development of technology, task-oriented dialogue systems have attracted extensive attention in the industry and academia. For example, in the e-commerce field, a task-oriented dialogue system can help users to obtain a reply to a question by themselves. However, the dialogue generation method in the current task-oriented dialogue system is based on a structured attribute library, and a reply to a question is matched from the structured attribute library by a keyword. However, in some application scenarios, the structured data in the structured attribute library is often not available. For example, on a second-hand transaction platform, the second-hand information is published by ordinary users, and most of the users publishing the second-hand information will not input the attribute values of the second-hand information according to the definition of the structured attribute library. In addition, there are some second-hand information whose attribute values are not standard attribute values. Therefore, it is difficult to automatically output a reply matching a question by using the current task-oriented dialogue system. SUMMARY

[0003] The present disclosure provides a method and device for training a dialogue generation model, an electronic device and a computer readable storage medium.

[0004] In a first aspect, the present disclosure provides a method for training a dialogue generation model, which comprises:

[0005] obtaining a training sample, wherein the training sample comprises a question in a dialogue text, a reply to the question, and multi-modal data of a target object corresponding to the dialogue text;

[0006] inputting the question and the multi-modal data into an encoding layer of the dialogue generation model to obtain a multi-modal feature representation perceived by the question;

[0007] inputting the reply to the question and the multi-modal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of the reply to the question;

[0008] training the dialogue generation model according to the prediction result of the reply to the question.

[0009] Further, the encoding layer obtains the multi-modal feature representation perceived by the question in the following manner:

[0010] obtaining a first interaction from the question to the multi-modal data and a second interaction from the multi-modal data to the question by using a bidirectional attention mechanism;

[0011] The multi-modal feature representation perceiving the question is obtained according to the first interaction and the second interaction.

[0012] Further, the decoding layer obtains the predicted result of the reply to the question by:

[0013] The reply feature representation corresponding to the reply is obtained by using an attention mechanism, and the predicted result of the reply is obtained based on the multi-modal feature representation and the reply feature representation by using an attention mechanism.

[0014] Further, the encoding layer comprises a first set of encoding modules and at least one second set of encoding modules.

[0015] The first set of encoding modules comprises a plurality of first encoding modules connected in sequence, and the processing result of a previous first encoding module is output to a next first encoding module, and the input of a first first encoding module comprises the question, and the output of a last first encoding module comprises a question feature representation corresponding to the question.

[0016] Each first encoding module comprises a first encoding submodule, and the first encoding submodule performs attention mechanism processing on the input.

[0017] The at least one second set of encoding modules, wherein one second set of encoding modules processes one kind of modal data in the multi-modal data, and each second set of encoding modules comprises a plurality of second encoding modules connected in sequence; in one second set of encoding modules, the processing result of a previous second encoding module is output to a next second encoding module, the input of a first second encoding module is the corresponding one kind of modal data in the multi-modal data, and the output of a last second encoding module comprises a multi-modal feature representation perceiving the question.

[0018] The second encoding module comprises a second encoding submodule and a bidirectional attention mechanism module, the second encoding submodule performs attention mechanism processing on the input, and the bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and the processing result of the second encoding submodule.

[0019] Further, the decoding layer comprises a set of decoding modules, and the set of decoding modules comprises a plurality of decoding modules connected in sequence, the output of a previous decoding module is used as the input of a next decoding module, the input of a first decoding module comprises the reply, and the output of a last decoding module comprises the predicted result corresponding to the reply.

[0020] The decoding module comprises a first decoding submodule and at least one second decoding submodule connected in sequence, the first decoding submodule processes input by using an attention mechanism, the first decoding submodule of the first decoding module obtains reply feature representations corresponding to known parts and a to-be-predicted part in the reply, input of the second decoding submodule comprises output of the first decoding submodule or a previous second decoding submodule and corresponding multi-modal feature representations, the second decoding submodule is configured to process input by using an attention mechanism, and output of the second decoding submodule of the last decoding module comprises the prediction result of the to-be-predicted part.

[0021] In a second aspect, an embodiment of the present application provides a dialogue generation method, comprising:

[0022] obtaining a question and multi-modal data of a target object corresponding to the question;

[0023] inputting the question and the multi-modal data into a dialogue generation model to obtain a target reply to the question; and the dialogue generation model is trained by using the method in any one of claims 1-6.

[0024] In a third aspect, an embodiment of the present application provides a dialogue generation method, comprising:

[0025] obtaining a question;

[0026] determining a target object corresponding to the question;

[0027] obtaining multi-modal data of the target object;

[0028] inputting the question and the multi-modal data into a dialogue generation model to obtain a target reply to the question; and the dialogue generation model is trained by using the method in any one of claims 1-6.

[0029] outputting the target reply.

[0030] In a fourth aspect, an embodiment of the present application provides a dialogue generation method, comprising:

[0031] obtaining a question and multi-modal data of a target object corresponding to the question;

[0032] inputting the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain multi-modal feature representations perceived by the question;

[0033] inputting a candidate reply and the multi-modal feature representations perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question.

[0034] Further, the encoding layer obtains the question-aware multimodal feature representation by:

[0035] obtaining a first interaction from the question to the multimodal data and a second interaction from the multimodal data to the question by using a bidirectional attention mechanism;

[0036] obtaining the question-aware multimodal feature representation according to the first interaction and the second interaction.

[0037] Further, the decoding layer obtains the predicted result of the reply to the question by:

[0038] obtaining a reply feature representation corresponding to the reply by using an attention mechanism, and obtaining the predicted result of the reply based on the multimodal feature representation and the reply feature representation by using an attention mechanism.

[0039] Further, the encoding layer comprises a first set of encoding modules and at least one second set of encoding modules.

[0040] The first set of encoding modules comprises a plurality of first encoding modules connected in sequence, and the processing result of a previous first encoding module is output to a next first encoding module, and the input of a first first encoding module comprises the question, and the output of a last first encoding module comprises a question feature representation corresponding to the question.

[0041] Each first encoding module comprises a first encoding submodule, and the first encoding submodule performs attention mechanism processing on the input.

[0042] The at least one second set of encoding modules, wherein one second set of encoding modules corresponds to processing one kind of modal data in the multimodal data, and each second set of encoding modules comprises a plurality of second encoding modules connected in sequence; in one second set of encoding modules, the processing result of a previous second encoding module is output to a next second encoding module, the input of a first second encoding module is the corresponding one kind of modal data in the multimodal data, and the output of a last second encoding module comprises a question-aware multimodal feature representation.

[0043] The second encoding module comprises a second encoding submodule and a bidirectional attention mechanism module, the second encoding submodule performs attention mechanism processing on the input, and the bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and the processing result of the second encoding submodule.

[0044] Further, the decoding layer includes a set of decoding modules, and the set of decoding modules includes a plurality of decoding modules connected in sequence, an output of a previous decoding module being an input of a next decoding module, an input of a first decoding module including the reply, and an output of a last decoding module including a prediction result corresponding to the reply;

[0045] The decoding module includes a first decoding submodule and at least one second decoding submodule connected in sequence, the first decoding submodule processing an input by using an attention mechanism, the first decoding submodule of a first decoding module obtaining a reply feature representation corresponding to a known part and a to-be-predicted part in the reply, an input of the second decoding submodule including an output of the first decoding submodule or a previous second decoding submodule and the corresponding multi-modal feature representation, the second decoding submodule being configured to process the input by using the attention mechanism, and an output of the second decoding submodule of a last decoding module including the prediction result of the to-be-predicted part.

[0046] In a fifth aspect, an embodiment of the present application provides a dialogue generation method, which includes the following steps.

[0047] Obtaining a question;

[0048] Determining a target object corresponding to the question;

[0049] Obtaining multi-modal data of the target object;

[0050] Inputting the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain multi-modal feature representation perceived by the question;

[0051] Inputting a candidate reply and the multi-modal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question;

[0052] Outputting the target reply.

[0053] In a sixth aspect, an embodiment of the present application provides a training device of a dialogue generation model, which includes the following steps.

[0054] A first obtaining module is configured to obtain a training sample; the training sample includes a question in dialogue text, a reply to the question, and multi-modal data of a target object corresponding to the dialogue text;

[0055] A first input module is configured to input the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain multi-modal feature representation perceived by the question;

[0056] The second input module is configured to input the reply to the question and the multimodal feature representation of the question perception into a decoding layer of the dialogue generation model to obtain a prediction result of the reply to the question.

[0057] The training module is configured to train the dialogue generation model according to the prediction result of the question reply.

[0058] In a seventh aspect, an embodiment of the present application provides a dialogue generation device, which comprises:

[0059] The second acquisition module is configured to acquire a question and multimodal data of a target object corresponding to the question.

[0060] The third acquisition module is configured to input the question and the multimodal data into a dialogue generation model to acquire a target reply to the question; the dialogue generation model is trained by using the device of the sixth aspect.

[0061] In an eighth aspect, an embodiment of the present application provides a dialogue generation device, which comprises:

[0062] The fourth acquisition module is configured to acquire a question.

[0063] The first determination module is configured to determine a target object corresponding to the question.

[0064] The fifth acquisition module is configured to acquire multimodal data of the target object.

[0065] The sixth acquisition module is configured to input the question and the multimodal data into a dialogue generation model to acquire a target reply to the question; the dialogue generation model is trained by using the device of the sixth aspect.

[0066] The first output module is configured to output the target reply.

[0067] In a ninth aspect, an embodiment of the present application provides a dialogue generation device, which comprises:

[0068] The seventh acquisition module is configured to acquire a question and multimodal data of a target object corresponding to the question.

[0069] The third input module is configured to input the question and the multimodal data into an encoding layer of a dialogue generation model to obtain a multimodal feature representation perceived by the question.

[0070] The fourth input module is configured to input the candidate reply and the multimodal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question.

[0071] In a tenth aspect, an apparatus for dialogue generation is provided in embodiments of the present application, and the apparatus comprises:

[0072] An eighth obtaining module configured to obtain a question;

[0073] A second determining module configured to determine a target object corresponding to the question;

[0074] A ninth obtaining module configured to obtain multi-modal data of the target object;

[0075] A tenth obtaining module configured to input the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain multi-modal feature representation perceived by the question;

[0076] A fifth inputting module configured to input the candidate reply and the multi-modal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question;

[0077] A second outputting module configured to output the target reply.

[0078] The functions can be implemented by hardware, or by hardware executing software in conjunction with the relevant functions. The hardware or software includes one or more modules corresponding to the functions described above.

[0079] In one possible design, the apparatus includes a memory and a processor, where the memory is configured to store one or more computer instructions supporting the apparatus to perform the corresponding method described above, and the processor is configured to execute the computer instructions stored in the memory. The apparatus can further include a communication interface configured to enable the apparatus to communicate with other devices or communication networks.

[0080] In an eleventh aspect, an electronic device is provided in embodiments of the present application, which includes a memory, a processor, and a computer program stored in the memory, where the processor executes the computer program to implement the method of any of the aspects described above.

[0081] In a twelfth aspect, a computer-readable storage medium is provided in embodiments of the present application, which is configured to store computer instructions for the apparatus described above, and the computer instructions are executed by a processor to implement the steps of the method of any of the aspects described above.

[0082] In a thirteenth aspect, a computer program product is provided in embodiments of the present application, which includes computer instructions, and the computer instructions are executed by a processor to implement the steps of the method of any of the aspects described above.

[0083] The technical solutions provided in the embodiments of the present application can have the following beneficial effects:

[0084] In the training process of the dialogue generation model, in addition to training the dialogue generation model by using the relationship between the traditional question and the reply, the relationship between the multi-modal data of the target object to which the question is directed and the question, and the relationship between the correct reply and the multi-modal data are used to train the dialogue generation model, and a new model structure is constructed, so that the dialogue generation model can learn the multi-modal feature representation capable of perceiving the question in the training process, and learn the interaction between the reply and the multi-modal data, and the internal interaction of the multi-modal data based on the multi-modal feature representation capable of perceiving the question and the correct reply. In the above manner, the multi-modal data of the target object is collected, and the deep relationship among the question raised by the user to the target object, the reply to the question, and the multi-modal data of the target object is learned in the training process of the dialogue generation model, so as to avoid the situation that the dialogue generation model cannot give an accurate reply due to the missing attribute value of the target object to which the question is directed, thereby improving the prediction accuracy of the dialogue generation model and improving the user experience.

[0085] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0086] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of the non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0087] Figure 1 A flowchart of a training method of a dialogue generation model according to an embodiment of the present disclosure is shown;

[0088] Figure 2 A model structure diagram of a dialogue generation model according to an embodiment of the present disclosure is shown;

[0089] Figure 3 A diagram showing the usefulness of multi-modal data when a dialogue generation model according to an embodiment of the present disclosure processes a sample is shown;

[0090] Figure 4 A flowchart of a dialogue generation method according to an embodiment of the present disclosure is shown;

[0091] Figure 5 A flowchart of a dialogue generation method according to another embodiment of the present disclosure is shown;

[0092] Figure 6 A flowchart of a dialogue generation method according to another embodiment of the present disclosure is shown;

[0093] Figure 7a flowchart showing a dialogue generation method according to another embodiment of the present disclosure;

[0094] Figure 8 a schematic diagram showing an application scenario of a dialogue generation model according to an embodiment of the present disclosure;

[0095] Figure 9 is a structural schematic diagram of an electronic device suitable for implementing a training method and / or a dialogue generation method of a dialogue generation model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0096] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, portions unrelated to the description of the exemplary embodiments are omitted in the accompanying drawings for the sake of clarity.

[0097] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there is existence of the features, numbers, steps, actions, parts, or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, parts, or combinations thereof exist or are added.

[0098] It should also be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0099] As described in the background, in view of the technical problems existing in the existing task-oriented dialogue system, the present inventors found that there are a large amount of multi-modal data of objects in the information platform, such as picture descriptions, text descriptions and other modal data of objects, which contain a large amount of attribute information. Therefore, the present disclosure proposes a training method of a dialogue generation model through the above multi-modal data which can be widely obtained, and solves the technical problem that the task-oriented dialogue system cannot output accurate replies according to the questions due to the missing of attribute values by learning multi-modal data in the dialogue generation process.

[0100] The details of the embodiments of the present disclosure will be described in detail below through specific embodiments.

[0101] Figure 1 a flowchart showing a training method of a dialogue generation model according to an embodiment of the present disclosure. As shown in the figure, the training method of the dialogue generation model includes the following steps: Figure 1

[0102] In step S101, a training sample is obtained; the training sample includes a question in a dialogue text, a reply to the question, and multi-modal data of a target object corresponding to the dialogue text; ​

[0103] In step S102, the question and the multi-modal data are input into an encoding layer of a dialogue generation model to obtain a multi-modal feature representation perceiving the question.

[0104] In step S103, the reply to the question and the multi-modal feature representation perceiving the question are input into a decoding layer of the dialogue generation model to obtain a prediction result of the reply to the question.

[0105] In step S104, the dialogue generation model is trained according to the prediction result of the question reply.

[0106] In the embodiment, the dialogue generation model can generate a proper reply according to a question in a dialogue with a current user, that is, give a reply to a question of the user. The dialogue text can be pre-collected, and the dialogue text can include questions raised by the user for the same or different target objects and replies that should be given to the questions, etc. It can be understood that in some embodiments, the dialogue text collected from an actual application scenario (for example, a communication scenario between a user and a customer service in an e-commerce platform) can include a series of questions and replies given by a customer service personnel to the series of questions, and the questions and replies can be artificially screened and processed.

[0107] The target object can be an object to which a question and a reply in the dialogue text are directed, and the object can include a product or a service provided by a system platform, etc. For example, the target object can be a commodity, a meal, a ticket, a venue, a hotel room, etc. provided by an e-commerce platform. The question in the dialogue text can be information about the target object itself or information derived from the target object, etc., such as questions about the attributes of the target object (quality, color, appearance, material, etc.), a gift or a gift service related to the target object, etc. The reply in the training sample can be artificially processed, or can be directly extracted from an actual application scenario, or can be content obtained by artificially processing the reply extracted from an actual application scenario.

[0108] The multi-modal data can include, but is not limited to, a combination of one or more of text data, audio data, video data, picture data, voice data, etc. The multi-modal data of the target object can be obtained from a system platform providing the target object, for example, the multi-modal data can include a title, other text introduction, user evaluation, pictures, videos, voice, etc. of the target object on the system platform. While collecting the dialogue text, the corresponding multi-modal data of the target object on the system platform can also be collected together as a training sample.

[0109] In some embodiments, the dialogue generation model can be pre-constructed and then trained using the collected training samples. The dialogue generation model can employ a sequence generation model and process an input sequence based on an encoding-decoding layer structure in the sequence generation model to obtain an output sequence. The dialogue generation model can include an encoding layer and a decoding layer. The encoding layer converts the input sequence into a continuous representation, and then the decoding layer generates the output sequence based on the representation.

[0110] In this embodiment, the input sequence of the encoding layer can be a question in the dialogue text and multi-modal data, and the output sequence of the decoding layer can be a predicted result of a reply to the question. It should be noted that the multi-modal data can include data of multiple modalities, such as text, pictures, videos, audio, voice, and data in multiple dimensions. The encoding layer can process the above different modalities of data and the question separately, for example, the input question can be processed separately to obtain a question feature representation, and each modality of data in the multi-modal data can be processed separately to obtain an intermediate modality feature. Then, a multi-modal feature representation perceiving the question is obtained based on the question feature representation and the intermediate modality feature corresponding to each modality. The number of multi-modal feature representations is the same as the number of modalities of multi-modal data in the current training sample. For example, when the multi-modal data includes a picture and a text, the multi-modal feature representation includes two, which correspond to the picture and the text, respectively. The multi-modal feature representation perceiving the question can be understood as a vector feature representation obtained through the interaction between the question and the multi-modal feature. The multi-modal feature representation is equivalent to a vector representation obtained by encoding the question and the multi-modal data using an encoding method.

[0111] The processing process of the encoding layer is illustrated below.

[0112] Suppose the multi-modal data in the training sample includes a text and a picture, the encoding layer processes the input question to obtain a question feature representation, which represents the content of the question in the form of a vector feature; the encoding layer processes the input text to obtain a text feature representation, which represents the content of the text in the form of a vector feature; and the encoding layer processes the input picture to obtain a picture feature representation, which represents the content in the picture in the form of a vector feature. It should be noted that the processing of the encoding layer for the text and the picture does not have a front-back order and can be processed in any order or in parallel. The encoding layer also obtains a text feature representation perceiving the question based on the question feature representation and the text feature representation, which is one of the multi-modal feature representations. The encoding layer also obtains a picture feature representation perceiving the question based on the question feature representation and the picture feature representation, which is another of the multi-modal feature representations.

[0113] In some embodiments, the decoding layer can process the reply to obtain a reply feature representation corresponding to the reply, and can process the reply feature representation and the multi-modal feature representation to learn the interaction between the reply and the multi-modal data and the interaction within the multi-modal data, and thus obtain a prediction result of the reply. The prediction result can be used to represent the possibility that the reply is a correct reply. It can be understood that in the training process, the input of the decoding layer is the correct reply that has been labeled in the training sample, and the prediction result output by the dialogue generation model should represent that the input reply is 100% correct. However, since the dialogue generation model does not have good enough prediction ability at the beginning of training, the prediction result output may be different from the true result. By comparing the prediction result with the true result, the model parameters of the dialogue generation model are adjusted so that the prediction result output by the dialogue generation model next time can be closer to the true result.

[0114] It should be noted that when the multi-modal feature representation includes multiple cases, that is, the multi-modal data includes data of multiple modalities, the decoding layer can learn the interaction between the reply and the multi-modal data and the interaction within the multi-modal data by sequentially processing the multiple multi-modal feature representations. For example, the decoding layer can process the reply feature representation and one of the multi-modal feature representations to embed the reply feature representation into the multi-modal feature representation to obtain an intermediate feature, to learn the interaction between the reply and the corresponding multi-modal data. Then, the next multi-modal feature representation is read, and the intermediate feature is embedded into the next multi-modal feature representation to learn the interaction between the reply and the two modal data and the interaction within the two modal data.

[0115] The processing process of the decoding layer is described below.

[0116] Suppose the multi-modal data in the training sample includes text and pictures, the decoding layer processes the input reply to obtain a reply feature representation, which represents the content of the reply in the form of a vector feature. The decoding layer also reads the text feature representation output by the encoding layer and embeds the reply feature representation into the text feature representation to obtain an intermediate feature. Then, the picture feature representation output by the encoding layer is read, and the intermediate feature is embedded into the picture feature representation.

[0117] During the training process, through the above processing, the decoding layer can learn the interaction between the reply, the text and the picture, and the final obtained feature vector can obtain the prediction result of the reply through the input reply, text and picture. It should be noted that the processing order of the decoding layer for the text feature representation and the picture feature representation can also be the order of processing the picture feature representation first and then processing the text feature representation, that is, the text feature representation and the picture feature representation are not limited to any processing order. In the case of including other modal data, the multi-modal feature representations corresponding to the plurality of modal data can also be processed in any order.

[0118] In the training process of the dialogue generation model, in addition to training the dialogue generation model by using the relationship between the traditional question and the reply, the dialogue generation model is also trained by using the relationship between the multi-modal data of the target object to which the question is directed and the question, the relationship between the correct reply and the multi-modal data, etc. And by constructing a new model structure, the dialogue generation model can learn the multi-modal feature representation that can perceive the question during the training process, and learn the interaction between the reply and the multi-modal data, the internal interaction between the multi-modal data based on the perceived multi-modal feature representation of the question and the correct reply. Through the above manner, the multi-modal data of the target object is collected, and the deep relationship between the question raised by the user for the target object, the reply to the question and the multi-modal data of the target object is learned during the training process of the dialogue generation model, thereby avoiding the case that the dialogue generation model cannot give an accurate reply due to the missing attribute value of the target object to which the question is directed, thereby improving the prediction accuracy of the dialogue generation model and improving the user experience.

[0119] In an optional implementation of the present embodiment, the encoding layer obtains the question-perceived multi-modal feature representation in the following manner:

[0120] Obtaining a first interaction from the question to the multi-modal data and a second interaction from the multi-modal data to the question by using a bidirectional attention mechanism;

[0121] Obtaining the question-perceived multi-modal feature representation according to the first interaction and the second interaction.

[0122] In the optional implementation, the encoding layer can process the question by using the attention mechanism to obtain a vector representation corresponding to the question, i.e., a question feature representation. In the process of processing the question by using the attention mechanism, the question feature representation is obtained by learning the interaction between the question context. In the process of processing the multi-modal data, the encoding layer can also process the multi-modal data by using the attention mechanism to obtain an intermediate modal feature corresponding to the multi-modal data. Further, in order to learn the relationship between the multi-modal data and the question, the disclosure embodiments learn the interaction between the question and the multi-modal data by using a bidirectional attention mechanism, i.e., the bidirectional attention mechanism is performed between the question feature representation and the intermediate modal feature obtained by processing the multi-modal data by using the attention mechanism, and finally a multi-modal feature representation capable of perceiving the question is obtained, i.e., the encoding layer obtains a first interaction feature from the question to the multi-modal data and a second interaction feature from the multi-modal data to the question by using the bidirectional attention mechanism, and then obtains the multi-modal feature representation capable of perceiving the question according to the first interaction feature and the second interaction feature.

[0123] The common attention mechanism has the following three characteristics: (1) the attention weight is usually obtained by summarizing the text context into a fixed-length vector, and then extracting strong relevant information from the text context to answer the question; (2) in the text domain (corresponding to the multi-modal data in the disclosure embodiments), the attention weight is usually dynamic in time, where the attention weight of the current time step is a function of the previous time step participation vector; (3) it is usually one-way attention weight from the question to the text (corresponding to the multi-modal data in the disclosure embodiments). The bidirectional attention mechanism is different, which calculates the attention weight from the question to the text and from the text to the question, so that the attention weights calculated in the two directions can complement each other.

[0124] The disclosure embodiments design the attention mechanism from the question to the multi-modal data (such as text, picture, video, etc.) and from the multi-modal data to the question, so that the dialogue generation model can learn the interaction between the question and the multi-modal data in the training process to obtain the multi-modal feature representation capable of perceiving the question.

[0125] In an optional implementation of the present embodiment, the decoding layer obtains the predicted result of the reply to the question by the following manner:

[0126] The reply feature representation corresponding to the reply is obtained by using the attention mechanism, and the predicted result of the reply is obtained based on the multi-modal feature representation and the reply feature representation by using the attention mechanism.

[0127] In the optional implementation, the decoding layer first processes the reply by using the attention mechanism to obtain a reply feature representation, which represents the content in the reply by using a vector. Then, the decoding layer processes the reply feature representation and the multi-modal feature representation by using the attention mechanism, so that the dialogue generation model can learn the interaction between the reply and the multi-modal data in the training process. In addition, the decoding layer can also process the previous processing result (i.e., the processing result of the previous multi-modal feature) and the next multi-modal feature representation by using the attention mechanism, so that the dialogue generation model can also learn the interaction between the internal multi-modal data, and finally obtain the prediction result of the reply. The prediction result can be used to represent the possibility that the reply is a correct reply. It can be understood that in the training process, the input of the decoding layer is the correct reply that has been labeled in the training sample, and the prediction result output by the dialogue generation model should represent that the input reply is 100% correct. However, since the dialogue generation model does not have good enough prediction ability at the beginning of the training, the output prediction result may be different from the true result. By comparing the prediction result with the true result, the model parameters of the dialogue generation model are adjusted, so that the prediction result output by the dialogue generation model next time can be closer to the true result.

[0128] In an optional implementation of the embodiment, the encoding layer includes a first set of encoding modules and at least one second set of encoding modules.

[0129] The first set of encoding modules includes a plurality of first encoding modules connected in sequence, and the processing result of a previous first encoding module is output to a next first encoding module. The input of a first first encoding module includes the question, and the output of a last first encoding module includes a question feature representation corresponding to the question.

[0130] Each first encoding module includes a first encoding submodule that processes the input by using the attention mechanism.

[0131] The at least one second set of encoding modules, wherein a second set of encoding modules corresponds to processing one type of modal data in the multi-modal data, and each second set of encoding modules includes a plurality of second encoding modules connected in sequence. In a second set of encoding modules, the processing result of a previous second encoding module is output to a next second encoding module. The input of a first second encoding module is the corresponding one type of modal data in the multi-modal data, and the output of a last second encoding module includes a multi-modal feature representation that perceives the question.

[0132] The second encoding module includes a second encoding submodule and a bidirectional attention mechanism module. The second encoding submodule performs attention mechanism processing on the input. The bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and the processing result of the second encoding submodule.

[0133] In the optional implementation, the encoding layer can include a group of first encoding modules and one or more groups of second encoding modules. The group of first encoding modules includes a plurality of first encoding modules connected in sequence. In the plurality of first encoding modules connected in sequence, the output of the previous first encoding module is the input of the next first encoding module, the input of the first first encoding module is the question, and the output of the last first encoding module is the question feature representation. The structure of each first encoding module can be the same.

[0134] The second encoding module can include one or more groups, and each group of second encoding modules can correspond to one of the modalities in the multi-modal data. Each group of second encoding modules can include a plurality of second encoding modules connected in sequence. The output of the previous second encoding module is the input of the next second encoding module, the input of the first second encoding module is the corresponding one of the modal data, and the output of the last second encoding module is the multi-modal feature representation corresponding to the modality data.

[0135] Each second encoding module can include a second encoding submodule and a bidirectional attention mechanism module. In the plurality of second encoding modules connected in sequence, the second encoding submodule of the first second encoding module is used to perform attention mechanism processing on the input modality data to obtain the intermediate modality feature corresponding to the modality data. The bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and the intermediate modality feature. The processing result is used as the input of the next second encoding module, that is, the attention mechanism processing is performed by the second encoding submodule of the next second encoding module. The bidirectional attention mechanism module of the next second encoding module performs bidirectional attention mechanism processing on the question feature representation and the processing result of the second encoding submodule. The processing is performed sequentially until the last second encoding module completes the processing, and the multi-modal feature representation of the perceived question is output.

[0136] In an optional implementation of the embodiment, the decoding layer includes a group of decoding modules, and the group of decoding modules includes a plurality of decoding modules connected in sequence. The output of the previous decoding module is used as the input of the next decoding module. The input of the first decoding module includes the reply, and the output of the last decoding module includes the predicted result corresponding to the reply.

[0137] The decoding module comprises a first decoding submodule and at least one second decoding submodule connected in sequence, the first decoding submodule processes the input by using an attention mechanism, the first decoding submodule of the first decoding module obtains the reply feature representation corresponding to the known part and the to-be-predicted part in the reply, the input of the second decoding submodule comprises the output of the first decoding submodule or the previous second decoding submodule and the corresponding multi-modal feature representation, the second decoding submodule is used for processing the input by using an attention mechanism, and the output of the second decoding submodule of the last decoding module comprises the prediction result of the to-be-predicted part.

[0138] In the optional implementation, the decoding layer can comprise a group of decoding modules, and the group of decoding modules comprises decoding modules connected in sequence, the output of a previous decoding module is used as the input of a next decoding module, the input of the first decoding module is the reply in the training sample, and the output of the last decoding module is the current to-be-predicted part in the reply. It should be noted that the reply needs to be obtained through multiple rounds of decoding, and one round of decoding can obtain a part of the result in the reply. For example, when the reply output by the dialogue generation model is pre-set to be a text comprising n words or characters, each round of decoding needs to be processed through the above multiple decoding modules, and one round of decoding can output one word or character in the reply.

[0139] Each decoding module can comprise a first decoding submodule and at least one second decoding submodule connected in sequence. The input of the first decoding submodule is the reply or the output of the previous decoding module. Taking the first decoding submodule of the first decoding module as an example, the first decoding submodule processes the known part and the to-be-predicted part in the reply by using an attention mechanism with a mask, and the processing result is output to the subsequent second decoding submodule. It can be understood that the known part in the reply input into the first decoding submodule of the first decoding module is empty, that is, there is no known part. Each second decoding submodule corresponds to the multi-modal feature representation of one modality, and the second decoding submodule processes the output of the first decoding submodule or the previous second decoding submodule and the corresponding multi-modal feature representation by using an attention mechanism, and the processing result is output to the next second decoding submodule, which is processed in sequence until the last second decoding submodule; the output of the last second decoding submodule is used as the input of the first decoding submodule of the next decoding module, and the output of the second decoding submodule in the last decoding module is the prediction result of the to-be-predicted part to be output, for example, which can be used to represent whether the to-be-predicted part input together with the known part in the reply is correct. When the to-be-predicted part obtained through the current prediction is not the last part of the reply, the to-be-predicted part predicted to be correct is supplemented into the known part of the reply, and then used as the input of the next round of decoding.

[0140] Figure 2A model structure diagram of a dialogue generation model according to an embodiment of the present disclosure is shown. As shown in Figure 2 The dialogue generation model adopts a Transformer structure, including an encoding layer and a decoding layer.

[0141] (I) Encoding layer

[0142] The encoding layer includes N Q x first encoding modules connected in sequence, N I x second encoding modules 1 connected in sequence, and N T x second encoding modules 2 connected in sequence.

[0143] The training sample includes a question, a picture corresponding to a product to which the question is directed, text, and a standard reply corresponding to the question. The first encoding module is used to process the question in the input training sample, the second encoding module 1 is used to process the picture in the input training sample, and the second encoding module 2 is used to process the text in the input training sample. The first encoding module, the second encoding module 1, and the second encoding module 2 are described in detail below.

[0144] 1. First encoding module

[0145] After the question in the current training sample is mapped into a vector representation by a word embedding module, it is input into the first first encoding module.

[0146] The multiple first encoding modules connected in sequence have the same structure. Each first encoding module includes two layers. The first layer includes a muti-head attention sublayer and a residual connection and normalization sublayer (Add&Norm), and the second layer includes a feed forward network sublayer and a residual connection and normalization sublayer (Add&Norm). The output of the first layer is input into the second layer, the output of the second layer is the output of the first encoding module, and the output of the last first encoding module is the question feature representation corresponding to the question. The question feature representation is output to each second encoding module 1 and each second encoding module 2.

[0147] 2. Second encoding module 1

[0148] After the picture in the current training sample is processed by a target detection module (Fast-RCNN), a vector representation corresponding to the picture is obtained, and the vector representation is input into the first second encoding module 1.

[0149] The second encoding module 1 connected in sequence also has the same structure, each second encoding module 1 includes three layers, the first layer includes a multi-head attention mechanism sublayer (Muti-head Attention) and a residual connection and normalization sublayer (Add&Norm), the second layer includes a fully connected feedforward network sublayer (Feed Forward) and a residual connection and normalization sublayer (Add&Norm). The third layer includes a bidirectional attention sublayer (Interaction Co-Attention). The output of the first layer is taken as the input of the second layer, the output of the second layer is taken as the input of the third layer, and the input of the third layer includes, in addition to the output of the second layer, the question feature representation output by the last first encoding module. The output of the third layer is taken as the output of the second encoding module 1 and input to the next second encoding module 1, and the output of the last second encoding module 1 is a multi-modal feature representation corresponding to the picture.

[0150] 3、Second encoding module 2

[0151] After the text in the current training sample is mapped into a vector representation by a word embedding module (Word Embedding), it is input into the first second encoding module 2. It should be noted that the word embedding module here can be the same as or different from the word embedding module for processing the question.

[0152] The second encoding module 2 connected in sequence also has the same structure, each second encoding module 2 includes three layers, the first layer includes a multi-head attention mechanism sublayer (Muti-head Attention) and a residual connection and normalization sublayer (Add&Norm), the second layer includes a fully connected feedforward network sublayer (Feed Forward) and a residual connection and normalization sublayer (Add&Norm). The third layer includes a bidirectional attention sublayer (Interaction Co-Attention). The output of the first layer is taken as the input of the second layer, the output of the second layer is taken as the input of the third layer, and the input of the third layer includes, in addition to the output of the second layer, the question feature representation output by the last first encoding module. The output of the third layer is taken as the output of the second encoding module 2 and input to the next second encoding module 2, and the output of the last second encoding module 2 is a multi-modal feature representation corresponding to the text.

[0153] It should be noted that in the case of other modal data, a fourth encoding module, a fifth encoding module, etc. can also be included, which can be selected according to actual conditions, and will not be described here.

[0154] (II) Decoding layer

[0155] The decoding layer includes N d x decoding modules connected in sequence. Each decoding module includes four layers.

[0156] After the reply in the current training sample is mapped into a vector representation by the word embedding module, the vector representation is input into a first layer of the first decoding module.

[0157] The first layer includes a masked muti-head attention sub-layer and a residual connection and normalization sub-layer, the second layer includes a muti-head attention sub-layer and a residual connection and normalization sub-layer, the third layer includes a muti-head attention sub-layer and a residual connection and normalization sub-layer, and the fourth layer includes a feed forward sub-layer and a residual connection and normalization sub-layer.

[0158] Taking the first decoding module as an example, the input of the first layer includes the vector representation corresponding to the reply. The muti-head attention sub-layer in the first layer shields the content after the current to-be-predicted part in the reply by a mask, and only performs attention mechanism processing on the vector representations corresponding to the known part and the current to-be-predicted part in the reply. The input of the first layer in other decoding modules is the output of the previous decoding module.

[0159] The processing result of the first layer is output to the second layer. In addition to the output of the first layer, the input of the second layer also includes the multi-modal feature representation corresponding to the picture. The muti-head attention sub-layer in the second layer learns the information related to the reply in the picture by embedding the output of the first layer into the multi-modal feature representation corresponding to the picture.

[0160] The processing result of the second layer is output to the third layer. In addition to the output of the second layer, the input of the third layer also includes the multi-modal feature representation corresponding to the text. The muti-head attention sub-layer in the third layer learns the information related to the reply in the text by embedding the output of the second layer into the multi-modal feature representation corresponding to the text.

[0161] The processing result of the third layer is output to the fourth layer. Taking the last decoding module as an example, the fourth layer outputs the prediction result of the current to-be-predicted part, which can be used as the known part in the reply input into the next decoding module. It should be noted that, in the training process, since the reply is known, and the prediction result of the current to-be-predicted part output by the fourth layer may be different from the true result, the prediction result can not be input into the next decoding module, but the reply in the training sample can be directly used. The output of the fourth layer in other decoding modules is used as the input of the next decoding module.

[0162] After the processing of multiple decoding modules, the output vector can obtain the prediction result of the current predicted part in the reply after passing through the full connection layer (Linear) and the multi-classification function (Softmax). Figure 2 The number N of decoding modules in the decoding layer is shown d x = 3, after three hops Hop1, Hop2 and Hop3, the prediction structure of the current predicted part in the reply can be obtained. For each part in the reply, the above multiple decoding modules are used for decoding, and the complete prediction result corresponding to the entire reply can be obtained. By fitting the gap between the complete prediction result and the true result, the model parameters in the decoding layer and the encoding layer can be adjusted by the back propagation method. After multiple rounds of training of multiple training samples, the model parameters are in a convergent state, and then the trained dialogue generation model can be obtained.

[0163] Table 1 shows the comparison data of the effects of the dialogue generation model (MIND) proposed in the embodiment of the present disclosure and multiple existing models under different indicators for two dialogue task data sets.

[0164] Table 1

[0165]

[0166] As can be seen from Table 1, the effects of the dialogue generation model proposed in the embodiment of the present disclosure on various indicators are far superior to the currently more popular known models. Artificial evaluation also shows that the replies generated by the dialogue generation model (MIND) proposed in the embodiment of the present disclosure have high readability and relevance. At the same time, through the multi-modal modeling method proposed in the embodiment of the present disclosure, the information concerned by the question can be learned, and important information can be extracted from the multi-modal data to generate corresponding replies.

[0167] Figure 3 A schematic diagram showing the usefulness of multi-modal data when the dialogue generation model according to the embodiment of the present disclosure processes a sample is shown. As Figure 3 shown, for picture and text content, by analyzing the multi-modal data, it can be found that the multi-modal data contains rich attribute information, such as the categories "dress", "white", "with pendant", "brand new", etc. The dialogue generation model trained according to the embodiment of the present disclosure can extract key information from these multi-modal data, and generate appropriate replies based on the key information and the question. The replies generated by the existing models and only using one kind of modal data such as text or picture are not very accurate.

[0168] Figure 4 A flowchart showing a dialogue generation method according to an embodiment of the present disclosure is shown. As Figure 4 shown, the dialogue generation method includes the following steps:

[0169] In step S401, a question and multi-modal data of a target object corresponding to the question are obtained.

[0170] In step S402, the question and the multi-modal data are input into a dialogue generation model to obtain a target reply to the question; the dialogue generation model is trained by using a training method of the dialogue generation model.

[0171] In this embodiment, an execution body of the dialogue generation model can obtain a question to be generated a reply and multi-modal data of a target object corresponding to the question. For example, the target object can be a certain commodity in an e-commerce platform, and the multi-modal data can be a theme picture and a title of the commodity on the e-commerce platform. The question and the multi-modal data can be input into the dialogue generation model, and the dialogue generation model outputs a corresponding target reply.

[0172] The specific details of the dialogue generation model can be referred to the description of the training method of the dialogue generation model above, which will not be repeated here.

[0173] In the training process of the dialogue generation model in the embodiment of the present disclosure, the multi-modal data is used in the dialogue generation task, and a training task for the interaction between the multi-modal data is designed, which improves the multi-modal interaction ability of the dialogue generation model. Therefore, by using the trained dialogue generation model, a target reply with better readability and relevance can be obtained based on the input question and multi-modal data.

[0174] Figure 5 A flowchart of a dialogue generation method according to another embodiment of the present disclosure is shown. As shown in the flowchart, the dialogue generation method includes the following steps: Figure 5

[0175] In step S501, a question is obtained.

[0176] In step S502, a target object corresponding to the question is determined.

[0177] In step S503, multi-modal data of the target object is obtained.

[0178] In step S504, the question and the multi-modal data are input into a dialogue generation model to obtain a target reply to the question; the dialogue generation model is trained by using a training method of the dialogue generation model.

[0179] In step S505, the target reply is output.

[0180] ​In this embodiment, the dialogue generation system can obtain a question from a user, and the dialogue generation system can also obtain multi-modal data of a target object according to the current obtained question, for example, the target object corresponds to a certain commodity in an e-commerce platform, and the dialogue generation system can obtain multi-modal data such as pictures and titles of the commodity. The dialogue generation system can also input the question and the multi-modal data into the dialogue generation model, and output a corresponding target reply from the dialogue generation model, and the dialogue generation system returns the target reply to the user.

[0181] The specific details of the dialogue generation model can be referred to the description of the training method of the dialogue generation model described above, which will not be repeated here.

[0182] In the dialogue generation model in the embodiments of the present disclosure, during the training process, the multi-modal data is used in the dialogue generation task, and a training task for the interaction between the multi-modal data is designed, which improves the multi-modal interaction capability of the dialogue generation model. Therefore, by using the trained dialogue generation model, a target reply with better readability and relevance can be obtained based on the input question and multi-modal data.

[0183] Figure 6 A flowchart of a dialogue generation method according to another embodiment of the present disclosure is shown. As shown in Figure 6 The dialogue generation method includes the following steps:

[0184] In step S601, a question and multi-modal data of a target object corresponding to the question are obtained;

[0185] In step S602, the question and the multi-modal data are input into an encoding layer of a dialogue generation model to obtain a multi-modal feature representation perceived by the question;

[0186] In step S603, the candidate reply and the multi-modal feature representation perceived by the question are input into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question.

[0187] In this embodiment, the dialogue generation model can generate appropriate replies according to the questions in the dialogue with the current user, that is, give replies to the questions of the user.

[0188] The target object can be an object to which the question and the reply are directed, which can include products or services provided by a system platform, etc. For example, the target object can be a commodity, a meal, a ticket, a venue, a hotel room, etc. provided by an e-commerce platform. The question can be information about the target object itself or information derived from the target object, etc., such as questions about the attributes of the target object (quality, color, appearance, material, etc.), related gifts or services, etc.

[0189] The multi-modal data can include, but is not limited to, a combination of one or more of text data, audio data, video data, picture data, speech data, etc. The multi-modal data of the target object can be obtained from a system platform providing the target object, for example, the multi-modal data can include a title, other text introduction, user evaluation, picture, video, speech, etc. of the target object on the system platform.

[0190] The target reply can be a reply output by the question dialogue generation model. The candidate reply can be a candidate answer of a part currently to be predicted in the target reply. In some embodiments, the candidate reply can be all or part of the words in the candidate word set, and the prediction result output by the decoding layer is used to represent whether the current candidate word is the correct word corresponding to the part currently to be predicted in the target reply.

[0191] The dialogue generation model can adopt a sequence generation model, and obtain an output sequence based on the encoding-decoding layer structure in the sequence generation model after processing an input sequence. The dialogue generation model can include an encoding layer and a decoding layer. The encoding layer converts the input sequence into a continuous representation, and then the decoding layer generates an output sequence based on the representation.

[0192] In this embodiment, the input sequence of the encoding layer can be the question and the multi-modal data, and the output sequence of the decoding layer can be the target reply to the question. It should be noted that the multi-modal data can include data of multiple modalities, such as text, picture, video, audio, speech, etc. in multiple dimensions. The encoding layer can process the above different modalities of data and the question separately, for example, the input question can be processed separately to obtain a question feature representation, each modality of data in the multi-modal data can be processed separately to obtain an intermediate modality feature, and then a multi-modal feature representation perceiving the question can be obtained based on the question feature representation and the intermediate modality feature corresponding to each modality. The number of multi-modal feature representations is the same as the number of modalities of multi-modal data, for example, when the multi-modal data includes a picture and a text, the multi-modal feature representation includes two, which correspond to the picture and the text respectively. The multi-modal feature representation perceiving the question can be understood as a vector feature representation obtained through the interaction between the question and the multi-modal feature. The vector feature representation is obtained by embedding the question into the multi-modal data, and the multi-modal feature representation is equivalent to a vector representation obtained by encoding the question and the multi-modal data in a certain encoding manner.

[0193] The processing process of the encoding layer is illustrated below.

[0194] If the multi-modal data includes text and picture, the encoding layer processes the input question to obtain a question feature representation, which represents the content of the question in the form of vector features; the encoding layer processes the input text to obtain a text feature representation, which represents the content of the text in the form of vector features; and the encoding layer also processes the input picture to obtain a picture feature representation, which represents the content of the picture in the form of vector features. It should be noted that the processing of the encoding layer on the text and the picture is not in a specific order, and can be processed in any order or in parallel. The encoding layer also obtains a text feature representation for perceiving the question based on the question feature representation and the text feature representation, i.e., one of the multi-modal feature representations, and obtains a content feature representation for perceiving the question based on the question feature representation and the picture feature representation, i.e., another of the multi-modal feature representations.

[0195] In some embodiments, the decoding layer can process the reply to obtain a reply feature representation corresponding to the candidate reply, and the decoding layer can also process the reply feature representation and the multi-modal feature representation to be able to learn the interaction between the candidate reply and the multi-modal data and the interaction between the internal multi-modal data, and thus obtain a prediction result of the candidate reply. The prediction result can be used to represent the possibility that the candidate reply is a correct reply. It can be understood that in the dialogue generation model, the output of the decoding layer at one time can be one word in the target reply, so for the current to-be-predicted part in the target reply, the candidate reply can be all or part of the words in the pre-set word set, and the dialogue generation model can output a prediction result for each candidate word in the candidate reply. The prediction result can be a probability value, which represents the probability that the candidate word is the correct word corresponding to the current to-be-predicted part, and the word with the highest probability value can be output as the correct word of the current to-be-predicted part.

[0196] It should be noted that when the multi-modal feature representation includes multiple, i.e., when the multi-modal data includes data of multiple modalities, the decoding layer can learn the interaction between the candidate reply and the multi-modal data and the interaction between the internal multi-modal data by sequentially processing the multiple multi-modal feature representations, for example, the decoding layer can process the reply feature representation and one of the multi-modal feature representations to embed the reply feature representation in the multi-modal feature representation to learn the interaction between the candidate reply and the corresponding multi-modal data, and then read the next multi-modal feature representation and embed the intermediate feature in the next multi-modal feature representation to learn the interaction between the candidate reply and the above two modal data and the interaction between the internal two modal data.

[0197] The processing process of the decoding layer is described below.

[0198] If the multi-modal data in the training sample includes text and pictures, the decoding layer processes the input candidate reply to obtain a reply feature representation, which represents the content of the candidate reply in the form of a vector feature; the decoding layer also reads the text feature representation output by the encoding layer, and embeds the reply feature representation into the text feature representation to obtain an intermediate feature, and then reads the picture feature representation output by the encoding layer, and embeds the intermediate feature into the picture feature representation.

[0199] During the training process, through the above processing, the decoding layer can learn the interaction between the candidate reply, the text and the picture, and the final obtained feature vector can obtain the prediction result of the candidate reply through the input candidate reply, the text and the picture. It should be noted that the processing order of the decoding layer for the text feature representation and the picture feature representation can also be the order of processing the picture feature representation first and then processing the text feature representation, that is, the text feature representation and the picture feature representation are not limited to any processing order, and in the case of including other modal data, the multi-modal feature representations corresponding to the plurality of modal data can also be processed in any order.

[0200] In the dialogue generation process, the embodiment of the present disclosure filters the target reply matched with the question from the candidate reply by using the relationship between the multi-modal data of the target object to which the question is directed and the question, and the relationship between the multi-modal data. Through the above manner, the dialogue generation model of the present disclosure can obtain a target reply with better readability and relevance to the question of the target object, avoiding the situation that an accurate reply cannot be given due to the missing attribute value of the target object to which the question is directed, thereby improving the user experience.

[0201] In an optional implementation of the embodiment, the encoding layer obtains the question-aware multi-modal feature representation in the following manner:

[0202] The first interaction from the question to the multi-modal data and the second interaction from the multi-modal data to the question are obtained by using a bidirectional attention mechanism;

[0203] The question-aware multi-modal feature representation is obtained according to the first interaction and the second interaction.

[0204] In an optional implementation of the embodiment, the decoding layer obtains the prediction result of the reply to the question in the following manner:

[0205] The reply feature representation corresponding to the reply is obtained by using an attention mechanism, and the prediction result of the reply is obtained based on the multi-modal feature representation and the reply feature representation by using an attention mechanism.

[0206] In an optional implementation of the embodiment, the encoding layer comprises a first set of encoding modules and at least one second set of encoding modules.

[0207] The first set of encoding modules comprises a plurality of first encoding modules connected in sequence, and a processing result of a previous first encoding module is output to a next first encoding module, and an input of a first first encoding module comprises the question, and an output of a last first encoding module comprises a question feature representation corresponding to the question.

[0208] Each first encoding module comprises a first encoding submodule, and the first encoding submodule performs attention mechanism processing on an input.

[0209] In the at least one second set of encoding modules, a second set of encoding modules corresponds to processing one kind of modal data in the multi-modal data, and each second set of encoding modules comprises a plurality of second encoding modules connected in sequence; in a second set of encoding modules, a processing result of a previous second encoding module is output to a next second encoding module, an input of a first second encoding module is the corresponding one kind of modal data in the multi-modal data, and an output of a last second encoding module comprises a multi-modal feature representation perceiving the question.

[0210] The second encoding module comprises a second encoding submodule and a bidirectional attention mechanism module, the second encoding submodule performs attention mechanism processing on an input, and the bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and a processing result of the second encoding submodule.

[0211] In an optional implementation of the embodiment, the decoding layer comprises a set of decoding modules, and the set of decoding modules comprises a plurality of decoding modules connected in sequence, an output of a previous decoding module is used as an input of a next decoding module, an input of a first decoding module comprises the reply, and an output of a last decoding module comprises a prediction result corresponding to the reply.

[0212] The decoding module comprises a first decoding submodule and at least one second decoding submodule connected in sequence, the first decoding submodule performs processing on an input by using an attention mechanism, the first decoding submodule of a first decoding module obtains a reply feature representation corresponding to a known part and a to-be-predicted part in the reply, an input of the second decoding submodule comprises an output of the first decoding submodule or a previous second decoding submodule and the corresponding multi-modal feature representation, the second decoding submodule is used for performing attention mechanism processing on an input, and an output of the second decoding submodule of a last decoding module comprises the prediction result of the to-be-predicted part.

[0213] The specific details in the above optional implementation manner can be referred to the description of the training method of the dialogue generation model, which will not be described here. It should be noted that each decoding in the decoding layer can obtain a part of the target reply, for example, a word. The word obtained by the last decoding can be used as the known part in the target reply. In the next decoding, all known parts in the target reply and the candidate answer corresponding to the current to-be-predicted part are jointly used as the input of the first decoding module in the decoding layer, and the output of the last decoding module is the prediction result corresponding to the current to-be-predicted part.

[0214] Figure 7 A flowchart of a dialogue generation method according to another embodiment of the present disclosure is shown. As shown in the flowchart, the dialogue generation method comprises the following steps: Figure 7

[0215] In step S701, the question is obtained.

[0216] In step S702, the target object corresponding to the question is determined.

[0217] In step S703, the multi-modal data of the target object is obtained.

[0218] In step S704, the question and the multi-modal data are input into the encoding layer of the dialogue generation model to obtain the multi-modal feature representation perceived by the question.

[0219] In step S705, the candidate reply and the multi-modal feature representation perceived by the question are input into the decoding layer of the dialogue generation model to obtain the prediction result of whether the candidate reply is the target reply of the question.

[0220] In step S706, the target reply is output.

[0221] In this embodiment, the dialogue generation system can obtain the question from the user, and the dialogue generation system can also obtain the multi-modal data of the target object according to the question. For example, the target object corresponds to a certain commodity in the e-commerce platform, and the dialogue generation system can obtain the multi-modal data such as the picture and title of the commodity. The dialogue generation system can also input the question and the multi-modal data into the dialogue generation model, and output the corresponding target reply. The dialogue generation system returns the target reply to the user.

[0222] The specific details of the dialogue generation model can be referred to the description of the training method of the dialogue generation model, which will not be described here.

[0223] ​The dialogue generation model in the embodiment of the present disclosure, in the training process, since the multi-modal data is used in the dialogue generation task, a training task for the interaction between the multi-modal data is designed, and the multi-modal interaction capability of the dialogue generation model is improved. Therefore, by using the trained dialogue generation model, a target reply with better readability and relevance can be obtained based on the input question and multi-modal data.

[0224] Figure 8 The application scenario of the dialogue generation model according to an embodiment of the present disclosure is shown. As shown in the figure, Figure 8 the server trains the dialogue generation model by using the collected training samples, and the trained dialogue generation model is configured in the cloud and called by the dialogue generation system such as the online customer service module in the e-commerce platform.

[0225] In the process of selecting goods in the second-hand shopping platform, the user enters the online customer service dialogue box by clicking the contact customer service button on the page of a certain good, inputs a question about the good in the dialogue box, the online customer service module in the platform obtains the multi-modal data of the good including the good picture and the good title through the link on the page of the good, and sends the question of the user and the multi-modal data to the dialogue generation model in the cloud. After processing by the dialogue generation model, the corresponding reply is returned, and the online customer service module displays the reply in the online customer service dialogue box.

[0226] Figure 8 The embodiment shown is only an example, and the application scenario of the embodiment of the present disclosure is not limited to the above-mentioned second-hand shopping platform. The dialogue generation model in the embodiment of the present disclosure can be deployed in the cloud and used by various e-commerce platforms, such as ordering platforms, ticketing platforms, hotels, group buying platforms and other e-commerce platforms. The dialogue generation model provided by the embodiment of the present disclosure can improve the accuracy of automatic reply to questions of the platform, save the labor cost of the platform, and improve the user experience.

[0227] The dialogue generation model and / or dialogue generation method in the embodiment of the present disclosure can be used in all small bees, robots, Tmall Genie, etc., to realize human-computer intelligent dialogue. In addition, the dialogue generation model and / or dialogue generation method can also be deployed in the cloud for partners to use.

[0228] The dialogue generation model and / or dialogue generation method in the embodiment of the present disclosure can be applied to scenarios including but not limited to finance, weather, new retail - discount combination, order, education (score, timetable), meeting, etc. to facilitate users to quickly and accurately find the information they need through human-computer dialogue and meet the needs of users.

[0229] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure.

[0230] The training device of the dialogue generation model according to an embodiment of the present disclosure can be implemented as part or all of an electronic device through software, hardware or a combination of both. The training device of the dialogue generation model comprises:

[0231] a first obtaining module configured to obtain training samples; the training samples comprise a question in a dialogue text, a reply to the question and multi-modal data of a target object corresponding to the dialogue text;

[0232] a first input module configured to input the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain multi-modal feature representation perceived by the question;

[0233] a second input module configured to input the reply to the question and the multi-modal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a predicted result of the reply to the question;

[0234] a training module configured to train the dialogue generation model according to the predicted result of the question reply.

[0235] In an optional implementation of the present embodiment, the encoding layer obtains the multi-modal feature representation perceived by the question in the following manner:

[0236] a first interaction from the question to the multi-modal data and a second interaction from the multi-modal data to the question are obtained by using a bidirectional attention mechanism;

[0237] the multi-modal feature representation perceived by the question is obtained according to the first interaction and the second interaction.

[0238] In an optional implementation of the present embodiment, the decoding layer obtains the predicted result of the reply to the question in the following manner:

[0239] a reply feature representation corresponding to the reply is obtained by using an attention mechanism, and the predicted result of the reply is obtained based on the multi-modal feature representation and the reply feature representation by using the attention mechanism.

[0240] In an optional implementation of the present embodiment, the encoding layer comprises a first set of encoding modules and at least a second set of encoding modules;

[0241] the first set of encoding modules comprises a plurality of first encoding modules connected in sequence, the processing result of a previous first encoding module is output to a next first encoding module, the input of a first first encoding module comprises the question, and the output of a last first encoding module comprises question feature representation corresponding to the question;

[0242] Each of the first encoding modules comprises a first encoding submodule, and the first encoding submodule performs attention mechanism processing on an input;

[0243] In the at least one group of second encoding modules, one group of the second encoding modules corresponds to processing one kind of modality data in the multi-modal data, and each group of the second encoding modules comprises a plurality of second encoding modules connected in sequence; in one group of the second encoding modules, a processing result of a previous second encoding module is output to a next second encoding module, an input of a first second encoding module is the corresponding one kind of modality data in the multi-modal data, and an output of a last second encoding module comprises a multi-modal feature representation perceiving the question;

[0244] The second encoding module comprises a second encoding submodule and a bidirectional attention mechanism module, the second encoding submodule performs attention mechanism processing on an input, and the bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and a processing result of the second encoding submodule.

[0245] In an optional implementation of the embodiment, the decoding layer comprises a group of decoding modules, and one group of the decoding modules comprises a plurality of decoding modules connected in sequence, an output of a previous decoding module is used as an input of a next decoding module, an input of a first decoding module comprises the reply, and an output of a last decoding module comprises a predicted result corresponding to the reply.

[0246] The decoding module comprises a first decoding submodule and at least one second decoding submodule connected in sequence, the first decoding submodule performs processing on an input by using attention mechanism, the first decoding submodule of a first decoding module obtains a reply feature representation corresponding to a known part and a to-be-predicted part in the reply, an input of the second decoding submodule comprises an output of the first decoding submodule or a previous second decoding submodule and the corresponding multi-modal feature representation, the second decoding submodule is configured to perform attention mechanism processing on an input, and an output of the second decoding submodule of a last decoding module comprises the predicted result of the to-be-predicted part.

[0247] The training device of the dialogue generation model in the embodiment corresponds to the training method of the dialogue generation model, and specific details can be referred to the description of the training method of the dialogue generation model.

[0248] The dialogue generation device according to an embodiment of the present disclosure can be realized as part or all of an electronic device by software, hardware, or a combination of both. The dialogue generation device comprises:

[0249] The second obtaining module is configured to obtain a question and multi-modal data of a target object corresponding to the question.

[0250] The third obtaining module is configured to input the question and the multi-modal data into a dialogue generation model to obtain a target reply to the question.

[0251] The dialogue generation device in the embodiment is consistent with the dialogue generation method in the embodiment and related embodiments shown in the above Figure 4 The dialogue generation method in the embodiment and related embodiments shown in the above Figure 4 The dialogue generation method in the embodiment and related embodiments shown in the above

[0252] According to the dialogue generation device of another embodiment of the present disclosure, the device can be realized by software, hardware or a combination of both to become part or all of an electronic device. The dialogue generation device comprises:

[0253] The fourth obtaining module is configured to obtain a question.

[0254] The first determining module is configured to determine a target object corresponding to the question.

[0255] The fifth obtaining module is configured to obtain multi-modal data of the target object.

[0256] The sixth obtaining module is configured to input the question and the multi-modal data into a dialogue generation model to obtain a target reply to the question. The dialogue generation model is trained by the training device of the dialogue generation model.

[0257] The first output module is configured to output the target reply.

[0258] The dialogue generation device in the embodiment is consistent with the dialogue generation method in the embodiment and related embodiments shown in the above Figure 5 The dialogue generation method in the embodiment and related embodiments shown in the above Figure 5 The dialogue generation method in the embodiment and related embodiments shown in the above

[0259] According to the dialogue generation device of another embodiment of the present disclosure, the device can be realized by software, hardware or a combination of both to become part or all of an electronic device. The dialogue generation device comprises:

[0260] The seventh obtaining module is configured to obtain a question and multi-modal data of a target object corresponding to the question.

[0261] a third input module configured to input the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain a question-aware multi-modal feature representation;

[0262] a fourth input module configured to input the candidate reply and the question-aware multi-modal feature representation into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question.

[0263] In an optional implementation of the embodiment, the encoding layer obtains the question-aware multi-modal feature representation in the following manner:

[0264] a first interaction from the question to the multi-modal data and a second interaction from the multi-modal data to the question are obtained by using a bidirectional attention mechanism;

[0265] the question-aware multi-modal feature representation is obtained according to the first interaction and the second interaction.

[0266] In an optional implementation of the embodiment, the decoding layer obtains the prediction result of the reply to the question in the following manner:

[0267] a reply feature representation corresponding to the reply is obtained by using an attention mechanism, and the prediction result of the reply is obtained based on the multi-modal feature representation and the reply feature representation by using the attention mechanism.

[0268] In an optional implementation of the embodiment, the encoding layer includes a first set of encoding modules and at least one second set of encoding modules.

[0269] The first set of encoding modules includes a plurality of first encoding modules connected in sequence, a processing result of a previous first encoding module is output to a next first encoding module, an input of a first first encoding module includes the question, and an output of a last first encoding module includes a question feature representation corresponding to the question.

[0270] Each first encoding module includes a first encoding submodule that performs attention mechanism processing on an input.

[0271] The at least one second set of encoding modules, wherein a second set of encoding modules in the at least one second set of encoding modules processes one type of modal data in the multi-modal data, and each second set of encoding modules includes a plurality of second encoding modules connected in sequence; in a second set of encoding modules, a processing result of a previous second encoding module is output to a next second encoding module, an input of a first second encoding module is the one type of modal data in the multi-modal data, and an output of a last second encoding module includes a question-aware multi-modal feature representation.

[0272] The second encoding module comprises a second encoding submodule and a bidirectional attention mechanism module, the second encoding submodule performs attention mechanism processing on the input, and the bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and the processing result of the second encoding submodule.

[0273] In an optional implementation of the embodiment, the decoding layer comprises a plurality of decoding modules connected in sequence, the output of a previous decoding module is used as the input of a next decoding module, the input of a first decoding module comprises the reply, and the output of a last decoding module comprises the predicted result corresponding to the reply.

[0274] The decoding module comprises a first decoding submodule and at least one second decoding submodule connected in sequence, the first decoding submodule performs processing on the input by using an attention mechanism, the first decoding submodule of a first decoding module obtains the question feature representation corresponding to the known part and the to-be-predicted part in the reply, the input of the second decoding submodule comprises the output of the first decoding submodule or a previous second decoding submodule and the corresponding multi-modal feature representation, the second decoding submodule is configured to perform attention mechanism processing on the input, and the output of the second decoding submodule of a last decoding module comprises the predicted result of the to-be-predicted part.

[0275] The dialogue generation apparatus in the embodiment corresponds to the dialogue generation method in the above Figure 6 embodiment and related embodiments, and specific details can be referred to the description of the dialogue generation method in the above Figure 6 embodiment and related embodiments, which will not be described herein again.

[0276] According to the dialogue generation apparatus of another embodiment of the present disclosure, the apparatus can be realized by software, hardware or a combination of both to become part or all of an electronic device. The dialogue generation apparatus comprises:

[0277] An eighth obtaining module configured to obtain a question;

[0278] A second determining module configured to determine a target object corresponding to the question;

[0279] A ninth obtaining module configured to obtain multi-modal data of the target object;

[0280] A tenth obtaining module configured to input the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain a multi-modal feature representation perceived by the question;

[0281] The fifth input module is configured to input the candidate reply and the multimodal feature representation perceived by the question to a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question.

[0282] The second output module is configured to output the target reply.

[0283] The dialogue generation apparatus in the embodiment corresponds to the dialogue generation method in the above Figure 7 embodiments and related embodiments, and specific details can be referred to the description of the dialogue generation method in the above Figure 7 embodiments and related embodiments, which will not be repeated here.

[0284] Figure 9 is a structural schematic diagram of an electronic device suitable for implementing the training method and / or dialogue generation method according to the dialogue generation model of the embodiments of the present disclosure.

[0285] As shown in Figure 9 , the electronic device 900 includes a processing unit 901, which can be implemented as a CPU, a GPU, an FPGA, an NPU, etc. The processing unit 901 can perform various processes in the embodiments of any of the methods of the present disclosure according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the electronic device 900 are also stored in the RAM 903. The processing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0286] The following components are connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 910 as necessary, so that a computer program read therefrom is installed in the storage portion 908 as necessary.

[0287] In particular, according to an embodiment of the present disclosure, the above-mentioned method can be implemented as a computer software program with reference to any one of the embodiments of the present disclosure. For example, an embodiment of the present disclosure includes a computer program product including a computer program tangibly embodied on a computer readable medium, the computer program containing program code for executing any one of the methods of the embodiments of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network by the communication section 909, and / or installed from the detachable medium 911.

[0288] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0289] The units or modules described in the embodiments of the present disclosure can be implemented by software, or by hardware. The described units or modules can also be arranged in a processor, and the names of the units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0290] As another aspect, the present disclosure also provides a computer readable storage medium, which can be the computer readable storage medium included in the apparatus described in the above embodiments, or can exist separately from the apparatus and not be assembled into the apparatus. The computer readable storage medium stores one or more programs for execution by one or more processors to perform the methods described in the present disclosure.

[0291] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the inventive scope involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the present disclosure (but not limited to) without departing from the inventive concept.

Claims

1. A method of training a dialogue generation model, wherein, The method comprises: obtaining a training sample; the training sample comprises a question in a dialogue text, a reply to the question, and multi-modal data of a target object corresponding to the dialogue text, wherein the multi-modal data comprises at least two of the following: text, picture, video, audio, and voice; inputting the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain a question-aware multi-modal feature representation; inputting the reply to the question and the question-aware multi-modal feature representation into a decoding layer of the dialogue generation model to obtain interactions between the reply to the question and the multi-modal data, and interactions between the multi-modal data, and further obtain a prediction result of the reply to the question; training the dialogue generation model according to the prediction result of the reply to the question; wherein the decoding layer comprises a group of decoding modules, and the group of decoding modules comprises a plurality of decoding modules connected in sequence, the output of a previous decoding module is used as the input of a next decoding module, the input of a first decoding module comprises the reply, and the output of a last decoding module comprises the prediction result corresponding to the reply.

2. The method of claim 1, wherein, The encoding layer obtains the question-aware multi-modal feature representation in the following manner: using a bidirectional attention mechanism to obtain a first interaction from the question to the multi-modal data, and a second interaction from the multi-modal data to the question; obtaining the question-aware multi-modal feature representation according to the first interaction and the second interaction.

3. The method of claim 1, wherein, The decoding layer obtains the prediction result of the reply to the question in the following manner: using an attention mechanism to obtain a reply feature representation corresponding to the reply, and using the attention mechanism to obtain the prediction result of the reply based on the multi-modal feature representation and the reply feature representation.

4. The method according to any one of claims 1 to 3, wherein, The encoding layer comprises a group of first encoding modules and at least one group of second encoding modules; a group of the first encoding modules comprises a plurality of first encoding modules connected in sequence, and the processing result of a previous first encoding module is output to a next first encoding module, and the input of a first first encoding module comprises the question, and the output of a last first encoding module comprises a question feature representation corresponding to the question; each first encoding module comprises a first encoding submodule that performs attention mechanism processing on the input; in the at least one group of second encoding modules, one group of the second encoding modules processes one kind of modal data in the multi-modal data, and each group of the second encoding modules comprises a plurality of second encoding modules connected in sequence; in one group of the second encoding modules, the processing result of a previous second encoding module is output to a next second encoding module, the input of a first second encoding module is the corresponding one kind of modal data in the multi-modal data, and the output of a last second encoding module comprises a question-aware multi-modal feature representation. The second encoding module comprises a second encoding submodule and a bidirectional attention mechanism module, the second encoding submodule performs attention mechanism processing on the input, and the bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and the processing result of the second encoding submodule.

5. The method of claim 4, wherein, The decoding module comprises a first decoding submodule and at least one second decoding submodule connected in sequence, the first decoding submodule processes the input by using an attention mechanism, the first decoding submodule of the first decoding module obtains the reply feature representation corresponding to the known part and the to-be-predicted part in the reply, the input of the second decoding submodule comprises the output of the first decoding submodule or the last second decoding submodule and the corresponding multi-modal feature representation, the second decoding submodule is used for performing attention mechanism processing on the input, and the output of the second decoding submodule of the last decoding module comprises the prediction result of the to-be-predicted part.

6. A dialogue generation method, wherein, The method comprises the following steps: acquiring a question and multi-modal data of a target object corresponding to the question; inputting the question and the multi-modal data into a dialogue generation model to obtain a target reply to the question; the dialogue generation model is trained by using the method in any one of claims 1-5.

7. A dialogue generation method, wherein, The method comprises the following steps: acquiring a question; determining a target object corresponding to the question; acquiring multi-modal data of the target object; inputting the question and the multi-modal data into a dialogue generation model to obtain a target reply to the question; the dialogue generation model is trained by using the method in any one of claims 1-5; outputting the target reply.

8. A dialogue generation method, wherein, The method comprises the following steps: acquiring a question and multi-modal data of a target object corresponding to the question; inputting the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain a question-aware multi-modal feature representation; the dialogue generation model is trained by using the method in any one of claims 1-5; inputting a candidate reply and the question-aware multi-modal feature representation into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question.

9. The method of claim 8, wherein, The encoding layer obtains the question-aware multi-modal feature representation in the following manner: obtaining a first interaction from the question to the multi-modal data and a second interaction from the multi-modal data to the question by using a bidirectional attention mechanism; obtaining the question-aware multi-modal feature representation according to the first interaction and the second interaction.

10. The method of claim 8, wherein, The decoding layer obtains the prediction result of the reply to the question in the following manner: obtaining a reply feature representation corresponding to the reply by using an attention mechanism, and obtaining the prediction result of the reply based on the multi-modal feature representation and the reply feature representation by using the attention mechanism.

11. The method according to any one of claims 9-10, wherein, The encoding layer comprises a group of first encoding modules and at least a group of second encoding modules; The first encoding module includes a first encoding submodule, and the first encoding submodule performs attention mechanism processing on an input. The second encoding module includes a second encoding submodule and a bidirectional attention mechanism module. The second encoding submodule performs attention mechanism processing on an input, and the bidirectional attention mechanism module performs bidirectional attention mechanism processing on the question feature representation and a processing result of the second encoding submodule. The decoding module includes a first decoding submodule and at least one second decoding submodule connected in sequence. The first decoding submodule performs processing on an input using attention mechanism. The first decoding submodule of the first decoding module obtains a reply feature representation corresponding to a known part and a to-be-predicted part of the reply. An input of the second decoding submodule includes an output of the first decoding submodule or a previous second decoding submodule and the corresponding multi-modal feature representation. The second decoding submodule is configured to perform attention mechanism processing on the input. An output of the second decoding submodule of the last decoding module includes the prediction result of the to-be-predicted part. The method includes the following steps:

12. The method of claim 11, wherein, acquiring a question; 13. A dialogue generation method, wherein, determining a target object corresponding to the question; acquiring multi-modal data of the target object; inputting the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain a multi-modal feature representation perceived by the question; training the dialogue generation model using the method of any one of claims 1-5; inputting a candidate reply and the multi-modal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question; and outputting the target reply. The method includes the following steps: a first acquiring module configured to acquire a training sample; 14. An apparatus for training a dialogue generation model, wherein, the training sample includes a question, a reply to the question, and multi-modal data of a target object corresponding to a dialogue text, wherein the multi-modal data includes at least two of the following: text, picture, video, audio, and voice; a first input module configured to input the question and the multi-modal data into an encoding layer of a dialogue generation model to obtain a multi-modal feature representation perceived by the question; and a training module configured to train the dialogue generation model using the method of any one of claims 1-5. ​ The second input module is configured to input the reply to the question and the multimodal feature representation of the question perception into a decoding layer of the dialogue generation model, to obtain interaction between the reply to the question and the multimodal data and interaction between the multimodal data, and further to obtain a prediction result of the reply to the question. The training module is configured to train the dialogue generation model according to the prediction result of the reply to the question. The decoding layer includes a group of decoding modules, and the group of decoding modules includes a plurality of decoding modules connected in sequence, the output of a previous decoding module being used as the input of a next decoding module, the input of a first decoding module including the reply, and the output of a last decoding module including the prediction result corresponding to the reply.

15. A dialogue generation apparatus, wherein, The second obtaining module is configured to obtain a question and multimodal data of a target object corresponding to the question. The third obtaining module is configured to input the question and the multimodal data into a dialogue generation model to obtain a target reply to the question. The dialogue generation model is trained by using the device of claim 14. The fourth obtaining module is configured to obtain a question.

16. A dialogue generation apparatus, wherein, The first determining module is configured to determine a target object corresponding to the question. The fifth obtaining module is configured to obtain multimodal data of the target object. The sixth obtaining module is configured to input the question and the multimodal data into a dialogue generation model to obtain a target reply to the question. The dialogue generation model is trained by using the device of claim 14. The first output module is configured to output the target reply. The seventh obtaining module is configured to obtain a question and multimodal data of a target object corresponding to the question. The third input module is configured to input the question and the multimodal data into an encoding layer of a dialogue generation model to obtain a multimodal feature representation perceived by the question.

17. A dialogue generation apparatus, wherein, The dialogue generation model is trained by using the device of claim 14. The fourth input module is configured to input a candidate reply and the multimodal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question. The eighth obtaining module is configured to obtain a question. The second determining module is configured to determine a target object corresponding to the question. The ninth obtaining module is configured to obtain multimodal data of the target object.

18. A dialogue generation apparatus, wherein, The tenth obtaining module is configured to input the question and the multimodal data into an encoding layer of a dialogue generation model to obtain a multimodal feature representation perceived by the question. The dialogue generation model is trained by using the device of claim 14. The fifth input module is configured to input a candidate reply and the multimodal feature representation perceived by the question into a decoding layer of the dialogue generation model to obtain a prediction result of whether the candidate reply is a target reply to the question. The second output module is configured to output the target reply. ​ ​ ​ ​ 19. An electronic device, comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement the method of any one of claims 1-13.

20. A computer readable storage medium having stored thereon computer instructions, wherein, The computer program product is stored in a computer readable medium.

21. A computer program product comprising computer instructions, wherein, The computer program product is stored in a computer readable medium. The computer program product is stored in a computer readable medium.

Images