Dialogue processing method, device, equipment and storage medium

Abstract

The present invention provides a method, device, equipment and storage medium for processing a conversation; the method comprises: obtaining input information of a current conversation; encoding the input information, context information of a historical conversation, and relevant document information of the historical conversation to obtain context information of the current conversation; first decoding the input information of the current conversation and the context information of the current conversation to obtain intermediate decoded information of the current conversation; second decoding the intermediate decoded information and relevant document information of the current conversation to obtain reply information of the current conversation. Through the present invention, accurate and context-coherent reply information can be obtained.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to a dialogue processing method, device, equipment and storage medium. Background Art

[0002] With the development of computer technology, dialogue systems have been applied to various fields, and people's lives are closely related to dialogue systems, such as mobile phone voice assistants. Dialogue systems have brought great convenience to users.

[0003] The electronic device can have a free conversation with the user through the dialogue system, that is, provide the user with a corresponding reply based on the user's input information. For example, the electronic device can get the corresponding reply "sunny" based on the input information "What is the weather like tomorrow?"

[0004] However, the response information obtained by the existing dialogue system is inaccurate. For example, when the input information is "Do you want to go to work tomorrow?", the response information obtained is "I don't know". Summary of the invention

[0005] The embodiments of the present invention provide a dialogue processing method, apparatus, device and storage medium, which can obtain accurate and context-coherent reply information.

[0006] The technical solution of the embodiment of the present invention is achieved as follows:

[0007] An embodiment of the present invention provides a dialog processing method, including:

[0008] Get the input information of the current conversation;

[0009] Encoding the input information, the context information of the historical conversation, and the relevant document information of the historical conversation to obtain the context information of the current conversation;

[0010] Performing a first decoding on the input information of the current conversation and the context information of the current conversation to obtain intermediate decoded information of the current conversation;

[0011] The intermediate decoded information and the relevant document information of the current conversation are decoded for the second time to obtain the reply information of the current conversation.

[0012] An embodiment of the present invention provides a dialog processing device, including:

[0013] The acquisition module is used to obtain the input information of the current conversation;

[0014] An encoding module, used to encode the input information, the context information of the historical conversation, and the relevant document information of the historical conversation to obtain the context information of the current conversation;

[0015] A first decoding module, configured to perform a first decoding on the input information of the current conversation and the context information of the current conversation to obtain intermediate decoding information of the current conversation;

[0016] The second decoding module is used to perform a second decoding on the intermediate decoding information and the relevant document information of the current conversation to obtain the reply information of the current conversation.

[0017] In the above solution, the input information is a coding vector of the input content; wherein the acquisition module is further used for:

[0018] According to the input content of the current conversation, obtain the word sequence vector and position information of the current conversation;

[0019] The word sequence vector and position information of the current conversation are encoded to obtain an encoding vector of the input content.

[0020] In the above solution, the acquisition module is also used for:

[0021] Encoding the word sequence vector and position information of the current conversation through an attention mechanism to obtain surface features of the current conversation;

[0022] The surface features of the current conversation are encoded through a neural network to obtain an encoding vector of the input information of the current conversation.

[0023] In the above solution, the relevant document information is the encoding vector of the relevant document; the dialogue processing device also includes:

[0024] A retrieval module is used to search a document knowledge base according to historical conversations to obtain relevant documents corresponding to the historical conversations; obtain a word sequence vector and position information of the relevant document according to the content of the relevant document; encode the word sequence vector and position information of the relevant document to obtain an encoding vector of the relevant document.

[0025] In the above solution, the encoding module is also used for:

[0026] Encoding the input information to obtain surface features of the input information;

[0027] Obtaining a first fusion feature according to the surface feature of the input information and the relevant document information of the historical conversation, wherein the first fusion feature includes a relationship between the input information and the relevant document information of the historical conversation;

[0028] The context information of the current conversation is obtained according to the first fusion feature and the context information of the historical conversation.

[0029] In the above solution, the encoding module is also used for:

[0030] Get relevant document information of historical conversations;

[0031] The surface features of the input information and the relevant document information of the historical conversation are fused according to the attention mechanism to obtain a first fused feature.

[0032] In the above solution, the encoding module is also used for:

[0033] Get context information of historical conversations;

[0034] The first fusion feature and the context information of the historical conversation are fused according to the attention mechanism to obtain the context information of the current conversation.

[0035] In the above solution, the first decoding module is further used to:

[0036] Performing a first decoding on the input information of the current conversation and the context information of the current conversation, and sequentially generating first words corresponding to the input information of the current conversation;

[0037] According to the order of generating the first vocabulary, the first vocabulary is composed into the intermediate decoding information of the current dialogue.

[0038] In the above solution, the first decoding module is further used to:

[0039] encoding the generated second vocabulary through an attention mechanism to obtain encoding features of the second vocabulary, where the second vocabulary is a vocabulary related to the input information of the current conversation;

[0040] fusing the encoding features of the second vocabulary and the context information of the current conversation through an attention mechanism to obtain second fused information;

[0041] fusing the second fused information and the input information of the current conversation through the attention mechanism to obtain third fused information;

[0042] The third fusion information is decoded by a neural network to generate a next first vocabulary corresponding to the input information of the current conversation.

[0043] In the above solution, the second decoding module is further used to:

[0044] Performing a second decoding on the intermediate decoded information and the document information related to the current conversation, and sequentially generating a third vocabulary corresponding to the input information of the current conversation;

[0045] According to the order in which the third words are generated, the third words are combined into reply information of the current conversation.

[0046] In the above solution, the second decoding module is further used to:

[0047] encoding the generated fourth vocabulary through an attention mechanism to obtain encoding features of the fourth vocabulary, where the fourth vocabulary is a vocabulary related to the input information of the current conversation;

[0048] fusing the encoding features of the fourth vocabulary and the relevant document information of the current conversation through an attention mechanism to obtain fourth fused information;

[0049] fusing the fourth fused information and the intermediate decoded information through the attention mechanism to obtain fifth fused information;

[0050] The fifth fusion information is decoded by a neural network to generate a next third vocabulary corresponding to the input information of the current conversation.

[0051] An embodiment of the present invention provides a device for dialog processing, including:

[0052] A memory for storing executable instructions;

[0053] The processor is used to implement the dialogue processing method provided by the embodiment of the present invention when executing the executable instructions stored in the memory.

[0054] An embodiment of the present invention provides a storage medium storing executable instructions for causing a processor to execute the dialog processing method provided by the embodiment of the present invention.

[0055] The embodiments of the present invention have the following beneficial effects:

[0056] Integrating the context of historical conversations and related documents into the current conversation makes the generated reply information more coherent with the context and more closely connected with related documents, and the reply information is closer to the reply required by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a schematic diagram of the structure of a dialogue system provided by an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of an application scenario of the dialogue processing system 10 provided in an embodiment of the present invention;

[0059] Figure 3 is a schematic diagram of the structure of a conversation processing device 500 provided in an embodiment of the present invention;

[0060] Figure 4is a schematic diagram of the structure of a dialogue processing device 555 provided in an embodiment of the present invention;

[0061] FIG. 5A to FIG. 5B is a flow chart of a conversation processing method provided by an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of a dialogue processing system provided by an embodiment of the present invention processing a dialogue;

[0063] Fig. 7A is a schematic diagram of the structure of a self-attention encoder provided by an embodiment of the present invention;

[0064] Figure 7B is a schematic diagram of the structure of an incremental Transformer encoder provided by an embodiment of the present invention;

[0065] Figure 7C It is a schematic diagram of the structure of the Tuikao decoder provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0067] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0068] In the following description, the terms "first\second\third\fourth\fifth" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third\fourth\fifth" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0070] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0071] 1) Document: A common type of knowledge, a written description of natural language, generally including at least one sentence. For example, a movie synopsis, character introduction, etc. are all representations of a document. The written description "Today's weather is very good, suitable for going out" is also a representation of a document.

[0072] 2) Baseline models: conventional machine learning models, including linear regression models (models that predict continuous values ​​from a series of feature values) and logistic regression models (models that classify structured data or natural language).

[0073] The following is an analysis of the solutions for dialogue processing provided by related technologies.

[0074] In the related art, the Transformer model includes two parts: an encoder and a decoder. The encoder consists of at least one layer, and each layer consists of two sublayers, including a first sublayer and a second sublayer: the first sublayer is a multi-head attention layer, and the second sublayer is a feedforward neural network layer; the decoder also consists of at least one layer, and each layer consists of two sublayers, including a first sublayer and a second sublayer: the first sublayer is a multi-head attention layer, and the second sublayer is a feedforward neural network layer.

[0075] Applying the Transformer model in the dialogue system can get corresponding responses based on the dialogue. Figure 1 As shown, knowledge related to the conversation is retrieved in the retrieval system, encoded by the encoder of the Transformer model and stored in the storage structure, the conversation and knowledge are encoded according to the encoder of the Transformer model, and the encoded conversation and knowledge are decoded by the decoder of the Transformer model to obtain the corresponding reply.

[0076] Although the relevant technology can obtain corresponding replies based on the conversation, the contextual coherence and accuracy of the reply information generated during the decoding process are very low.

[0077] In order to at least solve the above-mentioned technical problems of the related art, the embodiments of the present invention provide a dialogue processing method, apparatus, device and storage medium, which can obtain accurate and contextually coherent reply information. The following describes an exemplary application of the dialogue processing device provided by the embodiment of the present invention. The dialogue processing device provided by the embodiment of the present invention can be a server, such as a server deployed in the cloud, which provides reply information corresponding to the dialogue information to the user based on the dialogue information provided by the user; it can also be various types of user terminals such as laptops, tablet computers, desktop computers, mobile devices (for example, mobile phones, personal digital assistants), such as handheld terminals, which obtain corresponding reply information based on the dialogue information input by the user on the handheld terminal, and display it on the display interface of the handheld terminal, so as to realize the automatic dialogue process between the handheld terminal and the user.

[0078] As an example, see Figure 2 , Figure 2 It is a schematic diagram of an application scenario of the dialogue processing system 10 provided in an embodiment of the present invention. The terminal 200 is connected to the server 100 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0079] The terminal 200 can be used to obtain dialogue information. For example, when the user inputs dialogue information through the input interface, after the input is completed, the terminal automatically obtains the input information of the dialogue.

[0080] In some embodiments, the terminal 200 locally executes the dialogue processing method provided by the embodiment of the present invention to obtain accurate reply information based on the input information of the current dialogue, historical dialogue information and related documents. For example, an application (Application, APP) is installed on the terminal 200, such as a voice assistant APP, and the user inputs voice or text, that is, inputs information, in the voice assistant APP. The terminal 200 obtains the context information of the current input information based on the input information, the context information of the historical dialogues and the related documents, and obtains accurate reply information based on the current input information and the context information of the current input information, and displays the reply information on the display interface 210 of the terminal 200.

[0081] Terminal 200 can also send the user's input information on terminal 100 to server 100 through network 300, and call the dialogue processing function provided by server 100. Server 100 obtains reply information through the dialogue processing method provided by an embodiment of the present invention. For example, a voice assistant APP is installed on terminal 200, and the user inputs voice or text, that is, input information, in the voice assistant APP. The terminal sends the input information to server 100 through network 300. Server 100 obtains the context information of the current input information based on the input information, the context information of historical conversations, and related documents, and obtains accurate reply information based on the current input information and the context information of the current input information, and returns the reply information to the voice assistant APP, and displays the reply information on the display interface of terminal 200.

[0082] The structure of the dialogue processing device provided in the embodiment of the present invention is further described. The dialogue processing device may be various terminals, such as a mobile phone, a computer, etc., or may be a Figure 2 A server 100 is shown.

[0083] See also Figure 3 , Figure 3 is a structural diagram of a dialog processing device 500 provided in an embodiment of the present invention, Figure 3 The dialog processing device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the dialog processing device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 3 Various buses are labeled as bus system 540 .

[0084] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0085] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0086] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present invention is intended to include any suitable type of memory. The memory 550 optionally includes one or more storage devices that are physically far away from the processor 510.

[0087] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0088] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0089] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0090] a display module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0091] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0092] In some embodiments, the dialogue processing device provided by the embodiment of the present invention can be implemented in a combination of software and hardware. As an example, the device provided by the embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the dialogue processing method provided by the embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field-Programmable Gate Array) or other electronic components.

[0093] In other embodiments, the dialogue processing device provided by the embodiments of the present invention may be implemented in software. Figure 3 A dialogue processing device 555 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, and includes a series of modules, including an acquisition module 5551, an encoding module 5552, a first decoding module 5553, and a second decoding module 5554; wherein the acquisition module 5551, the encoding module 5552, the first decoding module 5553, and the second decoding module 5554 are used to implement the dialogue processing method provided in an embodiment of the present invention.

[0094] In conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present invention, the dialogue processing method provided in the embodiment of the present invention is explained. According to the above, it can be understood that the dialogue processing method provided in the embodiment of the present invention can be used by various types of dialogue processing devices, such as smart terminals and servers.

[0095] See also Figure 4 and Figure 5A , Figure 4 is a structural diagram of a dialogue processing device 555 provided in an embodiment of the present invention, showing a dialogue processing flow. Figure 5A is a flow chart of a method for processing a conversation provided by an embodiment of the present invention, which will be combined with Figure 4 right Figure 5A The steps shown are explained.

[0096] In step 101, input information of the current conversation is obtained.

[0097] In some embodiments, the user inputs information on the input interface of the terminal, that is, generates input information of the current conversation and saves it in the memory, and the terminal can directly obtain the input information of the current conversation.

[0098] As an example, see Figure 4 , the input information of the current dialogue is obtained through the acquisition module 5551 in the dialogue processing device 555.

[0099] In step 102, the input information, the context information of the historical conversation, and the relevant document information of the historical conversation are encoded to obtain the context information of the current conversation.

[0100] In some embodiments, after obtaining the input information of the current conversation, the word vector of the input information is obtained by word vectorization of the input information, and the word vector of the input information is encoded to obtain the surface features of the input information, which include the relationship between the words in the input information. According to the time of the current conversation, the historical conversation is obtained (the time of the historical conversation is earlier than the time of the current conversation), and the document knowledge base is searched according to the content of the historical conversation to obtain the relevant document information corresponding to the historical conversation. For example, the historical conversation is "XX role in a certain movie is okay", and the relevant document information obtained by searching the document knowledge base can be "XX role in the movie plays the role of leader, teacher and protector of the safe zone. She fights against the powerful invasion of Japan with her weak body". The user can set the document information in the document knowledge base in the local terminal in advance. The document knowledge base can also be a third-party document knowledge base, that is, the terminal is connected to the third-party application, and the relevant document information corresponding to the historical conversation is obtained through the interface of the third-party application.

[0101] According to the surface features of the input information and the relevant document information of the historical conversation, a first fusion feature can be obtained, wherein the first fusion feature includes the relationship between the input information and the relevant document information of the historical conversation. According to the first fusion feature and the context information of the historical conversation, the context information of the current conversation is obtained. Accurate context information is obtained by fusing the input information, the context information of the historical conversation, and the relevant document information of the historical conversation. After the context information of the current conversation is obtained, the context information of the current conversation is saved in a memory in the conversation processing device, and the context information of the current conversation will be used as the context information of the historical conversation in the next conversation.

[0102] After obtaining the input information of the current conversation, the historical conversation is obtained, and the document knowledge base is searched according to the content of the historical conversation to obtain the relevant document information corresponding to the historical conversation. The surface features of the input information and the relevant document information of the historical conversation are fused according to the attention mechanism to obtain the first fused feature. The attention mechanism refers to associating the surface features of the input information and each word vector in the relevant document information of the historical conversation. Each word vector is related to other word vectors to avoid losing input information or document information during the encoding process.

[0103] After the input information of the current conversation is obtained, the context information of the historical conversation can be obtained in the memory of the conversation processing device, and the first fusion feature and the context information of the historical conversation are fused through the attention mechanism to obtain the context information of the current conversation. The attention mechanism refers to associating each word vector in the first fusion feature and the context information of the historical conversation, and each word vector is related to other word vectors to avoid losing input information, document information or context information during the encoding process.

[0104] As an example, see Figure 4 , the input information, the context information of the historical conversation, and the related document information of the historical conversation are encoded by the incremental Transformer encoder in the encoding module 5552 in the dialogue processing device 555 to obtain the context information of the current conversation, wherein the encoder may include multiple cascaded encoding layers, and the input information, the context information of the historical conversation, and the related document information of the historical conversation are encoded in sequence through the multiple cascaded encoding layers, that is, the encoding result of one encoding layer will be input into the next encoding layer to continue encoding until all encoding layers are traversed and the context information of the current conversation is output.

[0105] Of course, the context information of the current conversation obtained is not limited to the incremental Transformer encoder, but can also be replaced by other encoders.

[0106] In step 103, the input information of the current dialogue and the context information of the current dialogue are decoded for the first time to obtain the intermediate decoded information of the current dialogue.

[0107] In some embodiments, the input information of the current conversation and the context information of the current conversation are first decoded, and first words corresponding to the input information of the current conversation can be sequentially generated. According to the order of generating the first words, the first words are combined into the intermediate decoded information of the current conversation. The intermediate decoded information is not the final reply information, but the intermediate information between the final reply information, and the intermediate information focuses on the connection with the context of the current conversation, that is, the first decoding process focuses on the coherence between the reply information and the context.

[0108] The first vocabulary generated through the first decoding process includes: encoding the generated second vocabulary through an attention mechanism to obtain the encoding features of the second vocabulary, the second vocabulary is a vocabulary related to the input information of the current dialogue, the attention mechanism refers to associating each word vector in the generated second vocabulary to avoid losing the generated second vocabulary information during the encoding process; fusing the encoding features of the second vocabulary and the context information of the current dialogue through the attention mechanism to obtain second fused information, the attention mechanism refers to associating each word vector in the second vocabulary and the context information of the current dialogue, each word vector is related to other word vectors, and the loss of the second vocabulary information or context information during the encoding process is avoided; fusing the second fused information and the input information of the current dialogue through the attention mechanism to obtain third fused information; decoding the third fused information through a neural network to generate the next first vocabulary corresponding to the input information of the current dialogue, the neural network can be a feedforward neural network, the feedforward neural network is a simplest neural network, each neuron is arranged in layers, each neuron is only connected to the neuron of the previous layer, receives the output of the previous layer, and outputs it to the next layer, and there is no feedback between the layers.

[0109] As an example, see Figure 4 The first decoding process is performed on the input information of the current dialogue and the context information of the current dialogue through the deliberation decoder in the first decoding module 5553 in the dialogue processing device 555. Through this decoding process, the intermediate decoded information obtained has good consistency with the context of the current dialogue.

[0110] In step 104, the intermediate decoded information and the relevant document information of the current conversation are decoded for the second time to obtain the reply information of the current conversation.

[0111] In some embodiments, the intermediate decoded information and the relevant document information of the current conversation are decoded for the second time, and the third vocabulary corresponding to the input information of the current conversation is sequentially generated; according to the order of generating the third vocabulary, the third vocabulary is used to form the reply information of the current conversation. By integrating the relevant document information of the current conversation into the intermediate decoded information, the association between the reply information and the document information is increased, and the obtained reply information is more accurate and close to the needs of the user. The first decoding process focuses on the coherence between the reply information and the context, and the second decoding process focuses on the connection between the reply information and the document information, that is, the accuracy of the document application.

[0112] The reply information generated through the second decoding process includes: encoding the generated fourth vocabulary through the attention mechanism to obtain the encoding features of the fourth vocabulary, wherein the fourth vocabulary is a vocabulary related to the input information of the current conversation, and the attention mechanism refers to associating each word vector in the generated fourth vocabulary to avoid losing part of the information in the fourth vocabulary during the encoding process; fusing the encoding features of the fourth vocabulary and the relevant document information of the current conversation through the attention mechanism to obtain fourth fusion information, wherein the attention mechanism refers to associating each word vector in the fourth vocabulary and the relevant document information of the current conversation, and each word vector is related to other word vectors to avoid losing the fourth vocabulary and the relevant document information during the encoding process; The fourth fusion information and the intermediate decoded information are fused through the attention mechanism to obtain fifth fusion information, wherein the attention mechanism refers to associating the fourth fusion information and each word vector in the intermediate decoded information, and each word vector is related to other word vectors, so as to avoid losing part of the fourth fusion information and the intermediate decoded information during the encoding process; the fifth fusion information is decoded through a neural network to generate a third vocabulary corresponding to the input information of the current dialogue, and the neural network can be a feedforward neural network, which is the simplest neural network, and each neuron is arranged in layers, each neuron is only connected to the neuron of the previous layer, receives the output of the previous layer, and outputs it to the next layer, and there is no feedback between the layers.

[0113] As an example, see Figure 4 The intermediate decoded information and the relevant document information of the current conversation are decoded for the second time through the second decoding module 5554 in the dialogue processing device 555. Through this decoding process, the reply information obtained focuses on the connection with the relevant document information, thereby improving the accuracy of the reply information.

[0114] In some embodiments, input information of the current conversation and related documents of the historical conversation are encoded so that accurate context information can be subsequently obtained based on the encoded information of the input information and related documents.

[0115] In some other embodiments, the input information of the current conversation is an encoding vector of the input content. Figure 5B ,based on Figure 5A , Figure 5BThe present invention is a flowchart of a method for processing a conversation provided by an embodiment of the present invention. The method obtains the input information of the current conversation, including: obtaining a word sequence vector and position information of the current conversation according to the input content of the current conversation; encoding the word sequence vector and position information of the current conversation to obtain an encoding vector of the input content. The position information of each word is generated by the order of words in the input content, and the position information is used to represent the position relationship of the word in the input content.

[0116] The word sequence vector and position information of the current conversation are encoded to obtain the encoding vector of the input information, including: encoding the word sequence vector and position information of the current conversation through an attention mechanism to obtain the surface feature of the current conversation, the surface feature is the hidden state of the input information output through the attention mechanism, the attention mechanism refers to associating each word vector in the input information to avoid losing the input information during the encoding process; encoding the surface feature of the current conversation through a neural network to obtain the encoding vector of the input information of the current conversation.

[0117] As an example, see Figure 4 , the input information is encoded by the Transformer encoder in the acquisition module 5551 in the dialogue processing device 555 to obtain the encoding vector of the input information of the current dialogue, wherein the Transformer encoder may include multiple cascaded encoding layers, and the input information is encoded in sequence through the multiple cascaded encoding layers, that is, the encoding result of one encoding layer will be input to the next encoding layer to continue encoding, until all encoding layers are traversed, and the encoding vector of the input information is output, wherein, according to experience, the multiple cascaded encoding layers in the Transformer encoder are set to 3 layers, and the encoding result obtained by the 3-layer encoding layer is more accurate and the encoding efficiency is relatively high.

[0118] In other embodiments, the relevant document information is an encoding vector of the relevant document. Before encoding the input information, the context information of the historical conversation, and the relevant document information of the historical conversation to obtain the context information of the current conversation, in step 105, the document knowledge base is searched according to the historical conversation to obtain the relevant document corresponding to the historical conversation; in step 106, based on the content of the relevant document, the word sequence vector and position information of the relevant document are obtained, and the position information is used to represent the positional relationship of the word in the relevant document; in step 107, the word sequence vector and position information of the relevant document are encoded to obtain the encoding vector of the relevant document, so as to subsequently encode according to the input information, the context information of the historical conversation, and the encoding vector of the relevant document information of the historical conversation.

[0119] As an example, see Figure 4, the relevant documents are encoded by the Transformer encoder in the retrieval module 5555 in the dialogue processing device 555 to obtain the encoding vector of the relevant document information, wherein the Transformer encoder may include multiple cascaded encoding layers, and the input information is encoded in sequence through the multiple cascaded encoding layers, that is, the encoding result of one encoding layer will be input into the next encoding layer to continue encoding, until all the encoding layers are traversed, and the encoding vector of the input information is output, wherein, according to experience, the multiple cascaded encoding layers in the Transformer encoder are set to 3 layers, and the encoding result obtained by the 3-layer encoding layer is more accurate and the encoding efficiency is relatively high.

[0120] So far, we have combined the dialogue processing method provided in the embodiment of the present invention and its exemplary application in the dialogue processing device. Now, we will continue to explain the solution of the cooperation of various modules in the dialogue processing device 555 provided in the embodiment of the present invention to implement dialogue processing.

[0121] In the process of the dialogue processing device 555 obtaining the reply information, the acquisition module 5551 obtains the input information of the current dialogue. When the input information is the encoding vector of the input content, the acquisition module 5551 obtains the word sequence vector and position information of the current dialogue according to the input content of the current dialogue; encodes the word sequence vector and position information of the current dialogue to obtain the encoding vector of the input content.

[0122] The acquisition module 5551 can also encode the word sequence vector and position information of the current conversation through the attention mechanism to obtain the surface features of the current conversation; encode the surface features of the current conversation through the neural network to obtain the encoding vector of the input information of the current conversation.

[0123] In the process of the dialogue processing device 555 obtaining the reply information, the encoding module 5552 encodes the input information, the context information of the historical dialogue, and the relevant document information of the historical dialogue to obtain the context information of the current dialogue.

[0124] The encoding module 5552 can also encode the input information to obtain the surface features of the input information; obtain the first fusion feature based on the surface features of the input information and the relevant document information of the historical conversation, the first fusion feature includes the relationship between the input information and the relevant document information of the historical conversation; obtain the context information of the current conversation based on the first fusion feature and the context information of the historical conversation.

[0125] The encoding module 5552 can also obtain relevant document information of the historical conversation; fuse the surface features of the input information and the relevant document information of the historical conversation according to the attention mechanism to obtain a first fused feature.

[0126] The encoding module 5552 can also obtain the context information of the historical conversation; the first fusion feature and the context information of the historical conversation are fused according to the attention mechanism to obtain the context information of the current conversation.

[0127] In the process of the dialogue processing device 555 obtaining the reply information, the first decoding module 5553 performs a first decoding on the input information of the current dialogue and the context information of the current dialogue to obtain the intermediate decoding information of the current dialogue.

[0128] The first decoding module 5553 can also perform a first decoding on the input information of the current conversation and the context information of the current conversation, and sequentially generate first words corresponding to the input information of the current conversation; and according to the order in which the first words are generated, the first words are composed of the intermediate decoding information of the current conversation.

[0129] The first decoding module 5553 can also encode the generated second vocabulary through the attention mechanism to obtain the encoding features of the second vocabulary, where the second vocabulary is a vocabulary related to the input information of the current conversation; fuse the encoding features of the second vocabulary and the context information of the current conversation through the attention mechanism to obtain second fusion information; fuse the second fusion information and the input information of the current conversation through the attention mechanism to obtain third fusion information; decode the third fusion information through a neural network to generate the next first vocabulary corresponding to the input information of the current conversation.

[0130] In the process of the dialogue processing device 555 obtaining the reply information, the second decoding module 5554 performs a second decoding on the intermediate decoded information and the relevant document information of the current dialogue to obtain the reply information of the current dialogue.

[0131] The second decoding module 5554 can also perform a second decoding on the intermediate decoding information and the relevant document information of the current conversation, and sequentially generate third words corresponding to the input information of the current conversation; and according to the order in which the third words are generated, the third words are combined into reply information of the current conversation.

[0132] The second decoding module 5554 can also encode the generated fourth vocabulary through the attention mechanism to obtain the encoding features of the fourth vocabulary, where the fourth vocabulary is a vocabulary related to the input information of the current conversation; fuse the encoding features of the fourth vocabulary and the relevant document information of the current conversation through the attention mechanism to obtain fourth fusion information; fuse the fourth fusion information and the intermediate decoding information through the attention mechanism to obtain fifth fusion information; decode the fifth fusion information through a neural network to generate the next third vocabulary corresponding to the input information of the current conversation.

[0133] In the process of the dialogue processing device 555 obtaining the reply information, the retrieval module 5555 searches the document knowledge base according to the historical dialogue to obtain the relevant documents corresponding to the historical dialogue; obtains the word sequence vector and position information of the relevant document according to the content of the relevant document; encodes the word sequence vector and position information of the relevant document to obtain the encoding vector of the relevant document. The retrieval module 5555 can output the encoding vector of the relevant document to the encoding module 5552, so that the encoding module 5552 encodes the encoding vector of the input information, the context information of the historical dialogue, and the relevant document information of the historical dialogue.

[0134] The embodiment of the present invention further provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the dialog processing method provided by the embodiment of the present invention, for example, FIG. 5A to FIG. 5B The dialogue processing method shown.

[0135] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EE PROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0136] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0137] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).

[0138] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0139] The following describes an exemplary application of an embodiment of the present invention in an actual dialogue application scenario.

[0140] The dialogue system in the embodiment of the present invention includes three modules: Figure 6 As shown:

[0141] 1) Self-attention encoder: corresponding to Figure 4 The Transformer encoder in , through which the related documents and dialogue sentences (input information of the current dialogue) can be encoded.

[0142] 2) Incremental Transformer encoder: corresponding to Figure 6 In the upper middle part, the incremental Transformer encoder is an improved Transformer encoder that can incorporate related documents in the process of encoding dialogues in an incremental encoding manner. The incremental Transformer encoder takes the representation of dialogues and related documents encoded by the self-attention encoder (the encoding vector of the related documents) as input, and constructs a joint representation of context and document knowledge through the attention mechanism, that is, the connection between the context of the dialogue and the related documents.

[0143] 3) Consider the decoder: corresponding Figure 6In the middle and lower part, the Tuikao decoder is an improved Transformer decoder with two decodings, which can better generate reply information based on the conversation. The first decoding process takes the representation (encoded vector) of the current sentence (the input information of the current conversation) after passing through the self-attention encoder and the representation (the context information of the current conversation) of the input information of the current conversation and the related documents after passing through the incremental Transformer encoder as input. The second decoding process takes the representation (encoded vector) of the sentence after the first decoding (intermediate decoding information) and the related documents after passing through the self-attention encoder as input. The first decoding process focuses more on the context of the conversation, while the second decoding process focuses more on the related documents to improve the reply effect and make the reply information more accurate.

[0144] The following is a description of the solution for implementing dialogue processing by cooperating among the self-attention encoder, incremental Transformer encoder, and deliberation decoder provided in an embodiment of the present invention.

[0145] 1) Self-Attention Encoder

[0146] like Fig. 7A As shown, Fig. 7A : is a schematic diagram of the structure of the self-attention encoder provided by an embodiment of the present invention. As an exemplary application, the self-attention encoder is used to encode the relevant documents to obtain the encoding vector of the relevant documents. The relevant documents include at least one sentence. The relevant documents are calculated by the attention mechanism, and the longest path of the long-distance dependency of the relevant documents can be set to ο(1), so that the words between the sentences are connected with each other word, thereby further determining the importance of each word.

[0147] like Fig. 7A As shown, the relevant document S is calculated through a self-attention encoder (k) , the input of the self-attention encoder is the word sequence vector and position information of the relevant document, as shown in the following formulas (1) and (2):

[0148]

[0149]

[0150] Where K is a natural number, such as Figure 6 As shown in , represents the Kth related document, j is a natural number, representing the number of words in the related document, Representing words The vectorized representation of (word sequence vector), PE(·) represents the encoding function of position information.

[0151] The self-attention encoder consists of N SThe same encoding layer. Each encoding layer consists of two sublayers. The first sublayer is a multi-head self-attention layer (MultiHead), and the second sublayer is a simple input-output fully connected feedforward neural network layer, as shown in the following formulas (3), (4), and (5):

[0152]

[0153] D (1) =FFN(A (1) ) (4)

[0154] FFN(x)=max(0,xW1+b1)W2+b2 (5)

[0155] Among them, A (1) represents the hidden state calculated by the multi-head attention layer in the first sub-layer, that is, a feature representation (surface feature) of the relevant document after the self-attention mechanism calculates it, D (1) Related Documents (k) The output after the first layer of the self-attention encoder, the FFN function includes two linear transformations and a ReLU activation function, where W1 and W2 represent parameter matrices, and b1 and b2 represent bias vectors.

[0156] For each encoding layer in the self-attention encoder, it must go through the calculation of the multi-head self-attention layer and the feedforward neural network layer, that is, repeating formulas (6) and (7):

[0157] A (n) =MultiHead(D (n-1) ,D (n-1) ,D (n-1) ) (6)

[0158] D (n) =FFN(A (n) ) (7)

[0159] Where n = 1, ..., N s , N S represents the number of encoding layers of the self-attention encoder, That is, the input of the first encoding layer is the word sequence vector and position information of the relevant document.

[0160] Related documents N through the self-attention encoder S After encoding layers, the final encoding vector d of the relevant document is obtained (k) , S.A. s (·) represents the entire calculation process of the self-attention encoder for related documents, as shown in the following formula (8):

[0161]

[0162] As an exemplary application, the input information of the current conversation is encoded by the self-attention encoder to obtain the encoding vector of the input content. (k) , the input of the self-attention encoder is the word sequence vector and position information of the input information, as shown in the following formula (9):

[0163]

[0164]

[0165] Where I is a natural number, indicating the number of words in the input information. Representing words The vectorized representation of (word sequence vector), PE(·) represents the encoding function of position information.

[0166] The self-attention encoder consists of N S The first sublayer is a multi-head self-attention layer (MultiHead), and the second sublayer is a simple input-output fully connected feedforward neural network layer.

[0167] The input information passes through the self-attention encoder N S After the encoding layers, the final encoding vector SA of the input information is obtained u (u (k) ), SA u (·) represents the entire calculation process of the self-attention encoder on the input information.

[0168] SA s (·) represents the entire calculation process of the self-attention encoder for related documents, SA u (·) represents the entire calculation process of the self-attention encoder on the input information, where SA s (·) and SA u (·) Have the same structure, but the parameters can be different.

[0169] 2) Incremental Transformer Encoder

[0170] The incremental Transformer fuses the context information of related documents and historical conversations when encoding the input information of the current conversation based on the attention mechanism to obtain the context information of the current conversation, as shown in the following formula (11):

[0171]

[0172] Where ITE(·) represents the encoding function of the incremental Transformer encoder, c (k) Indicates the current conversation u (k) The context information, c (k-1) Indicates historical conversation u (k-1) Context information, that is, the context information of historical conversations, d (k) Indicates related documents (k) The encoding vector of Indicates the current conversation u (k) The word vector of .

[0173] like Figure 7B As shown, the incremental Transformer encoder includes N u The same encoding layer. Each encoding layer contains four sub-layers, the first sub-layer is a multi-head self-attention layer (MultiHead), as shown in the following formula (12):

[0174] B (n) =MultiHead(C (n-1) ,C (n-1) ,C (n-1) ) (12)

[0175] Among them, C (n-1) is the output of the fourth sublayer of the previous coding layer, n = 1, ..., N u , N u represents the number of encoding layers of the incremental Transformer encoder, That is, the input of the first encoding layer is the current dialogue u (k) The word vector of B (n) Represents the output of the first sub-layer, that is, the surface features of the input information.

[0176] The second sublayer of each encoding layer is a knowledge attention layer, as shown in the following formula (13):

[0177] E (n) =MultiHead(B (n) ,d (k) ,d (k) ) (13)

[0178] Among them, B (n) represents the output of the first sub-layer, d (k) Represents the encoding vector of the relevant document k, such as Figure 6 As shown, the relevant document k is retrieved from the document knowledge base to obtain the relevant documents corresponding to the historical dialogue, E (n) Represents the output of the second sub-layer, that is, the first fusion feature.

[0179] The third sublayer of each encoding layer is a contextual attention layer, as shown in formula (14):

[0180] F (n) =MultiHead(E (n) ,c (k-1) ,c (k-1) ) (14)

[0181] Among them, E (n) represents the output of the second sublayer, c (k-1) Indicates historical conversation u (k-1) The context information, E (n) F (n) Represents the output of the third sub-layer.

[0182] The fourth sublayer of each encoding layer is a simple input-output fully connected feedforward neural network layer, as shown in the following formula (15):

[0183] C (n) =FFN(F (n) ) (15)

[0184] Among them, F (n) Represents the output of the third sub-layer. The FFN function includes two linear transformations and a ReLU activation function. (n) Represents the output of the fourth sub-layer.

[0185] For each encoding layer in the incremental Transformer encoder, four sub-layer calculations are required, that is, formulas (12)-(15) are repeated to obtain u The context information of the current conversation of the encoding layer is shown in the following formula (16):

[0186]

[0187] in, Indicates that after N u The current session of the encoding layer, c (k) Represents the context information of the current conversation.

[0188] 3) Consider the decoder

[0189] The decoder consists of two decoding processes. The first decoding process uses the current dialogue SA u (u (k) ) and the context of the current conversation c (k) The first decoding process focuses on the consistency between the reply information and the context, and the second decoding process uses the result of the first decoding and the relevant documents s (k+1) For input, related documents (k+1)Indicates the relevant documents of the current conversation. The second decoding process focuses on the association between the reply information and the document.

[0190] It should be noted that the process of generating a reply message through the decoder is based on the current conversation SA u (u (k) ), the context of the current conversation c (k) And related documents (k+1) The words in the reply message are generated sequentially, and according to the order in which the words are generated, the words are combined into the reply message of the current conversation.

[0191] When you need to generate the i-th word in the reply message When the generated word As the input of the deliberation encoder, Indicates the words that have been generated The matrix is ​​shown in the following formula (17):

[0192]

[0193] in, is the representation vector of the start identifier, indicating the start of generating the word in the reply message.

[0194] like Figure 7C As shown, the decoder includes a first decoder and a second decoder, the first decoder and the second decoder have the same structure but different parameters, and the primary decoder and the secondary decoder each include N y There are four identical decoding layers, each with four sublayers.

[0195] For the first decoder, the first sublayer in each decoding layer is a multi-head self-attention layer, as shown in the following formula (18):

[0196]

[0197] in, is the output of the previous decoding layer, n=1,...,N y , N y represents the number of decoding layers of the first decoder, The words that have been generated The matrix of Represents the output of the first sub-layer, that is, the encoded features of the generated words.

[0198] The second sublayer in each decoding layer of the first decoder is a multi-head contextual attention layer, as shown in the following formula (19):

[0199]

[0200] in, represents the output of the first sub-layer, c (k) Represents the context of the current conversation, represents the output of the second sub-layer, i.e., the second fusion information, which includes the generated words Relationship with the context of the current conversation,This formula focuses on the relationship between the encoded reply information and the,context of the current conversation.

[0201] The third sublayer in each decoding layer of the first decoder is a multi-head previous sentence attention layer, as shown in the following formula (20):

[0202]

[0203] in, represents the output of the second sublayer, SA u (u (k) ) represents the encoding vector of the current conversation, represents the output of the third sub-layer, i.e., the third fusion information, which includes the generated words Relationship to the current conversation.

[0204] The fourth sublayer in each decoding layer of the first decoder is a simple input-output fully connected feedforward neural network layer, as shown in the following formula (21):

[0205]

[0206] in, represents the output of the third sublayer, Represents the output of the fourth sub-layer.

[0207] The generated words have passed through the first decoder N y After the encoding layer, the output of the fourth sub-layer is calculated by the normalized exponential function, i.e., the Softmax function, to obtain the probability of generating a word in the first decoding, as shown in the following formula (22):

[0208]

[0209] in, It represents the reply word obtained by one decoding of the deliberation decoder, that is, the word in the intermediate decoded information.

[0210] The words in the generated intermediate decoding information are combined into the intermediate decoding information of the current dialogue in the order of generation.

[0211] For the second decoder, the first sublayer in each decoding layer is a multi-head self-attention layer, as shown in the following formula (23):

[0212]

[0213] in, is the output of the previous decoding layer, n=1,...,N y , N y represents the number of decoding layers of the second decoder, The words that have been generated The matrix of Represents the output of the first sub-layer, that is, the encoded features of the generated words.

[0214] The second sublayer in each decoding layer of the second decoder is a multi-head knowledge attention layer, as shown in the following formula (24):

[0215]

[0216] in, represents the output of the first sub-layer, d (k+1) Represents the relevant documents of the current conversation, represents the output of the second sub-layer, i.e., the fourth fusion information, which includes the generated words The relationship with the relevant documents of the current conversation,This formula focuses on the relationship between the encoded reply information and the relevant documents of the current conversation, making the reply information more accurate and closer to the needs of users.

[0217] The third sublayer in each decoding layer of the second decoder is a multi-headed decoding attention layer, as shown in the following formula (25):

[0218]

[0219] in, represents the output of the second sublayer, The encoding vector representing the intermediate decoding information, Represents the output of the third sub-layer, that is, the fifth fusion information, which includes the relationship between the fourth fusion information and the intermediate decoding information.

[0220] The fourth sublayer in each decoding layer of the second decoder is a simple input-output fully connected feedforward neural network layer, as shown in the following formula (26):

[0221]

[0222] in, represents the output of the third sublayer, Represents the output of the fourth sub-layer.

[0223] The generated words and intermediate decoding information are passed through the second decoder N yAfter the encoding layer, the output of the fourth sub-layer is calculated by the normalized exponential function, i.e., the Softmax function, to obtain the probability of generating a word in the second decoding, as shown in the following formula (27):

[0224]

[0225] in, A word representing the reply information obtained through secondary decoding by the deliberation decoder.

[0226] The words in the generated reply message are combined into a complete reply message in the order in which they are generated.

[0227] The model in the embodiment of the present invention is compared with several baseline models:

[0228] A. Model without document knowledge:

[0229] 1) Seq2Seq: A simple encoder-decoder model with a global attention mechanism added. The context sentence is concatenated into a long sentence (with sentences separated by separators) as input.

[0230] 2) HRED: A multi-layer encoder-decoder model consisting of a word-level LSTM RNN and a sentence-level LSTM RNN.

[0231] 3) Transformer: A neural machine translation model based on a multi-head self-attention mechanism. The context sentences are concatenated into a long sentence (separated by separators) as input.

[0232] B. Models using knowledge:

[0233] 1) Seq2Seq (+ knowledge) and HRED (+ knowledge): Based on Seq2Seq and HRED respectively. Both models concatenate the document representation and the word vector generated in the previous step of decoding as the input of decoding.

[0234] 2) Wizard Transformer: A Transformer-based multi-turn open-domain dialogue model that leverages unstructured factual knowledge. This model concatenates dialogue context and unstructured factual textual knowledge (dialogue history and unstructured knowledge are separated by a delimiter) as input.

[0235] 3) ITE+CKAD: This model uses an incremental Transformer encoder (ITE) and a self-attention decoder (CKAD). This model is used to test the effectiveness of the proposed deliberation decoder.

[0236] 4) KAT: This model is a simplified version of the Incremental Transformer Encoder (ITE) that does not contain the contextual attention sublayer. Multiple sentences of the dialogue history are concatenated into one long sentence (with delimiters between sentences). The decoder of this model is a simplified contextual knowledge attention decoder (CK AD), which also does not have a contextual attention sublayer, and is used to verify the effectiveness of processing dialogue context in the model of the embodiment of the present invention.

[0237] There are two evaluation indicators in the embodiment of the present invention, as shown below:

[0238] 1) Automatic evaluation: Use perplexity (PPL) and BLEU to automatically evaluate the quality of generated responses.

[0239] Each model uses the perplexity of the standard reply on the trained model (PPL) as the evaluation criterion. The lower the perplexity, the better the model performance.

[0240] BLEU represents the degree of n-gram overlap between the generated response and a standard response. Since there is only one standard response for each response, if there may be multiple feasible responses, the BLEU score will be very low.

[0241] The model proposed in the embodiment of the present invention is automatically evaluated with other baseline system models. The automatic evaluation results are shown in Table 1:

[0242] Table 1 Automatic evaluation results

[0243]

[0244]

[0245] As shown in Table 1, in the automatic evaluation, the dialogue system in the embodiment of the present application has lower perplexity and higher BLEU score than the baseline model. Regarding the BLEU index, in the baseline system, the HRED model with document knowledge achieved the highest 0.77 BLEU score; among all models, the ITE+DD model achieved the highest 0.95 BLEU score. In terms of perplexity index, Wizard Transformer achieved the lowest perplexity (70.30) of the baseline model; the ITE+DD model achieved a fairly low perplexity of 15.11.

[0246] 2) Manual evaluation: Manual evaluation is necessary for dialogue generation. 30 complete dialogues were randomly selected from the test set, containing 606 dialogue sentences. The model in the embodiment of the present invention and the baseline model (a total of 9 models) were tested and 5454 replies were obtained. Users scored the generated replies based on the dialogue history and related documents.

[0247] Define three manual evaluation indicators - fluency, knowledge relevance and context consistency. All indicator scores range from 0 / 1 / 2. The following is a detailed explanation of these manual indicators:

[0248] Fluency: whether the generated response is natural and fluent. 0 means it is not fluent or incomprehensible; 1 means it is partially fluent but still understandable; 2 means the generated response is completely fluent.

[0249] Knowledge relevance: whether the generated response uses relevant and correct knowledge. 0 means no relevant knowledge is used; 1 means relevant knowledge is included but used incorrectly; 2 means relevant knowledge is used and is correct.

[0250] Contextual consistency: whether the generated response is consistent with the context and whether it can guide the next conversation. 0 points means that the generated response is irrelevant to the context or directly leads to the end of the conversation; 1 point means that the generated response is relevant to the context but cannot guide the conversation process based on the relevant document knowledge; 2 points means that it not only conforms to the context well, but can also guide the next conversation based on the relevant document knowledge.

[0251] The model proposed in the embodiment of the present invention is manually evaluated with other baseline system models. The manual evaluation results are shown in Table 2:

[0252] Table 2. Manual evaluation results

[0253]

[0254] As shown in Table 2, in the manual evaluation, among all baseline models, the Wizard Transformer model and the Seq2Seq model without knowledge achieved the highest fluency (1.62), the Wizard Transformer model achieved the highest knowledge relevance (0.47), and the Transformer without document achieved the highest context consistency (0.67). Among all models, the ITE+CKAD model achieved the highest fluency (1.68), and the ITE+DD model proposed in the embodiment of the present invention achieved the highest knowledge relevance (0.56) and the highest context consistency (0.90).

[0255] The model proposed in the embodiment of the present invention is evaluated for knowledge relevance and context consistency with other baseline system models. The evaluation results are shown in Table 3:

[0256] Table 3 Ratio of the scores (0 / 1 / 2) of knowledge relevance and contextual consistency for Wizard Transformer model, ITE+CKAD, and ITE+DD

[0257]

[0258] The incremental Transformer+elaboration decoder ITE+DD proposed in the embodiment of the present invention achieves a very low perplexity, indicating that the secondary decoder makes good use of the document knowledge and significantly reduces the perplexity.

[0259] As shown in Table 3, the incremental Transformer+decoder ITE+DD proposed in the embodiment of the present invention has a higher proportion of 2 points in the manual evaluation of knowledge relevance and context consistency compared with other models. This result shows that the decoder can improve the correctness of knowledge and guide the next dialogue in the document knowledge domain.

[0260] Although the perplexity (PPL) of the ITE+CKAD model is only slightly better than that of KAT, its BLEU score, fluency, knowledge relevance, and context consistency all significantly exceed those of the KAT model. This result shows that the incremental Transformer can better handle multi-round dialogues based on document knowledge.

[0261] Wizard Transformer performs well in terms of knowledge relevance. However, its context consistency is worse than other baseline models.

[0262] The model proposed in the embodiment of the present invention is evaluated for knowledge relevance and context consistency with other baseline system models. The evaluation results are shown in Table 3:

[0263] As shown in Table 3, the proportion of Wizard Transformer with a score of 1 on the knowledge relevance index is about twice that of 2 points. This result shows that this model is more inclined to generate responses that are relevant to document knowledge but incorrect. At the same time, Wizard Transformer's poor performance on the context consistency index also shows that it cannot respond well to the previous text. Therefore, simply concatenating the conversation context and document knowledge as model input has limitations.

[0264] Table 4 lists some responses generated by the incremental Transformer encoder + deliberation decoder (ITE+DD) proposed in some embodiments of the present invention and the best performing baseline model Wizard Transformer. Compared with the baseline model Wizard Transformer, the model proposed in some embodiments of the present invention can generate responses that are more relevant to knowledge and more consistent with the context.

[0265] In order to prove the effectiveness of the two decodings, the results of the first decoding and the second decoding are compared (Table 5). In Example 1, the result of the second decoding corrects the knowledge errors in the first decoding result; in Example 2, the second decoding result contains more detailed knowledge than the first decoding result; in Example 3, the result of the second decoding can not only respond to the previous sentence well, but also ask relevant questions based on the given document knowledge to guide the next conversation.

[0266] Table 4 Comparison of response examples generated by the incremental Transformer encoder + deliberation decoder (ITE+DD) and the Wizard Transformer model (Wizard). These conversations are a continuous conversation process, and Document is the document knowledge related to this continuous conversation. Utterance i+1 is the standard response of Utterance i.

[0267]

[0268]

[0269] Table 5. Examples of replies generated by the two decoding processes. The bolded parts are different between the two results. For each example, the top reply is generated by the first decoder and the bottom one is generated by the second decoder.

[0270]

[0271]

[0272] In summary, the embodiment of the present invention provides a dialogue system that integrates the context of historical dialogues and related documents into the current dialogue, so that the reply information generated by the device is more consistent with the context and more closely connected with the related documents, so that the reply information is closer to the reply required by the user. Based on the model provided by the embodiment of the present invention, both the automatic indicators and the manual indicators on the relevant data sets far exceed the latest baseline model for comparison, and can generate replies with more consistent contexts and more correct knowledge usage.

[0273] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement and improvement made within the spirit and scope of the present invention are included in the protection scope of the present invention.

Claims

1. A method for processing a conversation, characterized in that: include: Get the input information of the current conversation; Encoding the input information to obtain surface features of the input information; Obtaining a first fusion feature according to the surface feature of the input information and the relevant document information of the historical conversation, wherein the first fusion feature includes the relationship between the input information and the relevant document information of the historical conversation; obtaining context information of the current conversation according to the first fusion feature and the context information of the historical conversation; The relevant documents of the historical conversation are retrieved from the document knowledge base according to the content of the historical conversation; Performing a first decoding on the input information of the current conversation and the context information of the current conversation to obtain intermediate decoded information of the current conversation; The intermediate decoded information and the relevant document information of the current conversation are decoded for the second time to obtain the reply information of the current conversation.

2. The method according to claim 1, characterized in that: The input information is the encoding vector of the input content; wherein, The step of obtaining the input information of the current conversation includes: According to the input content of the current conversation, obtain the word sequence vector and position information of the current conversation; The word sequence vector and position information of the current conversation are encoded to obtain an encoding vector of the input content.

3. The method according to claim 2, characterized in that The encoding of the word sequence vector and position information of the current conversation to obtain the encoding vector of the input information includes: Encoding the word sequence vector and position information of the current conversation through an attention mechanism to obtain surface features of the current conversation; The surface features of the current conversation are encoded through a neural network to obtain an encoding vector of the input information of the current conversation.

4. The method according to claim 1, characterized in that: The relevant document information is the encoding vector of the relevant document; wherein, Before encoding the input information, the context information of the historical conversation, and the relevant document information of the historical conversation to obtain the context information of the current conversation, the process includes: Searching a document knowledge base according to historical conversations to obtain relevant documents corresponding to the historical conversations; According to the content of the relevant document, obtain the word sequence vector and position information of the relevant document; The word sequence vector and position information of the relevant document are encoded to obtain the encoding vector of the relevant document.

5. The method according to claim 1, characterized in that The obtaining of the first fusion feature according to the surface feature of the input information and the relevant document information of the historical conversation includes: Get relevant document information of historical conversations; The surface features of the input information and the relevant document information of the historical conversation are fused according to the attention mechanism to obtain a first fused feature.

6. The method according to claim 1, characterized in that The obtaining the context information of the current conversation according to the first fusion feature and the context information of the historical conversation includes: Get context information of historical conversations; The first fusion feature and the context information of the historical conversation are fused according to the attention mechanism to obtain the context information of the current conversation.

7. The method according to claim 1, characterized in that The first decoding of the input information of the current conversation and the context information of the current conversation to obtain the intermediate decoded information of the current conversation includes: Performing a first decoding on the input information of the current conversation and the context information of the current conversation, and sequentially generating first words corresponding to the input information of the current conversation; According to the order of generating the first vocabulary, the first vocabulary is composed into the intermediate decoding information of the current dialogue.

8. The method according to claim 7, characterized in that The first decoding of the input information of the current conversation and the context information of the current conversation is performed to sequentially generate first words corresponding to the input information of the current conversation, including: encoding the generated second vocabulary through an attention mechanism to obtain encoding features of the second vocabulary, where the second vocabulary is a vocabulary related to the input information of the current conversation; fusing the encoding features of the second vocabulary and the context information of the current conversation through an attention mechanism to obtain second fused information; fusing the second fused information and the input information of the current conversation through the attention mechanism to obtain third fused information; The third fusion information is decoded by a neural network to generate a next first vocabulary corresponding to the input information of the current conversation.

9. The method according to claim 1, characterized in that: The second decoding of the intermediate decoded information and the relevant document information of the current conversation to obtain the reply information of the current conversation includes: Performing a second decoding on the intermediate decoded information and the document information related to the current conversation, and sequentially generating a third vocabulary corresponding to the input information of the current conversation; According to the order in which the third words are generated, the third words are combined into reply information of the current conversation.

10. The method according to claim 9, characterized in that The second decoding of the intermediate decoded information and the document information related to the current conversation is performed to sequentially generate a third vocabulary corresponding to the input information of the current conversation, including: encoding the generated fourth vocabulary through an attention mechanism to obtain encoding features of the fourth vocabulary, where the fourth vocabulary is a vocabulary related to the input information of the current conversation; fusing the encoding features of the fourth vocabulary and the relevant document information of the current conversation through an attention mechanism to obtain fourth fused information; fusing the fourth fused information and the intermediate decoded information through the attention mechanism to obtain fifth fused information; The fifth fusion information is decoded by a neural network to generate a next third vocabulary corresponding to the input information of the current conversation.

11. A dialogue processing device, characterized in that: include: The acquisition module is used to obtain the input information of the current conversation; An encoding module, used for encoding the input information to obtain surface features of the input information; According to the surface features of the input information and the relevant document information of the historical conversation, a first fusion feature is obtained, wherein the first fusion feature includes the relationship between the input information and the relevant document information of the historical conversation; according to the first fusion feature and the context information of the historical conversation, the context information of the current conversation is obtained; wherein the relevant documents of the historical conversation are retrieved from a document knowledge base according to the content of the historical conversation; A first decoding module, configured to perform a first decoding on the input information of the current conversation and the context information of the current conversation to obtain intermediate decoding information of the current conversation; The second decoding module is used to perform a second decoding on the intermediate decoding information and the relevant document information of the current conversation to obtain the reply information of the current conversation.

12. A conversation processing device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the dialogue processing method according to any one of claims 1 to 10 when executing the executable instructions stored in the memory.

13. A storage medium, characterized in that: Executable instructions are stored, which are used to cause the processor to execute and implement the dialogue processing method described in any one of claims 1 to 10.

Images