Multi-task learning dialogue understanding method and model

Abstract

The invention provides a multi-task learning dialogue understanding method and model. The method comprises the following steps: retrieving a knowledge triple matched with an entity in a first dialogue statement in a domain knowledge base; explicitly inserting randomly selected k knowledge triples into the first dialogue statement in a natural language form to obtain target data; k is an integer greater than 1; according to the target data, identifying the intention of the first dialogue statement; according to the embodiment of the invention, the effect of task-based dialogue understanding is improved in an explicit knowledge injection mode, and a method for dynamically inserting knowledge is provided for dialogue understanding. The model does not need to carry out repeated coding and updating on knowledge, plug and play can also be realized for new knowledge, and the requirement of dynamic development of knowledge sources is met.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a multi-task learning dialogue understanding method and model. Background Art

[0002] The Task-Oriented Dialogue System (TOD) assists users in completing tasks such as appointments and searches through human-computer interaction. Among them, dialogue understanding aims to identify user intentions and dialogue content, and is a key module in the task-oriented dialogue system. Generally speaking, the process mainly includes two tasks: dialogue intention recognition and semantic slot filling.

[0003] In traditional methods, these two tasks are modeled and implemented separately. After modeling the dialogue sentences using the powerful sequence modeling capabilities of the Recurrent Neural Network (RNN) and its variants, these two tasks are completed separately. In addition, the semantic slot filling task often uses a combination of Conditional Random Field (CRF) and RNN to improve model performance.

[0004] The traditional method of training the two tasks separately ignores the strong correlation between intent recognition and semantic slot filling tasks, making it difficult to fully utilize all supervised information.

[0005] Some studies have proposed joint models for intent recognition and semantic slot filling, including an attention-based sequence to sequence (Seq2Seq) model for joint training, and a joint intent recognition and semantic slot filling model based on the large-scale pre-trained model BERT. At the same time, some people have proposed a semantic slot gating structure to learn the relationship between intent and semantic slot vectors, and obtain better understanding effects through joint optimization.

[0006] In addition, there are methods to improve the ability of dialogue understanding models by integrating knowledge, which can be divided into implicit knowledge integration and explicit knowledge integration.

[0007] For implicit knowledge integration, existing models obtain knowledge representation based on the TransE algorithm, calculate the attention of the dialogue history encoding and knowledge encoding, and implicitly integrate knowledge. In addition, there are models that encode knowledge based on BERT and integrate dialogue history representation, external knowledge representation, and sentence representation through the attention mechanism for joint training of intent recognition and semantic slot filling. On this basis, there are methods that further use the gating mechanism to filter irrelevant knowledge triplets.

[0008] For explicit knowledge incorporation, the existing knowledge-enhanced pre-trained language model K-BERT connects the knowledge triples matching the entities in the original sentence with the corresponding entities, and augments the sentence sequence into a tree structure. By setting specific position encodings for this tree structure, the input of the tree structure can be re-input into the pre-trained model as a sequence.

[0009] The problems existing in the prior art are as follows:

[0010] 1) For methods that implicitly incorporate knowledge, such methods are based on implicit encoding of knowledge. Each time the knowledge base is updated, the corresponding knowledge representation needs to be retrained, which has poor scalability and high training costs.

[0011] 2) Explicitly injecting knowledge will increase the length of the original sentence, causing changes in the semantic slot labels, confusing the inserted knowledge and the original sentence, and introducing knowledge noise. Therefore, this method cannot be directly applied to task-based dialogue understanding problems. Summary of the invention

[0012] The purpose of the embodiments of the present invention is to provide a multi-task learning dialogue understanding method and model to solve the problem that the explicit knowledge injection method in the prior art cannot be directly applied to task-based dialogue understanding.

[0013] In order to solve the above problems, an embodiment of the present invention provides a multi-task learning dialogue understanding method, including:

[0014] Retrieving knowledge triples matching the entities in the first dialogue sentence from the domain knowledge base;

[0015] Inserting k randomly selected knowledge triples into the first dialogue sentence explicitly in natural language form to obtain target data; k is an integer greater than 1;

[0016] Based on the target data, the intent of the first dialogue sentence is identified.

[0017] The k randomly selected knowledge triples are explicitly inserted into the first dialogue sentence in the form of natural language to obtain target data, including:

[0018] Connect k knowledge triples with the entities in the first dialogue sentence to form a knowledge-sentence tree;

[0019] Add a position index to each word in the knowledge-sentence tree form to construct the target data in the knowledge-sentence tree form.

[0020] The step of identifying the intention of the first dialogue sentence according to the target data includes:

[0021] Encode the target data and the conversation history respectively to obtain the feature vector corresponding to the target data and the feature vector corresponding to the conversation history;

[0022] Concatenate the feature vector corresponding to the target data and the feature vector corresponding to the conversation history to form a current conversation representation that integrates knowledge and conversation history;

[0023] Based on the current dialogue representation of the fused knowledge and dialogue history, multi-task learning is performed using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence.

[0024] The target data is encoded to obtain a feature vector corresponding to the target data, including:

[0025] Constructing a visual matrix, wherein the visual matrix is ​​used to represent the influence relationship between the associated knowledge triple and the first dialogue sentence;

[0026] Adding the visual matrix to the self-attention mechanism of the pre-trained model BERT to obtain an enhanced BERT;

[0027] The target data in the form of a knowledge-sentence tree, each word in the knowledge-sentence tree form is added with a position index, and the visual matrix is ​​input into the enhanced BERT, and the enhanced BERT outputs a feature vector corresponding to the target data.

[0028] The dialogue history is encoded to obtain the feature vector corresponding to the dialogue history, including:

[0029] Concatenate each sentence in the conversation history and insert a special token for classification before the first word to get the updated conversation history text;

[0030] The updated conversation history text is input into the pre-trained model BERT, and BERT outputs the feature vector corresponding to the conversation history.

[0031] Among them, based on the current dialogue representation of the fusion knowledge and dialogue history, multi-task learning is performed using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence, including:

[0032] Based on the current dialogue representation of fused knowledge and dialogue history, the intention of the first dialogue sentence is identified through joint training of dialogue intent recognition and semantic slot filling tasks and auxiliary learning of associated knowledge recognition tasks.

[0033] The embodiment of the present invention also provides a multi-task learning dialogue understanding model, including:

[0034] The knowledge injection module is used to retrieve knowledge triples matching the entities in the first dialogue sentence from the domain knowledge base; and explicitly insert randomly selected k knowledge triples into the first dialogue sentence in the form of natural language to obtain target data; k is an integer greater than 1;

[0035] An understanding module is used to identify the intention of the first dialogue sentence based on the target data.

[0036] Wherein, the knowledge injection module is further used for:

[0037] Connect k knowledge triples with the entities in the first dialogue sentence to form a knowledge-sentence tree;

[0038] Add a position index to each word in the knowledge-sentence tree form to construct the target data in the knowledge-sentence tree form.

[0039] Wherein, the understanding module includes:

[0040] The dialogue encoding module is used to encode the target data and the dialogue history respectively to obtain the feature vector corresponding to the target data and the feature vector corresponding to the dialogue history; and concatenate the feature vector corresponding to the target data and the feature vector corresponding to the dialogue history to form a current dialogue representation that integrates knowledge and dialogue history;

[0041] A multi-task learning module is used to perform multi-task learning based on the current dialogue representation of the fused knowledge and dialogue history, using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence.

[0042] Wherein, the dialogue encoding module is further used for:

[0043] Constructing a visual matrix, wherein the visual matrix is ​​used to represent the influence relationship between the associated knowledge triple and the first dialogue sentence;

[0044] Adding the visual matrix to the self-attention mechanism of the pre-trained model BERT to obtain an enhanced BERT;

[0045] The target data in the form of a knowledge-sentence tree, each word in the knowledge-sentence tree form is added with a position index, and the visual matrix is ​​input into the enhanced BERT, and the enhanced BERT outputs a feature vector corresponding to the target data.

[0046] Wherein, the dialogue encoding module is further used for:

[0047] Concatenate each sentence in the conversation history and insert a special token for classification before the first word to get the updated conversation history text;

[0048] The updated conversation history text is input into the pre-trained model BERT, and BERT outputs the feature vector corresponding to the conversation history.

[0049] Wherein, the multi-task learning module is further used for:

[0050] Based on the current dialogue representation of fused knowledge and dialogue history, the intention of the first dialogue sentence is identified through joint training of dialogue intent recognition and semantic slot filling tasks and auxiliary learning of associated knowledge recognition tasks.

[0051] An embodiment of the present invention also provides a multi-task learning dialogue understanding model, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-task learning dialogue understanding method as described above is implemented.

[0052] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multi-task learning dialogue understanding method as described above.

[0053] The above technical solution of the present invention has at least the following beneficial effects:

[0054] In the multi-task learning dialogue understanding method and model of the embodiment of the present invention, the number of candidate knowledge is limited by random knowledge screening to prevent the injection of too much knowledge and the division of the original semantics, and to ensure the diversity of the injected knowledge. After that, the knowledge is inserted into the original sentence in the form of natural language to construct the target data and realize the explicit fusion of related knowledge; the effect of task-based dialogue understanding is improved by explicit knowledge injection, and a method of dynamically inserting knowledge is provided for dialogue understanding. The model does not need to repeatedly encode and update the knowledge, and can also achieve "plug and play" in the face of new knowledge, meeting the needs of dynamic development of knowledge sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart showing the steps of the multi-task learning dialogue understanding method provided by an embodiment of the present invention;

[0056] Figure 2 A schematic diagram showing the structure of a multi-task learning dialogue understanding model provided by an embodiment of the present invention;

[0057] Figure 3 A schematic diagram showing the principles of a multi-task learning module in a multi-task learning dialogue understanding model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1 As shown, an embodiment of the present invention provides a multi-task learning dialogue understanding method, including:

[0060] Step 101, searching the domain knowledge base for knowledge triples that match entities in the first dialogue sentence;

[0061] Step 102, explicitly inserting randomly selected k knowledge triples into the first dialogue sentence in natural language form to obtain target data; k is an integer greater than 1;

[0062] In this step, the number of candidate knowledge (i.e., knowledge triples) is limited through random knowledge screening to prevent the injection of too much knowledge and the division of the original semantics, and to ensure the diversity of the injected knowledge; after that, the knowledge is inserted into the original sentence in the form of natural language to construct the target data and realize the explicit fusion of related knowledge.

[0063] Step 103: Identify the intention of the first dialogue sentence according to the target data.

[0064] The embodiment of the present invention explicitly injects relevant knowledge into the current dialogue text (i.e., the first dialogue sentence) as natural language. At the same time, the embodiment of the present invention further screens the matching knowledge (i.e., randomly selects k knowledge triples) to limit the amount of knowledge and prevent noise caused by adding too much knowledge.

[0065] The embodiment of the present invention explicitly inserts knowledge into the dialogue text in the form of natural language, enhancing the model's ability to understand unfamiliar content; by explicitly injecting knowledge, the effect of task-based dialogue understanding is improved, and a method for dynamically inserting knowledge is provided for dialogue understanding. The model does not need to repeatedly encode and update knowledge, and can also achieve "plug and play" when facing new knowledge, meeting the needs of the dynamic development of knowledge sources and improving the versatility, practicality and efficiency of the model.

[0066] Most of the existing knowledge injection methods are implicit knowledge injection. In actual use, knowledge is changeable and may need to be increased or decreased at any time, and implicit injection often obtains knowledge encoding representations through algorithms or models, and integrates dialogue history representations to obtain knowledge gains. Therefore, each time the knowledge base is updated, regardless of the scale of the update, the corresponding knowledge representation needs to be retrained. The explicit knowledge injection method proposed in the embodiment of the present invention adds knowledge to the dialogue sentences in the form of natural language, realizes the plug-and-play of knowledge, and meets the needs of dynamic development of knowledge sources.

[0067] As an optional embodiment, step 102 includes:

[0068] Connect k knowledge triples with the entities in the first dialogue sentence to form a knowledge-sentence tree;

[0069] Add a position index to each word in the knowledge-sentence tree form to construct the target data in the knowledge-sentence tree form.

[0070] In at least one embodiment of the present invention, step 103 includes:

[0071] Encode the target data and the conversation history respectively to obtain the feature vector corresponding to the target data and the feature vector corresponding to the conversation history;

[0072] Concatenate the feature vector corresponding to the target data and the feature vector corresponding to the conversation history to form a current conversation representation that integrates knowledge and conversation history;

[0073] Based on the current dialogue representation of the fused knowledge and dialogue history, multi-task learning is performed using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence.

[0074] The embodiment of the present invention encodes the knowledge-sentence tree and the dialogue history respectively, and splices the vector representations of the two parts to form a current dialogue representation that integrates the knowledge and the dialogue history; further based on the integrated current dialogue representation, through the joint training of intent recognition and semantic slot filling tasks, and the auxiliary learning of the associated knowledge recognition task, it finally realizes the accurate capture of the current dialogue semantics.

[0075] The existing explicit knowledge injection method connects the matching knowledge triples with the corresponding entities, expands the sentence sequence into a tree structure, and then sets specific position codes through the tree structure to obtain a sequence that can be input into the pre-trained model; this method will cause the problem of knowledge noise. The embodiment of the present invention reduces knowledge noise, achieves more efficient knowledge injection, and obtains better model effects by designing the two basic tasks of associated knowledge recognition task and auxiliary dialogue understanding.

[0076] Optionally, based on the current dialogue representation of the fused knowledge and dialogue history, multi-task learning is performed using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence, including:

[0077] Based on the current dialogue representation of fused knowledge and dialogue history, the intention of the first dialogue sentence is identified through joint training of dialogue intent recognition and semantic slot filling tasks and auxiliary learning of associated knowledge recognition tasks.

[0078] The embodiment of the present invention adopts a multi-task learning method, which performs multi-task learning through the main task of dialogue understanding and the task of related knowledge identification. Multi-task learning fully considers the interdependence of the two basic tasks of dialogue intent identification and semantic slot filling in the dialogue understanding model, and obtains better and richer semantic representation.

[0079] The embodiment of the present invention proposes a related knowledge recognition task. This task prevents the added knowledge semantics from overwriting the original dialogue sentence semantics, and is jointly trained with the dialogue intention recognition task and the semantic slot filling task, thereby achieving a more accurate understanding of the current semantics, avoiding knowledge noise, and improving model accuracy.

[0080] As an optional embodiment, encoding the target data to obtain a feature vector corresponding to the target data includes:

[0081] Constructing a visual matrix, wherein the visual matrix is ​​used to represent the influence relationship between the associated knowledge triple and the first dialogue sentence;

[0082] Adding the visual matrix to the self-attention mechanism of the pre-trained model BERT to obtain an enhanced BERT;

[0083] The target data in the form of a knowledge-sentence tree, each word in the knowledge-sentence tree form is added with a position index, and the visual matrix is ​​input into the enhanced BERT, and the enhanced BERT outputs a feature vector corresponding to the target data.

[0084] As another optional embodiment, encoding the conversation history to obtain a feature vector corresponding to the conversation history includes:

[0085] Concatenate each sentence in the conversation history and insert a special token for classification before the first word to get the updated conversation history text;

[0086] The updated conversation history text is input into the pre-trained model BERT, and BERT outputs the feature vector corresponding to the conversation history.

[0087] In summary, the embodiment of the present invention limits the number of candidate knowledge by random knowledge screening, prevents the injection of too much knowledge and the division of the original semantics, and ensures the diversity of the injected knowledge. After that, the knowledge is inserted into the original sentence in the form of natural language to construct the target data and realize the explicit fusion of relevant knowledge; the effect of task-based dialogue understanding is improved by explicit knowledge injection, and a method for dynamic knowledge insertion is provided for dialogue understanding. The model does not need to repeatedly encode and update the knowledge, and can also achieve "plug and play" in the face of new knowledge, meeting the needs of dynamic development of knowledge sources. The embodiment of the present invention further screens the matching knowledge, effectively limits the number of knowledge, avoids the injection of too much knowledge and destroys the original semantics, and ensures the diversity of added knowledge. In addition, the auxiliary task of related knowledge identification is added for multi-task learning to avoid the injected knowledge covering the original semantics. Compared with the general model, the embodiment of the present invention requires less learning time to achieve the same training accuracy and has more efficient domain transfer capabilities. At the same time, in task-oriented dialogue understanding, the embodiment of the present invention has higher experimental accuracy and better use effect than the baseline model.

[0088] like Figure 2 As shown, the embodiment of the present invention also provides a multi-task learning dialogue understanding model, including:

[0089] The knowledge injection module is used to retrieve knowledge triples matching the entities in the first dialogue sentence from the domain knowledge base; and explicitly insert randomly selected k knowledge triples into the first dialogue sentence in the form of natural language to obtain target data; k is an integer greater than 1;

[0090] An understanding module is used to identify the intention of the first dialogue sentence based on the target data.

[0091] As an optional embodiment, the knowledge injection module is further used for:

[0092] Connect k knowledge triples with the entities in the first dialogue sentence to form a knowledge-sentence tree;

[0093] Add a position index to each word in the knowledge-sentence tree form to construct the target data in the knowledge-sentence tree form.

[0094] In at least one embodiment of the present invention, the understanding module includes:

[0095] The dialogue encoding module is used to encode the target data and the dialogue history respectively to obtain the feature vector corresponding to the target data and the feature vector corresponding to the dialogue history; and concatenate the feature vector corresponding to the target data and the feature vector corresponding to the dialogue history to form a current dialogue representation that integrates knowledge and dialogue history;

[0096] A multi-task learning module is used to perform multi-task learning based on the current dialogue representation of the fused knowledge and dialogue history, using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence.

[0097] Optionally, the dialogue encoding module is further used for:

[0098] Constructing a visual matrix, wherein the visual matrix is ​​used to represent the influence relationship between the associated knowledge triple and the first dialogue sentence;

[0099] Adding the visual matrix to the self-attention mechanism of the pre-trained model BERT to obtain an enhanced BERT;

[0100] The target data in the form of a knowledge-sentence tree, each word in the knowledge-sentence tree form is added with a position index, and the visual matrix is ​​input into the enhanced BERT, and the enhanced BERT outputs a feature vector corresponding to the target data.

[0101] Optionally, the dialogue encoding module is further used for:

[0102] Concatenate each sentence in the conversation history and insert a special token for classification before the first word to get the updated conversation history text;

[0103] The updated conversation history text is input into the pre-trained model BERT, and BERT outputs the feature vector corresponding to the conversation history.

[0104] As an optional embodiment, the multi-task learning module is further used to:

[0105] Based on the current dialogue representation of fused knowledge and dialogue history, the intention of the first dialogue sentence is identified through joint training of dialogue intent recognition and semantic slot filling tasks and auxiliary learning of associated knowledge recognition tasks.

[0106] The model proposed in the embodiment of the present invention mainly includes a knowledge injection module, a dialogue encoding module and a multi-task learning module. Among them, the knowledge injection module first limits the number of candidate knowledge through random knowledge screening to prevent the injection of too much knowledge and the division of the original semantics, and ensures the diversity of the injected knowledge. After that, the knowledge is inserted into the original sentence in the form of natural language to construct a knowledge-sentence tree to achieve explicit fusion of related knowledge; the dialogue encoding module encodes the knowledge-sentence tree and the dialogue history respectively, and splices the vector representations of the two parts to form a current dialogue representation that integrates knowledge and dialogue history; the multi-task learning module is based on the fused current dialogue representation, through the joint training of intent recognition and semantic slot filling tasks, and the auxiliary learning of related knowledge recognition tasks, and finally realizes the accurate capture of the current dialogue semantics.

[0107] In order to more clearly describe the dialogue understanding method of the multi-task learning dialogue understanding model provided by an embodiment of the present invention, an example is provided below for illustration.

[0108] The first dialogue sentence (also called the current dialogue sentence) is recorded as in Represents the current dialogue sentence u t The i-th word in l represents the dialogue sentence u t Length;

[0109] The domain knowledge base is denoted as KB = [ε1,ε2,…,ε n ], where ε i =(w i ,r ij ,w j ) represents a knowledge triple, represented by w i and w j Two entities, and the relationship between them ij constitute;

[0110] The conversation history is recorded as C = [u1,u2,…,u t-1 ], where u i is the i-th sentence in the dialogue history; y intent ∈R ni Represents the current dialogue sentence u t The corresponding intent label, ni represents the number of all intent labels, Represents the current dialogue sentence u t The semantic slot label corresponding to each word in , ns represents the number of all semantic slot labels.

[0111] This example is a neural network model, which can combine the domain knowledge base KB and the conversation history information C to obtain the current conversation sentence u t The fusion of H all , and based on this, accurately predict the intention y of the current conversation intent and semantic slot y slot .

[0112] This example explicitly injects relevant knowledge into the current conversation text as natural language. At the same time, this example further filters the matching knowledge to limit the amount of knowledge and prevent noise caused by adding too much knowledge.

[0113] For the current dialogue sentence u t , retrieve the domain knowledge base KB that matches u t Entities in All knowledge triples that match Due to E allNot every triplet in E is related to this round of dialogue. all Randomly select k pieces of knowledge as candidate triples E.

[0114] E = random_select_k(E all )

[0115] In order to explicitly inject knowledge into the dialogue sentences in the form of natural language, this scheme connects the candidate triples with the entities in the current dialogue sentence to form a knowledge-sentence tree:

[0116]

[0117] At the same time, a specific position index is added to each word to save the structural information of the knowledge-sentence tree, which is denoted as P.

[0118] The dialogue encoding module is used to obtain the current dialogue representation that integrates knowledge and dialogue history, and provides the basic feature vector for the multi-task learning module. This module selects the pre-trained model BERT as the basic encoder, which includes two parts: knowledge-sentence tree encoding and dialogue history encoding.

[0119] 1) Knowledge-Sentence Tree Encoding

[0120] After the current dialogue sentence passes through the knowledge injection module, it becomes a sentence tree with added knowledge. Therefore, the tree structure needs to be converted into a sequence form that can be input to BERT. The expanded form of the knowledge-sentence tree is as follows Figure 2 The key role is played by the position index that preserves the knowledge-sentence tree structure information.

[0121] However, adding knowledge directly into sentences as branches as part of natural language will split the original semantics and bring knowledge noise. Therefore, it is necessary to construct a reasonable attention matrix so that the knowledge on the branches can only affect the entities associated with them, but not other words in the original sentence. This solution first constructs a visual matrix M to represent the influence relationship between the associated knowledge and the original sentence. The visual matrix M is defined as:

[0122]

[0123] Among them, w i ⊕w j Indicates w i and w j On the same branch, Indicates w i and w j On the same branch.

[0124] Secondly, in order to make full use of the visual relationship of the sentence tree structure in M, M is added to the self-attention mechanism of BERT, and its calculation process is as follows:

[0125] Q i+1 ,K i+1 ,V i+1 =h i W q ,h i W k ,h i W v

[0126]

[0127] h i+1 =S i+1 V i+1

[0128] Among them, W q ,W k ,W v is a trainable model parameter, h i is the hidden state of the ith mask-self-attention block, d k is the scale factor and M is the visual matrix.

[0129] For the knowledge obtained by the knowledge injection module - sentence tree SentTree t , combined with the defined positional encoding P and visual matrix M, the output can be obtained by inputting BERT:

[0130] H knowSent =BERT(SentTree t ,P,M)

[0131] Among them, H knowSent ={h s,0 ,h s,1 ,…,h s,l+k} represents the knowledge-enhanced representation of the current dialogue, h s,i represents the latent vector representation corresponding to the i-th word, l represents the length of the original sentence, k represents the length of the inserted knowledge, so l+k is the sequence length after the knowledge-sentence tree is expanded.

[0132] 2) Conversation history encoding

[0133] The conversation history C=[u1,u2,…,u t-1 ] and insert a special token [CLS] for classification before the first word. The updated conversation history text is:

[0134] C=[[CLS],w 1,1 ,w1,2 ,…,w 1,n ,…,w i,j ,…,w t-1,m ]

[0135] Then, based on BERT encoding, the output can be obtained:

[0136] H context =BERT(C)

[0137] Among them, H context ={h c,0 ,h c,1 ,…,h c,cl} represents the vector representation of the conversation history, cl represents the length of the conversation history, and this example takes the hidden state h of the first special token [CLS] c,0 , as the overall vector representation of the conversation history.

[0138] The multi-task learning module trains the dialogue understanding model based on the current dialogue representation of the fusion knowledge and dialogue history. Dialogue intent recognition and semantic slot filling are two basic tasks of dialogue understanding. Due to the interdependence of these two tasks, using a unified model for joint training can learn better and richer representations, thereby improving the effects of both tasks at the same time. In addition, in order to prevent related knowledge from occupying the main semantics and better play the auxiliary understanding function of knowledge, this example adds the related knowledge recognition task, which is jointly trained with the two main tasks of dialogue intent recognition and semantic slot filling, and finally achieves accurate understanding of the current dialogue semantics. Figure 2 This module first converts the output of the dialogue encoding module, i.e., the knowledge-enhanced dialogue representation H knowSent ={h s,0 ,h s,1 ,…,h s,l+k} and the conversation history representation h c,0 Perform concatenation to obtain the current dialogue representation that integrates knowledge and dialogue history. Secondly, combine different tasks for multi-task learning to improve the accuracy of dialogue understanding.

[0139] 1) Intent recognition task

[0140] The task of intent recognition aims to determine the purpose of the current conversation. This solution defines it as a multi-label classification problem. The knowledge-enhanced conversation is represented by H knowSent ={h s,0 ,h s,1 ,…,h s,l+k}The encoding corresponding to the special token [CLS] in h s,0 and the conversation history encoding h c,0 Concatenate and get the overall representation h of the current dialogue that integrates knowledge and dialogue history all,0The specific process of the intention recognition task is expressed as:

[0141] h all,0 =concat(h s,0 ,h c,0 )

[0142]

[0143] in, represents the probability distribution of the current dialogue sentence on the intent label, W intent are trainable model parameters.

[0144] 2) Semantic slot filling task

[0145] The semantic slot filling task is used to mark the meaningful content in the dialogue text. This example defines it as a sequence labeling problem. Since this example explicitly injects knowledge into the dialogue sentence, the content of the current dialogue sentence changes. Therefore, it is necessary to add the label "O" to the word-unit belonging to the associated knowledge in the semantic slot filling task to distinguish it from the entity slot in the original sentence, such as Figure 3 As shown in [Main Task 1: New Semantic Slot Labels], first, encode the conversation history h c,0 And knowledge enhancement representation:

[0146] H knowSent ={h s,0 ,h s,1 ,…,h s,l+k The encoding h corresponding to each word in} s,i Concatenate and get the representation H of each word unit fusion knowledge and dialogue history all ={h all,0 ,h all,1 ,…,h all,l+k}. Secondly, multi-classification is performed on the semantic slot labels. The specific process is as follows:

[0147] h all,i =concat(h s,i ,h c,0 ),i∈1,…,l+k

[0148]

[0149] in, represents the probability distribution of the i-th word on the semantic slot label, W slot are trainable model parameters.

[0150] 3) Related knowledge identification task

[0151] The task of identifying related knowledge aims to distinguish the original sentence from the inserted knowledge. This solution defines it as a sequence labeling problem. For the word-units belonging to related knowledge, set their labels to "[ENT]", and for other word-units, set their labels to "O", such as Figure 3 As shown in [Auxiliary Task: Associated Knowledge Labels] in [Auxiliary Task: Associated Knowledge Labels]. Similar to the semantic slot filling task, the fusion representation h corresponding to each word is all,i Classify on the associated knowledge tag:

[0152]

[0153] in, represents the probability distribution of the i-th word on the labels "O" and "[ENT]", W entity are trainable model parameters.

[0154] 4) Multi-task learning

[0155] Since all three tasks are classification problems, this scheme uses the cross entropy loss function as the objective function of the three tasks:

[0156]

[0157]

[0158]

[0159] The model loss L all The losses are composed of three parts:

[0160] L all =L intent +L slot +λL entity

[0161] Among them, λ is the reconciliation coefficient, which is used to reconcile the weights between the two main tasks of intent recognition and semantic slot filling and the auxiliary task of associated knowledge recognition.

[0162] The embodiment of the present invention limits the number of candidate knowledge through random knowledge screening, prevents the injection of too much knowledge and the division of the original semantics, and ensures the diversity of the injected knowledge. After that, the knowledge is inserted into the original sentence in the form of natural language, the target data is constructed, and the explicit fusion of related knowledge is realized; the effect of task-based dialogue understanding is improved by explicit knowledge injection, and a method for dynamically inserting knowledge is provided for dialogue understanding. The model does not need to repeatedly encode and update the knowledge, and can also achieve "plug and play" in the face of new knowledge, meeting the needs of dynamic development of knowledge sources. The embodiment of the present invention further screens the matching knowledge, effectively limits the number of knowledge, avoids the injection of too much knowledge and destroys the original semantics, and ensures the diversity of added knowledge. In addition, the auxiliary task of related knowledge identification is added for multi-task learning to avoid the injected knowledge covering the original semantics. Compared with the general model, the embodiment of the present invention requires less learning time to achieve the same training accuracy, and has more efficient domain transfer capabilities. At the same time, in task-oriented dialogue understanding, the embodiment of the present invention has higher experimental accuracy and better use effect than the baseline model.

[0163] An embodiment of the present invention also provides a multi-task learning dialogue understanding model, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, each process in the multi-task learning dialogue understanding method embodiment as described above is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0164] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the embodiment of the multi-task learning dialogue understanding method described above is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A device that specifies functions in one or more processes and / or one or more blocks.

[0167] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions may also be loaded onto a computer or other programmable data processing device to cause the computer or other programmable device to execute a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-task learning dialogue understanding method, characterized in that: include: Retrieving knowledge triples matching the entities in the first dialogue sentence from the domain knowledge base; Inserting k randomly selected knowledge triples into the first dialogue sentence explicitly in natural language form to obtain target data; k is an integer greater than 1; Based on the target data, the intent of the first dialogue sentence is identified.

2. The method according to claim 1, characterized in that The randomly selected k knowledge triples are explicitly inserted into the first dialogue sentence in natural language form to obtain target data, including: Connect k knowledge triples with the entities in the first dialogue sentence to form a knowledge-sentence tree; Add a position index to each word in the knowledge-sentence tree form to construct the target data in the knowledge-sentence tree form.

3. The method according to claim 1 or 2, characterized in that: Identifying the intention of the first dialogue sentence according to the target data includes: Encode the target data and the conversation history respectively to obtain the feature vector corresponding to the target data and the feature vector corresponding to the conversation history; Concatenate the feature vector corresponding to the target data and the feature vector corresponding to the conversation history to form a current conversation representation that integrates knowledge and conversation history; Based on the current dialogue representation of the fused knowledge and dialogue history, multi-task learning is performed using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence.

4. The method according to claim 3, characterized in that Encode the target data to obtain the feature vector corresponding to the target data, including: Constructing a visual matrix, wherein the visual matrix is ​​used to represent the influence relationship between the associated knowledge triple and the first dialogue sentence; Adding the visual matrix to the self-attention mechanism of the pre-trained model BERT to obtain an enhanced BERT; The target data in the form of a knowledge-sentence tree, each word in the knowledge-sentence tree form is added with a position index, and the visual matrix is ​​input into the enhanced BERT, and the enhanced BERT outputs a feature vector corresponding to the target data.

5. The method according to claim 3, characterized in that: Encode the conversation history to obtain the feature vector corresponding to the conversation history, including: Concatenate each sentence in the conversation history and insert a special token for classification before the first word to get the updated conversation history text; The updated conversation history text is input into the pre-trained model BERT, and BERT outputs the feature vector corresponding to the conversation history.

6. The method according to claim 3, characterized in that Based on the current dialogue representation of the fused knowledge and dialogue history, multi-task learning is performed using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence, including: Based on the current dialogue representation of fused knowledge and dialogue history, the intention of the first dialogue sentence is identified through joint training of dialogue intent recognition and semantic slot filling tasks and auxiliary learning of associated knowledge recognition tasks.

7. A multi-task learning dialogue understanding model, characterized in that: include: A knowledge injection module, used to retrieve knowledge triples matching the entities in the first dialogue sentence from the domain knowledge base; and explicitly inserting randomly selected k knowledge triples into the first dialogue sentence in natural language form to obtain target data; k is an integer greater than 1; An understanding module is used to identify the intention of the first dialogue sentence based on the target data.

8. The model according to claim 7, characterized in that The knowledge injection module is further used for: Connect k knowledge triples with the entities in the first dialogue sentence to form a knowledge-sentence tree; Add a position index to each word in the knowledge-sentence tree form to construct the target data in the knowledge-sentence tree form.

9. The model according to claim 7 or 8, characterized in that The understanding module includes: The dialogue encoding module is used to encode the target data and the dialogue history respectively to obtain the feature vector corresponding to the target data and the feature vector corresponding to the dialogue history; and concatenate the feature vector corresponding to the target data and the feature vector corresponding to the dialogue history to form a current dialogue representation that integrates knowledge and dialogue history; A multi-task learning module is used to perform multi-task learning based on the current dialogue representation of the fused knowledge and dialogue history, using dialogue intent recognition, semantic slot filling tasks, and associated knowledge recognition tasks to recognize the intent of the first dialogue sentence.

10. The model according to claim 9, characterized in that The dialogue encoding module is further used for: Constructing a visual matrix, wherein the visual matrix is ​​used to represent the influence relationship between the associated knowledge triple and the first dialogue sentence; Adding the visual matrix to the self-attention mechanism of the pre-trained model BERT to obtain an enhanced BERT; The target data in the form of a knowledge-sentence tree, each word in the knowledge-sentence tree form is added with a position index, and the visual matrix is ​​input into the enhanced BERT, and the enhanced BERT outputs a feature vector corresponding to the target data.

11. The model according to claim 9, characterized in that The dialogue encoding module is further used for: Concatenate each sentence in the conversation history and insert a special token for classification before the first word to get the updated conversation history text; The updated conversation history text is input into the pre-trained model BERT, and BERT outputs the feature vector corresponding to the conversation history.

12. The model according to claim 9, characterized in that The multi-task learning module is further used to: Based on the current dialogue representation of fused knowledge and dialogue history, the intention of the first dialogue sentence is identified through joint training of dialogue intent recognition and semantic slot filling tasks and auxiliary learning of associated knowledge recognition tasks.

13. A multi-task learning dialogue understanding model, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the multi-task learning dialogue understanding method according to any one of claims 1 to 6 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the multi-task learning dialogue understanding method as described in any one of claims 1 to 6 are implemented.

Images