Unified dialogue understanding method and framework
By uniformly transforming the dialogue understanding task into a generative task and training it with a unified model, the problem of independent modeling for different tasks is solved, achieving better performance and scalability, especially in dialogue understanding capabilities under few-shot and zero-shot conditions.
Patent Information
- Application Number
- CN202210342533.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-03-31
AI Technical Summary
In existing technologies, different dialogue understanding tasks need to be modeled independently, resulting in poor scalability and high memory requirements. They also fail to effectively explore the influence between tasks, and generative methods do not deeply consider the impact of multi-task training methods.
All types of dialogue understanding tasks are uniformly converted into generative tasks, trained using a unified generative model, and parameters are shared through a multi-task learning algorithm. The model outputs task identifiers and answers, and uses methods such as homomorphic uncertainty weights and gradient normalization gradient norms to balance task weights.
It enables knowledge sharing across multiple dialogue understanding tasks, improving overall performance, especially in few-shot and zero-shot scenarios. It also improves the learnable weighting method, enhancing the exploration of influences between tasks and scalability.
Smart Images

Figure CN114896988B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent speech, and in particular to a unified dialogue understanding method and framework. Background Art
[0002] The development of conversational systems plays an important role in the popularization of intelligent devices, such as smart assistants. In recent years, people have become more and more interested in neural dialogue systems. Dialogue understanding is the core technology in neural dialogue systems, which aims to accurately analyze dialogues from different fine-grained perspectives. However, there are many types of dialogue understanding tasks, such as determining the corresponding answer given a question stem and question, or determining short key sentences given a long text. Due to the different formats of each task, these DU (Dialogue Understanding) tasks are still learned independently.
[0003] In the dialogue task learning, there are:
[0004] Application of multi-task learning in dialogue understanding: Use the corpus of different understanding tasks to train a shared dialogue encoding model. Different tasks still have their own independent decoding modules. For classification problems, the dialogue representation represented by special symbols is used for classification calculations. For sequence labeling problems, labeling calculations are performed on each word. For generation problems, there will be independent autoregressive decoding calculations.
[0005] Application of generative methods in dialogue modeling: Generative dialogue modeling generally refers to modeling the dialogue content using a language model, turning dialogue modeling into a dialogue generation problem, and then using other annotation information to help improve the quality of dialogue generation.
[0006] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:
[0007] Regarding the application of multi-task learning in dialogue understanding: Different dialogue understanding problems still need to be set as different dialogue understanding tasks, for example, dialogue intent recognition is a classification task, dialogue slot filling is a sequence labeling task, and dialogue abbreviation is a generation task. Different tasks have their own independent set of parameters, which is not conducive to exploring the impact between different tasks because it is still limited by the influence of the model. In addition, this multi-task training method has poor scalability, and different tasks still need to be modeled independently, with high memory requirements, which almost increases linearly with the number of tasks.
[0008] The application of generative methods in dialogue modeling: It only takes the dialogue generation problem as the main problem, without considering the impact of different multi-task training methods on different dialogue understanding tasks during the training process. It also does not discuss in depth the impact of a large number of other forms of dialogue understanding tasks. Summary of the invention
[0009] In order to at least solve the problem that different tasks in the prior art still need to be modeled independently, which is not conducive to different dialogue understanding tasks, the performance requirements of all dialogue understanding tasks are considered globally, but other tasks are used to improve the performance of the dialogue generation task, without considering the impact of different multi-task training methods. In the first aspect, the embodiment of the present invention provides a unified dialogue understanding method, including:
[0010] Convert various types of dialogue understanding tasks into multiple generative dialogue tasks according to a unified generative paradigm;
[0011] The multiple generative dialogue tasks are input into a task unified generative model, and task identifiers and task answers corresponding to the multiple generative dialogue tasks are output to solve dialogue understanding tasks of various categories.
[0012] In a second aspect, an embodiment of the present invention provides a unified dialogue understanding framework, including:
[0013] A unified conversion program module, which is used to convert various categories of dialogue understanding tasks into multiple generative dialogue tasks according to a unified generation paradigm;
[0014] The dialogue understanding program module is used to input the multiple generative dialogue tasks into the task unified generative model, and output task identifiers and task answers corresponding to the multiple generative dialogue tasks to solve dialogue understanding tasks of various categories.
[0015] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the unified dialogue understanding method of any embodiment of the present invention.
[0016] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the unified dialogue understanding method of any embodiment of the present invention are implemented.
[0017] The beneficial effects of the embodiments of the present invention are as follows: the method proposes a unified generative dialogue understanding framework for sharing knowledge in multiple dialogue understanding tasks. In order to alleviate the problem of bias generation, the existing learnable weight method is improved, and the method can achieve the best overall performance. Compared with the existing model, the UniDU method of the method achieves better performance on multiple DU tasks. The influencing factors are further studied in depth. Finally, the experimental results show that the UniDU model of the method can also achieve excellent performance in small sample and zero sample settings. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 is a flow chart of a unified conversation understanding method provided by an embodiment of the present invention;
[0020] Figure 2 This is a schematic diagram of time slot filling task conversion of a unified dialogue understanding method provided by an embodiment of the present invention;
[0021] Figure 3 This is a diagram of a dialog state tracking task conversion of a unified dialog understanding method provided by an embodiment of the present invention;
[0022] Figure 4 It is a schematic diagram of intention detection task conversion of a unified dialogue understanding method provided by an embodiment of the present invention;
[0023] Figure 5 It is a schematic diagram of multiple task conversions of a unified dialogue understanding method provided by an embodiment of the present invention;
[0024] Figure 6 is a structural diagram of a unified dialogue understanding method provided by an embodiment of the present invention;
[0025] Figure 7 It is a schematic diagram of eight learning strategies used in the test results of five DU tasks of a unified dialogue understanding method provided by an embodiment of the present invention;
[0026] Figure 8 It is a schematic diagram of the overall performance of a hybrid, HWU and MATS method of a unified dialogue understanding method provided by an embodiment of the present invention;
[0027] Fig. 9 It is a schematic diagram of the effect of each task corpus of a unified dialogue understanding method provided by an embodiment of the present invention;
[0028] Fig.10 It is a unified understanding schematic diagram of different studies on a unified dialogue understanding method provided by an embodiment of the present invention;
[0029] Fig.11 It is a schematic diagram of the encoding and decoding structure of different pre-trained language models of a unified dialogue understanding method provided by an embodiment of the present invention;
[0030] Fig.12 It is a schematic diagram of the results of a unified dialogue understanding method provided by an embodiment of the present invention on BART and UniDU;
[0031] Fig.13 is a structural diagram of a unified dialogue understanding framework provided by an embodiment of the present invention;
[0032] Fig.14 A schematic structural diagram of an embodiment of an electronic device for unified dialogue understanding provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] like Figure 1 FIG. 1 is a flowchart of a unified method for understanding dialogue provided by an embodiment of the present invention, comprising the following steps:
[0035] S11: Convert various types of dialogue understanding tasks into multiple generative dialogue tasks according to a unified generative paradigm;
[0036] S12: Input the multiple generative dialogue tasks into a task unified generative model, and output task identifiers and task answers corresponding to the multiple generative dialogue tasks to solve dialogue understanding tasks of various categories.
[0037] In this embodiment, the conversation is represented as C = (H n , U n ), where H n =(U1, U2, ..., U n-1 ) indicates the transition of the first n-1 dialogue history utterances. n is the utterance of the nth turn, which can consist of multiple sentences uttered by a speaker. For task-oriented dialogues, the domain scope is restricted by the dialogue ontology designed by dialogue experts. The ontology O consists of the dialogue domain D = {D}, domain slot S = {S}, and user intent candidate I = {I}.
[0038] For step S11, all types of dialogue understanding tasks are converted into multiple generative dialogue tasks according to a unified generative paradigm, where the categories of dialogue understanding tasks include: dialogue summary, dialogue completion, intention detection, time gap filling and dialogue state tracking. Specifically:
[0039] DS (Dialogue Summary) aims to extract important information from a conversation. This is a typical generation problem that takes the entire conversation context C as input and generates a summary description. DS requires the model to focus on the entire conversation process and important concepts.
[0040] The purpose of DC (Dialogue Completion) is to alleviate the problems of coreference and information omission that often occur in the context of dialogue. It is also a typical generation task. The input dialogue history H n and the current discourse U n , and then infer the current utterance U n DC requires the model to pay attention to the connection between the current utterance and the dialogue history.
[0041] SF (Slot Filling) is a slot type to extract entities mentioned by the user. It is a slot tagging problem, i.e., the utterance is tagged in IOB (Inside, Outside and Beginning) format. The input is just the current utterance.
[0042] ID (Intent Detection) is to identify intent from a predefined abstract intent expression I. It is usually described as a classification problem. The input is the current utterance U n , the output is the likelihood distribution of all intent candidates I.
[0043] The purpose of DST (Dialogue State Tracking) is to record the user's constraints. It consists of three sets of domain time slot values. For example, hotel pricecheap means that the user wants a cheap hotel. The DST input of the nth round is the previous n rounds (U1, ..., U n ).
[0044] The above five different dialogue understanding tasks are unified into a sequence-to-sequence format. Then the composition of each task is explained in detail, especially how to use intent detection, time slot filling and dialogue state tracking as generation tasks.
[0045] As an implementation method, various types of dialogue understanding tasks are converted into multiple generative dialogue tasks including task identifiers, dialogue content, and task descriptions according to a unified generation paradigm;
[0046] In this embodiment, the input of UniDU (Unified Generative Dialogue Understanding) includes three parts: task identification, dialogue content, and task query. Task identification is represented by special tags, for example, a dialogue summary identified by "[DS]". Dialogue content refers to task-related input, such as the dialogue history of the dialogue summary. Task query can be regarded as a task-specific prompt, including task definition and domain-related information. The output of UniDU has two elements: task identification and query answer. The query answer is the understanding result of the task query given by the dialogue content. The unified input and output can be formalized as:
[0047] Input: [T1] dialogue content [C] task query
[0048] Output: [T1] query answer
[0049] Among them, "[C]" is an independent character, and "[TI]" is a task identifier (which can be replaced by "[DS]", "[DC]", "[SF]", "[ID]", and "[DST]", corresponding to dialogue summary, dialogue completion, time slot filling, intent detection, and dialogue state tracking, respectively). At inference time, the UniDU model must first predict the task identification.
[0050] For step S12, after the task unified generative model is used, task identifiers and task answers corresponding to multiple generative dialogue tasks are output, for example:
[0051] Conversation summarization and dialogue completion are generative tasks. The dialogue content input is the entire conversation context C and multi-turn discourse H n Since these two tasks are independent of the conversation domain, there is no domain information in the task query. For the conversation summary, the task query is "What is the summary of this conversation?". For the completion conversation, the query is "U n What is the semantic completion statement of ? ", where U n is the tth sentence. In the output, their understanding answers are the annotated dialogue summary and the rewritten utterance, respectively.
[0052] The slot filling task requires the model to extract all mentioned slot values and their slot types in a sentence. In the method, the UniDU model predicts the value slot by slot, which is an iterative generation process of the slot candidate list. Figure 2 The slot filling formats of the conventional method and the present method are shown.
[0053] It should be clear that not all candidate lists are listed here. Generally speaking, each sample name can be processed in the same way, which will not be repeated here. It can be formalized as follows:
[0054] Input: [SF]U n [C]what is s of d
[0055] Output: [SF] slot value
[0056] where s and d are predefined slots and domains. If s is in U n If there is no value in s, the slot value will be "not ignored". If s has multiple values, they will be separated by commas in the slot value. When the value is "not mentioned", it is called a negative sample, otherwise it is a positive sample. In order to balance the ratio of positive and negative samples during training, the ratio is set to less than 2:1. If the number of negative samples exceeds the threshold, the number of randomly drawn negative samples is twice the number of positive samples.
[0057] For the dialogue state tracking task, the classification method always achieves better performance than the generation method. However, under the UniDU framework, DST is also described as a slot value generation task similar to the slot filling task. The DST task format is as follows: Figure 3 As shown, the output of the original DST model is the distribution of all candidate values of the time slot. The input and output of the DST task under UniDU can be formalized as:
[0058] Input: [DST](Hn,Un)[C]what is the user's constraint about s of d
[0059] Output: [DST] slot value
[0060] Among them (H n , U n ) is the dialogue context. If slot s of domain d is not in a dialogue state, its value is “not mentioned”, which is a negative sample. Different utterances are separated by a special token “[T]” in the input. During training, the ratio of negative samples to positive samples is also set to less than 2:1.
[0061] The original method usually formulates the intent detection task as an intent classification problem and outputs the distribution of all candidate intents. The UniDU model directly generates the intent name of the current utterance, which can be formalized as:
[0062] Input: [ID]U n [C]what is the user's intent on domain d
[0063] Output: [SF]intent name
[0064] The domain d is usually known in advance. Specific examples of the UniDU format of the prior art and the present method are as follows: Figure 4 shown.
[0065] Not all intents are listed in the above figure. In order to integrate the generalization ability into the UniDU model, negative samples are also constructed for the intent detection task. The intent name of the negative sample is "undefined", where the input utterance is not sampled from the out-of-domain dialogue. The ratio of negative and positive samples is set to 2:1. So far, all five dialogue understanding tasks are defined as a unified sequence-to-sequence generation task. Specific examples are as follows Figure 5 As shown. Thus, the corresponding task identifier and task answer are obtained.
[0066] It can be seen from this implementation that all dialogue understanding tasks are unified into generating questions, outputting task identifiers and task answers. The proposed UniDU method can use a unified task generative model to process any type of dialogue task in the example, with stronger scalability and more accurate dialogue understanding performance under small sample and zero sample conditions.
[0067] As an implementation method, the task unified generative model is trained using the following steps:
[0068] Convert the training data of each category of dialogue understanding tasks into multiple generative dialogue task training data according to a unified generation paradigm;
[0069] Performing multi-task joint training on the task unified generative model using the multiple generative dialogue task training data, and outputting prediction task identifiers and prediction task answers corresponding to the multiple generative dialogue task training data, wherein the multiple generative dialogue tasks share the same training parameters;
[0070] The task unified generative model is optimized based on the error between the benchmark task identifier and the benchmark task answer training and the predicted task identifier and the predicted task answer, until the training ends when the error converges.
[0071] In this embodiment, the above multiple dialogue understanding tasks are formulated as a unified framework. Since the output space gap between the above multiple DU tasks is large, how to effectively train five different tasks at the same time becomes an important issue.
[0072] During the training process, the training data of the dialogue understanding tasks of various categories are also converted into multiple generative dialogue task training data according to the unified generation paradigm. The specific conversion process has been illustrated in the above steps and will not be repeated here. Multi-task training is performed on these multiple generative dialogue task training data. Specifically:
[0073] The training strategy of the multi-task joint training includes:
[0074] Assign equal weights to all generative dialogue tasks, implementing an average and policy; and / or
[0075] Using heuristic training plans to plan the training process for different generative dialogue tasks to achieve configurable manual scheduling strategies; and / or
[0076] Determine the loss weights of different generative dialogue tasks, and use them to balance the learning weight strategy for global optimization of each loss weight. The learning weight strategy for global optimization of each loss weight includes: homomorphic uncertainty weight, gradient normalization gradient norm.
[0077] In this embodiment, the multi-task training strategies can be divided into three categories: average sum method, manual scheduling method and learnable weight method.
[0078] The average sum method assigns all samples with the same weight. In other words, the losses of different samples are directly averaged, and the formula is Where T is the number of tasks, L t is the loss of the Tth task.
[0079] Manual scheduling methods design a heuristic training schedule to plan the learning process of different tasks. For example, curriculum learning is a typical manual scheduling method, which first trains simple samples and then adds more complex cases. The manual scheduling method can be expressed as In the formula, is an indicator function whose value is either 0 or 1.
[0080] Learnable weights aim to parameterize the loss weights of different tasks. The goal of parameterized weights is to balance the impact of task instances, avoid the model being biased towards one or more tasks, and achieve global optimization. There are two classic learnable weight algorithms: homomorphic uncertainty weights (HUW) and gradient normalized gradient norm (GradNorm). For these tasks, the loss function is formulated as Where W t Is the learnable weight greater than 0. In the HUW algorithm, the weight is updated as the following loss function:
[0081]
[0082] Among them, log(Wt ) is used to adjust the weights, which is applicable to both regression and classification tasks. The motivation of the gradient norm method is to slow down the learning scale of tasks with large gradients and fast convergence speed.
[0083] After obtaining the prediction task identifier and the prediction task answer, the benchmark and prediction are further used for training. In the above HUW formula, the learnable weight Wt depends only on the corresponding task. Therefore, the weight can be regarded as the task Wt. φ (t), where а is a parameter shared among the five tasks. In the UniDU framework, the five tasks share the same encoder-decoder model, which is a function with constant weight W φ (t). The task format depends on the task attributes, such as input, output, and data size. In order to extract the features of the five tasks, a vector is manually designed as the task feature to represent a task. Each dimension in the task function has its physical meaning related to the model-agnostic setting. Function parameters can be shared between different tasks, and the weights are no longer independent of each other like the original learnable weight method. The improved formalization based on the above HUW formula is:
[0084]
[0085] The training is finally completed based on the above examples.
[0086] In simple terms, the structure of this method is as follows Figure 6 As shown:
[0087] 1. First, the dialogue understanding task is transformed into a generation problem, where the input part includes three components: a special identifier [XX] for the task, the dialogue content (single or multiple rounds) and the task description. The output part is the task identifier and the task answer.
[0088] 2. All dialogue understanding tasks share the same parameter model and perform joint training with multi-task learning algorithms
[0089] 3. When making inferences, output the corresponding task identifier and the answer to the corresponding question to solve all dialogue understanding tasks.
[0090] The proposed method is experimentally illustrated and experiments are conducted on ten dialogue understanding corpora. There are two corpora for each task. The proposed UniDU framework is evaluated with eight different training strategies. Compared with well-designed models, the proposed UniDU can achieve better performance on five benchmarks. Then, different factors affecting the performance of the UniDU model are deeply analyzed, including DU tasks, unified format, and pre-trained language models. Finally, some experiments are conducted to verify the generalization ability of UniDU.
[0091] There are 10 dialogue understanding corpora covering five tasks: dialogue summarization (DS), dialogue completion (DC), slot filling (SF), intent detection (ID), and dialogue state tracking (DST). Two well-studied corpora are selected for each task: one is an evaluation corpus and the other is an auxiliary corpus.
[0092] Dialogue Summarization: SAMSUM and DIALOGSUM datasets are selected. A common metric for summarization tasks is the score, which measures the overlap between the n items in the generated summary and the reference summary.
[0093] Conversation completion: uses tasks and languages. The metrics are BLEU score and exact match (EM) accuracy. BLEU measures how similar the rewritten sentence is to the baseline sentence. Exact match is the rate at which the generated data is exactly equal to the baseline.
[0094] Intent Detection: Experiments are conducted on BANKING77 and HWU64, where 77 and 64 represent the number of predefined intents. The evaluation metric is detection accuracy (ACC).
[0095] Slot filling: We choose to experiment on RESTAURANTS8K and SNIPS. The F1 score for extracting the correct span for each user's utterance. It should be noted that the correct prediction of negative samples is not calculated in the F1 score, which is comparable to traditional methods.
[0096] Dialogue state tracking: WOZ2.0 and MULTIWOZ2.2. The metric is the joint goal accuracy (JGA), which measures the percentage of successes among all dialogue turns, where a turn is considered successful if and only if all slot values are correctly predicted. Note that only the “hotel” domain data of MULTIWOZ2.2 is used during the training phase.
[0097] The multi-task training strategies can be divided into three categories: average sum, manual planning, and learnable weights. Before introducing the MTL (Multi-task Learning) training method, there is an intuitive baseline based on its own data, named ST (single training). In ST, the sequence-to-sequence model is trained only on five evaluation datasets respectively. In the average sum method, there are two training strategies: TT (task transfer learning) and MIX (mixture learning). The purpose of task transfer learning is to improve learning performance by using external data from auxiliary corpora with the same task setting. This is the main reason for selecting two corpora for each task. Mixed learning directly mixes all training samples from 10 corpora together. In both methods, the learning weights of each sample are evenly distributed. In the curriculum, according to the two manual learning methods. From the input perspective, the five tasks can be divided into three categories: utterance-level input for intent detection and slot filling, flip-level input for dialogue completion and dialogue state tracking, and dialogue-level input for dialogue summarization.
[0098] The order of input becomes gradually more complex: utterance layer, turn layer, and dialogue layer. Therefore, the intuitive method (abbreviated as CL) trains five tasks in this order. It should be noted that the previous data will be retained in the next training stage. From the perspective of task setting, dialogue summary and dialogue completion are domain-independent tasks. The other three tasks are domain-related tasks. There is another training route: from general tasks to domain-specific tasks. Among the learnable weight methods, we evaluated three methods introduced in Section 4: GradNorm, HWU, and our proposed MATS (model-agnostic training strategy).
[0099] In the method, BART (Bidirectional and Auto-Regressive Transformers) is used as the backbone of the unified encoder-decoder model. The BART model is implemented by the HuggingFace library. All experiments were performed on a 2080TI GPU with 11G memory. Each experiment was performed for 60 epochs, which took 72 hours. The batch size was 32, and the gradient accumulation strategy was adopted (updated every 8 epochs). The learning rates of the unified model and the learnable weights were 1e-5 and 1e-4, respectively. In the MATS method, the weight function consists of two linear layers with ReLU activation function, and its hidden size is 64.
[0100] like Figure 7In this paper, the best evaluation performance of five tasks and eight training strategies are reported. Experimental results show that different training strategies have a great impact on the performance of the five tasks under the UniDU framework. Our method achieves the best or near-best performance except for dialogue summarization. On atypical generation tasks (intent detection, slot filling, and dialogue state tracking), UniDU with the MATS method can achieve promising improvements compared to well-designed models. A simple task transfer learning method (TT) cannot significantly improve the learning performance compared to single training. Mixing operations brings consistent performance improvements on the five tasks. However, the improvements are still limited compared to TT, except for completing dialogues. Compared with MATS proposed in our method, MIX tends to converge to more complex DU tasks (dialogue summarization and dialogue completion). Two manual scheduling methods (G2S and CL) do not have any obvious advantages. Among the learnable weight methods, GradNorm only achieves excellent performance on dialogue summarization. HWU achieves performance improvements on intent detection, slot filling, and dialogue state tracking. We continue to fine-tune the best UniDU model (underlined) on the corresponding corpus. We find that only dialogue summarization and dialogue completion have significant performance improvements, which also reflects the necessity of the UniDU framework for simpler generative tasks.
[0101] exist Figure 7 In , we report the task-specific performance of the UniDU model, whose checkpoints are selected by task-specific metrics. Figure 8 The uniform performance of the hybrid, HWU, and MATS methods on the five tasks is shown. A single checkpoint of the UniDU model is evaluated, which has the highest overall evaluation score on the five tasks. The overall score is Figure 8 The average values of the five main indicators are shown in Figure 2. It can be seen that the MATS proposed in this method achieves the highest overall performance and also achieves the best performance in the four DU tasks.
[0102] This method analyzes the factors that affect the performance of the UniDU model, including DU tasks, unified format, and pre-trained language models.
[0103] In order to verify the effect of the dialogue understanding task, we directly delete one of the five DU corpora and use Fig. 9The MATS method shown in Figure 1 trains the UniDU model. In general, the five DU tasks benefit each other, except that the dialogue summary has a negative impact on the dialogue state tracking task. The general dialogue summary task simply summarizes a dialogue into a sentence, while ignoring the information in a specific field. On the other hand, it is found that the dialogue completion task has the greatest impact on the other four DU tasks. Studies have shown that coreference and information omission are still the main factors affecting the ability of dialogue understanding. This phenomenon helps the dialogue understanding community pay more attention to the completion of the dialogue. For example, when pre-training a scaled dialogue model, the pre-trained task should be close to the dialogue completion task.
[0104] The dialogue understanding task is specified in the QA format. There is an intuitive alternative: the prefix format, where the task query is concatenated on the decoder side. At inference time, the decoder directly receives the task query and then generates an answer. Fig.10 Shown is a summary of the four tasks, Fig.10 Shown is a summary of the five tasks.
[0105] In order to verify the effects of different pre-trained backbones, the encoder of the UniDU model is initialized using random mechanism, BART and T5. Trans-B and Trans-L are represented as follows Fig.11 Randomly initialized transformers trained from scratch with the same parameters as the BART base model (BART-B) and BART large model (BART-L). T5-S and T5-B denote T5-small and T5-base, respectively. It can be seen that the pre-trained language models achieve an absolute performance gain compared to the randomly initialized models. BART-B achieves better performance than T5-S. T5-base performs best when the parameter ratio increases. The results show that large PLMs can significantly improve complex dialogue summarization tasks.
[0106] To further evaluate the generalization ability of the UniDU model, we first conduct experiments on a domain-dependent slot filling task with few-shot learning. We test the zero-word learning ability of UniDU on unseen dialogue data.
[0107] Few-shot learning: The UniDU model was selected, which achieved the best overall performance evaluation in the five tasks learned using the MATS method. For the slot filling task, another dialogue corpus DSTC8 was extended. The “bus” domain data in DSTC8 was selected, which was not seen during the training process of UniDU. This has obvious advantages, especially in extremely limited resource situations. When there is only 1% of training data, vanilla BART will disable learning, such as Fig.12 shown.
[0108] Zero-word learning performance: The UniDU model trained with the MATS method is validated on unseen “taxi” domain conversation data collected from the MULTIWOZ2.2 corpus. The UniDU model achieves an accuracy of 18.24% on ID, an F1 score of 39.69% on SF, and a JGA score of 1.6% on DST.
[0109] In summary, this method proposes a unified generative dialogue understanding framework (UniDU) for sharing knowledge in five dialogue understanding tasks. To alleviate the bias generation problem, the existing learnable weight method is improved, and the method can achieve the best overall performance. Compared with the well-designed models, our UniDU method achieves better performance on a total of five DU tasks. The influencing factors are further studied in depth. Finally, experimental results show that our UniDU model can also achieve excellent performance in few-shot and zero-shot settings.
[0110] like Fig.13 The figure is a schematic diagram of the structure of a unified dialogue understanding framework provided by an embodiment of the present invention. The system can execute the unified dialogue understanding method described in any of the above embodiments and be configured in a terminal.
[0111] The unified dialogue understanding framework 10 provided in this embodiment includes: a unified conversion program module, and a dialogue understanding program module 12.
[0112] Among them, the unified conversion program module 11 is used to convert dialogue understanding tasks of various categories into multiple generative dialogue tasks according to a unified generation paradigm; the dialogue understanding program module 12 is used to input the multiple generative dialogue tasks into a unified generation model of tasks, and output task identifiers and task answers corresponding to the multiple generative dialogue tasks to solve dialogue understanding tasks of various categories.
[0113] Furthermore, the unified conversion program module is used to:
[0114] The dialog understanding tasks of various categories are converted into multiple generative dialog tasks including task identification, dialog content, and task description according to a unified generation paradigm, wherein the categories of the dialog understanding tasks include: dialog summary, dialog completion, intent detection, time slot filling, and dialog state tracking.
[0115] Furthermore, the task unified generative model is trained using the following steps:
[0116] Convert the training data of each category of dialogue understanding tasks into multiple generative dialogue task training data according to a unified generation paradigm;
[0117] Performing multi-task joint training on the task unified generative model using the multiple generative dialogue task training data, and outputting prediction task identifiers and prediction task answers corresponding to the multiple generative dialogue task training data, wherein the multiple generative dialogue tasks share the same training parameters;
[0118] The task unified generative model is optimized based on the error between the benchmark task identifier and the benchmark task answer training and the predicted task identifier and the predicted task answer, until the training ends when the error converges.
[0119] Furthermore, the training strategy of the multi-task joint training includes:
[0120] Assign equal weights to all generative dialogue tasks, implementing an average and policy; and / or
[0121] Using heuristic training plans to plan the training process for different generative dialogue tasks to achieve configurable manual scheduling strategies; and / or
[0122] Determine the loss weights for different generative dialogue tasks, and use them to balance the learning weight strategy for global optimization of each loss weight.
[0123] The learning weight strategy for balancing the global optimization of each loss weight includes: homomorphic uncertainty weight and gradient normalization gradient norm.
[0124] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the unified dialogue understanding method in any of the above method embodiments;
[0125] As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:
[0126] Convert various types of dialogue understanding tasks into multiple generative dialogue tasks according to a unified generative paradigm;
[0127] The multiple generative dialogue tasks are input into a task unified generative model, and task identifiers and task answers corresponding to the multiple generative dialogue tasks are output to solve dialogue understanding tasks of various categories.
[0128] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the unified dialogue understanding method in any of the above method embodiments is executed.
[0129] Fig.14 FIG. 1 is a schematic diagram of the hardware structure of an electronic device of a unified dialogue understanding method provided by another embodiment of the present application, such as Fig.14 As shown, the device includes:
[0130] One or more processors 1410 and memory 1420, Fig.14 A processor 1410 is taken as an example. The device of the unified dialogue understanding method may further include: an input device 1430 and an output device 1440.
[0131] The processor 1410, the memory 1420, the input device 1430 and the output device 1440 may be connected via a bus or other means. Fig.14 The example of connecting through bus is taken in the following.
[0132] The memory 1420 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the unified dialogue understanding method in the embodiment of the present application. The processor 1410 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1420, that is, realizing the unified dialogue understanding method in the above method embodiment.
[0133] The memory 1420 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1420 may optionally include a memory remotely arranged relative to the processor 1410, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0134] The input device 1430 may receive input digital or character information. The output device 1440 may include a display device such as a display screen.
[0135] The one or more modules are stored in the memory 1420, and when executed by the one or more processors 1410, the unified dialogue understanding method in any of the above method embodiments is executed.
[0136] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.
[0137] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0138] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the unified dialogue understanding method of any embodiment of the present invention.
[0139] The electronic device of the embodiment of the present application exists in various forms, including but not limited to:
[0140] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.
[0141] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.
[0142] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.
[0143] (4) Other electronic devices with data processing functions.
[0144] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.
[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A unified approach to conversation understanding, including: Convert various types of dialogue understanding tasks into multiple generative dialogue tasks including task identification, dialogue content, and task description according to a unified generation paradigm; The plurality of generative dialogue tasks are input into a task-unified generative model, and task identifiers and task answers corresponding to the plurality of generative dialogue tasks are output to solve dialogue understanding tasks of various categories, wherein the task-unified generative model is trained by the following steps: Convert the training data of each category of dialogue understanding tasks into multiple generative dialogue task training data according to a unified generation paradigm; Performing multi-task joint training on the task unified generative model using the multiple generative dialogue task training data, and outputting prediction task identifiers and prediction task answers corresponding to the multiple generative dialogue task training data, wherein the multiple generative dialogue tasks share the same training parameters; The task unified generative model is optimized based on the error between the benchmark task identifier and the benchmark task answer training and the predicted task identifier and the predicted task answer until the training ends when the error converges, wherein the error loss function is L MATS : Where, T is the number of tasks for multi-task joint training, Lt is the loss of the task, and W φ () is a function with constant weight, and log() is an adjustment weight parameter for regression tasks and classification tasks.
2. The method according to claim 1, wherein: The categories of dialogue understanding tasks include: dialogue summarization, dialogue completion, intent detection, time slot filling, and dialogue state tracking.
3. A unified dialogue understanding framework, including: A unified conversion program module is used to convert various types of dialogue understanding tasks into multiple generative dialogue tasks including task identification, dialogue content, and task description according to a unified generation paradigm; The dialog understanding program module is used to input the multiple generative dialog tasks into the task unified generative model, and output task identifiers and task answers corresponding to the multiple generative dialog tasks to solve dialog understanding tasks of various categories, wherein the task unified generative model is trained by the following steps: Convert the training data of each category of dialogue understanding tasks into multiple generative dialogue task training data according to a unified generation paradigm; Performing multi-task joint training on the task unified generative model using the multiple generative dialogue task training data, and outputting prediction task identifiers and prediction task answers corresponding to the multiple generative dialogue task training data, wherein the multiple generative dialogue tasks share the same training parameters; The task unified generative model is optimized based on the error between the benchmark task identifier and the benchmark task answer training and the predicted task identifier and the predicted task answer until the training ends when the error converges, wherein the error loss function is L MATS : Where, T is the number of tasks for multi-task joint training, Lt is the loss of the task, and W φ () is a function with constant weight, and log() is an adjustment weight parameter for regression tasks and classification tasks.
4. The dialogue understanding framework according to claim 3, wherein: The categories of dialogue understanding tasks include: dialogue summarization, dialogue completion, intent detection, time slot filling, and dialogue state tracking.
