A method for constructing a universal dialogue corpus framework
By building a general dialogue corpus framework, using multiple heterogeneous data sets and cross-fusion technologies, the context coupling and data set quality problems of task-based dialogue systems in composite conversation scenarios are solved, and efficient model training and maintenance are achieved. Improves the robustness and scalability of the model.
Patent Information
- Application Number
- CN202210306071.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-03-25
AI Technical Summary
Existing task-based dialogue systems have problems such as high context coupling, difficult to measure data set quality, poor model maintainability and migration when dealing with composite conversation scenarios, and a single type of data set cannot cope with general conversation tasks.
A general dialogue corpus framework is proposed. By acquiring multiple heterogeneous data sets, defining new data structures, and using active learning and cross-fusion technology to construct the general data set of Encoder-Decoder, reducing the difficulty of supervised learning and Model training costs improve the robustness and scalability of the model.
It realizes the construction of a model that supports chat and task-based dialogue without the need for intent classification supervision, which reduces the cost of model training and maintenance, improves user participation and satisfaction, and improves the robustness and scalability of the model. , adapt to complex and changeable dialogue needs.
Smart Images

Figure CN114661865B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing dialogue systems, and specifically refers to a method for constructing a universal dialogue corpus framework. Background Art
[0002] Human-computer dialogue is an important application in the field of natural language processing, among which task-oriented dialogue system (TOD) and chatbot have attracted research interests from both academia and industry.
[0003] Chatbot refers to a system where the user does not have a clear purpose. The system only needs to chat with the user without completing a specific goal. There is often no explicit connection between rounds. Chatbots are generally fully data-driven. The dialogue material is constructed from flat text pairs, which are easy to obtain. They do not require external expert knowledge bases or APIs, and require few supervisory signals. Therefore, it is more difficult to measure the quality of the data set.
[0004] TOD means that the user has a clear purpose, and the system needs to access an external database through limited conversation rounds to guide the user to complete the task and achieve the purpose. For example, check the weather, recommend attractions, and book hotels. In actual development, TOD involves querying the database and tracking the completion of the task, so a complete conversation sample requires several rounds. Each round requires the user and the system to explicitly identify the slot-value pair information and action information, which increases the cost of building the data set.
[0005] Traditional dialogue systems have made rapid progress in performance on specific data sets and tasks, and their prediction performance on artificially constructed test sets has been getting better and better. However, they have ignored the challenges of solving more complex real-world conversation scenarios:
[0006] First, in real life, the process of dialogue is not limited to a single task, nor does it strictly follow the predefined task completion process, but rather integrates a complex process of chat-type dialogue or task-based dialogue. Obviously, the parameterized model learned on a single type of corpus cannot cope with this general dialogue task.
[0007] Secondly, the supervisory information in the corpus used in the existing TOD model is too complex, which not only increases the cost of data collection and construction, but also damages the algorithmic maintainability and domain transferability of the conversational robot.
[0008] The construction method of the classic TOD dataset is adapted to the process of the TOD model when handling multiple rounds of interactions. The workflow of the TOD system generally includes four modules, namely natural language understanding (NLU), dialogue state tracking (DST), dialogue policy management (POL) and natural language generation (NLG). The NLU module is responsible for understanding the user's intentions and actions, the DST module is responsible for extracting entities and attributes from the understood user semantic information to track the completion of the current task (recorded as the dialogue state), and the POL module is responsible for the system to query the knowledge base according to the results of DST and select the appropriate action; finally, the NLG module generates a system response in natural language form and feeds it back to the user.
[0009] The problem with the TOD system is that in multiple rounds of interaction, the system response strategy is highly coupled to the context. The "context" includes not only the explicit chat history or task completion progress, but also the implicit scene and audience of the conversation. The above conditions jointly determine the system's response strategy, which is more complex than the conditions in a single round.
[0010] A dialogue action is a hand-crafted structured representation of a system's response strategy, such as inform, request, recommend, greeting, etc. Since human language has infinite possibilities, defining a suitable action space is a serious challenge.
[0011] Dialogue State has three meanings: 1. Measurement of the progress of the current task; 2. Tracking of user intentions; 3. Content of database query.
[0012] Generally, the dialog state is expressed as a set of slot-value pairs to prepare for subsequent dialog management and determine the next system action.
[0013] Ontology refers to all entities known to the system, including predefined domains, slot names, and all candidate values of values. In multi-domain conversations, it is unrealistic to enumerate a complete ontology, and the ontology can be expanded and changed at any time, which greatly increases the learning burden of the dialogue model.
[0014] Currently, the Chinese TOD datasets publicly available in the academic community include CrossWOZ and RisaWOZ, while the representative Chinese chat datasets are LCCC-base and NaturalConv. In order to realize a general dialogue model with generalized intent, a single type of dataset is not enough. It is necessary to design a corpus fusion framework to align the dataset into a suitable format, improve some fields with insufficient scalability, and improve the robustness and scalability of the trained dialogue model. Summary of the invention
[0015] Based on the above considerations, the present invention proposes a method for constructing a universal dialogue corpus framework, which supports heterogeneous dialogue tasks such as small talk and task-based dialogue without the need for supervision of intent classification. It abandons the traditional data labeling method as a whole, greatly reduces the difficulty of model parameters and supervised learning, reduces the cost of model training and maintenance, and improves user participation and satisfaction.
[0016] The method for constructing the general dialogue corpus framework comprises the following specific steps:
[0017] Step 1: Obtain round samples of the open source datasets RisaWOZ, LCCC-base, and NaturalConv respectively;
[0018] In the RisaWOZ dataset, a turn consists of three parts: utterance, action, and state.
[0019] LCCC-base multi-round chat dataset, there are two versions: LCCC-base and LCCC-large.
[0020] The NaturalConv multi-round chat dataset is based on topic-driven Chinese dialogue generation with natural properties, including scene assumptions, free topic expansion, and greetings.
[0021] Step 2: For a single dialogue turn sample in the RisaWOZ dataset, the content of the turn sample is abstracted into inputs, NLU, Query, and NLG to form a complete data structure of the sample unit;
[0022] Inputs include the Last Belief State, Dialogue Context, and User Input.
[0023] NLU is responsible for identifying the two dimensions of Info Msg and Req Msg, respectively modeling the information provided and requested in the user's intent.
[0024] Query represents the matching results, including the matching entity table (DB Items) and the matching status (DB Status);
[0025] NLG includes: the Raw Response of the sentence generated by the decoder and the Final Response of the sentence translated by the Rewriter.
[0026] Step 3: Use the heuristic method to define the RisaWOZ data set data segments Query Msg, DB Status, Style and Raw Resp, and obtain the classification results of all discrete data segments in the data set as the corpus of the data set.
[0027] First, for the above data fragments, a small number of samples are labeled through rule mapping, and the neural model is actively learned and reversely trained;
[0028] Then, the trained neural model is used to predict all data segments of the RisaWOZ dataset to obtain the classification results of all discrete data segments;
[0029] The discrete classification results of the data segments, Action Types, are shown in Table 1;
[0030] Table 1
[0031]
[0032] Step 4: For the LCCC-base and NaturalConv chat datasets, align all sample data into a common data structure. The aligned sample corpus is recorded as
[0033] The specific alignment process is:
[0034] First, the user input in a single sample is treated as an open domain topic, that is, no information related to the predefined domain is provided or requested;
[0035] Then, set both the Info Msg and Query Msg dimensions of NLU to empty;
[0036] Next, set the DB Status under Query to a special identifier <notquery>, indicating that the database will not be checked;
[0037] Finally, the Raw Response under NLG is directly copied from the sentence Final Response to simulate the diversity and randomness of the system responses in open domain dialogues.
[0038] Step 5: Convert the corpus of the RisaWOZ dataset The corpus aligned with the chat dataset Perform cross-fusion to build a general dataset of Encoder-Decoder.
[0039] The specific cross-fusion process is as follows:
[0040] First, the corpus With Corpus The context part of each sample is separated from the input-output pair and then cross-grafted to each other;
[0041] Corpus The context fields of the sample are grafted onto the corpus The input and output pairs of the sample form part of the enhanced corpus.
[0042] Corpus The context fields of the sample are grafted onto the corpus The input and output pairs of the sample form part of the enhanced corpus.
[0043] The context fields include the conversation history and the status of the last conversation.
[0044] For enhanced corpus Manual strategy judgment is required Is the sample legal?
[0045] If the user has not informed any new information in the previous round, and has not changed the entities in the locked domain in the previous round, then this round is considered The contextual part of the sample is suitable for grafting "language material", synthetic The sample is legal.
[0046] The general dataset for constructing the Encoder-Decoder is and Collection of .
[0047] The advantages of the present invention are:
[0048] 1) A method for constructing a general dialogue corpus framework defines a new data structure for a general dataset and adds special control fields through active learning, making it more compatible with heterogeneous dialogue tasks;
[0049] 2) A method for constructing a universal dialogue corpus framework, in which context encoders share knowledge, learn universal semantic features, and have stronger modeling capabilities for context.
[0050] 3) A method for constructing a universal dialogue corpus framework with an isomorphic downstream decoding module realizes complex task interaction and has significant advantages in universality.
[0051] 4) A method for constructing a general dialogue corpus framework, which proposes an efficient and easy-to-use automatic framework for constructing mixed corpus, targeting the randomness of user speech in real-life conversation scenarios and improving the robustness of the trained model;
[0052] 5) A method for constructing a general dialogue corpus framework to meet complex, changeable and unpredictable dialogue needs and improve the robustness and scalability of the model.
[0053] 6) A method for constructing a universal dialogue corpus framework reduces the supervisory information required for parameter optimization, has low algorithm complexity, reduces maintenance and expansion costs, and has higher prediction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a flow chart of a method for constructing a universal dialogue corpus framework of the present invention;
[0055] Figure 2 is a schematic diagram of a turn in the RisaWOZ dataset of the present invention;
[0056] Figure 3 It is the complete data structure of a single dialogue turn sample of the present invention;
[0057] Figure 4 This is the corpus of the present invention With Corpus Specific cross-integration process. DETAILED DESCRIPTION
[0058] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0059] The existing training data for learning multi-round dialogue models only focuses on a single task, and the structure of the supervision information is quite different, making it difficult to unify and align. The manually annotated corpus complies with predefined standards, resulting in excessive programming, so the learned parameter model has very poor robustness. Therefore, the present invention proposes a framework for efficiently constructing a composite dialogue dataset for a dialogue system that copes with general tasks. By aligning and fusing multiple heterogeneous datasets, the constructed corpus has a significant effect on improving the generality, scalability, and robustness of the model.
[0060] The present invention is based on the following assumptions: small talk and task-based dialogue are mutually reinforcing. Introducing domain knowledge into small talk can make it more meaningful, and introducing tasks into small talk can make users more motivated to have conversations, which also helps to increase the probability of successful completion of tasks. In order to design a dialogue system that can handle general tasks, the data set required for the learning model also needs to be constructed in a personalized manner.
[0061] The method for constructing the general dialogue corpus framework is as follows: Figure 1 As shown, the specific steps are:
[0062] Step 1: Obtain round samples of the open source datasets RisaWOZ, LCCC-base, and NaturalConv respectively;
[0063] RisaWOZ is an open-source, task-oriented, multi-domain, multi-round dataset from Soochow University, covering 12 fields including scenic spots, restaurants, hotels, airplanes, trains, weather, movies, TV series, computers, cars, hospitals, and tutoring classes. Conversations in the computer and car fields are more like knowledge-based question-answering, while conversations in other fields are more business-oriented. The dataset contains about 11.2K multi-round dialogue scenarios and a total of more than 150K rounds of user-system interaction pairs, making it the largest medium-task dialogue dataset to date.
[0064] In the RisaWOZ dataset, a turn consists of three parts: utterance, action, and state. The schematic diagram of a turn is shown in the figure below. Figure 2 shown.
[0065] Utterance is the interaction text between the user and the system in natural language form, including user input statements and system response statements.
[0066] Action abstracts the utterance of both sides into a structured dictionary in the form of {Domain:Intent(slot1=value1,slot2=value2,…)}. Intent can be inform, request, recommend or greeting, etc. It is a hand-crafted abstract representation of different types of interaction strategies.
[0067] State represents the state of the conversation and is used to track the completion of the task. It is a container that loads the attribute information of the locked entities in all the fields involved so far. Its structure is a tree structure like {Domain: slot1 = value1, slot2 = value2...}.
[0068] LCCC-base and NaturalConv are open-domain chat corpora, which are much larger than RisaWOZ.
[0069] LCCC-base multi-round chat dataset, the LCCC (Large-scale Cleaned Chinese Conversation) dataset open sourced from the paper "A Large-Scale Chinese Short-Text Conversation Dataset" published by Tsinghua University, has two versions: LCCC-base and LCCC-large. LCCC-base contains 6.8M conversations and is currently the largest and most strictly cleaned Chinese conversation dataset open sourced.
[0070] (3) NaturalConv multi-round chat dataset
[0071] NaturalConv is a multi-round dialogue dataset from Tencent AI Lab, which provides a dataset with more topics, more natural transitions and closer to humans, based on the shortcomings of existing document-based dialogue datasets. The feature of NaturalCon is the topic-driven Chinese dialogue generation, which is closer to human-like dialogue and has natural properties, including scene assumptions, free topic expansion, greetings, etc. It contains about 400K sentences and 19.9K dialogues, covering multiple fields (including but not limited to sports, entertainment, and technology). The average number of rounds is 20, which is significantly longer than other corpora.
[0072] An example of a conversation in the "movie" field in the dataset (excerpt):
[0073] Speaker 1: I have been looking forward to this movie for a long time since the trailer.
[0074] Speaker 2: Actually, I don’t really like watching police and gangster movies, but this one is really good.
[0075] Speaker 1: I heard that Yao Rao won an award for this movie.
[0076] Speaker 2: As a newcomer, her acting skills are indeed very good.
[0077] Speaker 1: Have you seen I Am Not a God of Medicine?
[0078] Speaker 2: Yes, I saw it. How could I miss such a good movie?
[0079] Step 2: For a single dialogue turn sample in the RisaWOZ dataset, the content of the turn sample is abstracted into inputs, NLU, Query, and NLG to form a complete data structure of the sample unit;
[0080] The complete data structure of a single dialogue turn sample is as follows Figure 3 As shown in the figure, this sample is taken from a dialogue round in the RisaWOZ dataset where the domain switching occurs between attractions and hotels. At this time, the user has asked about the information of the Jinxi Ancient Town attraction in Kunshan District and wants to find a hotel. Since there are too few restrictions on hotels (only knowing to search in Kunshan District), the number of hotel entities matched after checking the database exceeds 10. Therefore, the system adopts such a strategy - not to feedback the results immediately, but to request other conditions from the user. The above dialogue content is abstracted into 4 parts, namely inputs, NLU, Query and NLG.
[0081] Inputs include the Last Belief State, Dialogue Context, and User Input.
[0082] NLU is responsible for identifying the two dimensions of Info Msg and Req Msg, respectively modeling the information provided and requested in the user's intent.
[0083] Query represents the matching results, including the matching entity table (DB Items) and the matching status (DB Status);
[0084] The matching entity table refers to the set of all specific entities that meet the conditions in the current field. It only determines the slot value information that may appear in the response and has nothing to do with the system's strategy. The matching status is the abstraction of the matching results, which directly affects what strategy the system chooses for interaction.
[0085] NLG includes: the Raw Response of the sentence generated by the decoder and the Final Response of the sentence translated by the Rewriter.
[0086] Step 3: Use the heuristic method to define the RisaWOZ data set data segments Query Msg, DB Status, Style and Raw Resp, and obtain the classification results of all discrete data segments in the data set as the corpus of the data set.
[0087] The present invention uses a heuristic method to define possible discrete types of write data fragments: Query Msg, DBStatus, Style, Raw Resp and other fields, and then uses an active learning engineering method to reversely train a neural model to predict these three fragments, and then assists with manual cleaning to obtain a complete data structure containing all the required fragments.
[0088] Specifically:
[0089] First, for the above data fragments, a small number of samples are labeled through rule mapping, and the neural model is actively learned and reversely trained;
[0090] Then, the trained neural model is used to predict all data segments of the RisaWOZ dataset to obtain the classification results of all discrete data segments;
[0091] The discrete classification results of the data segments, Action Types, are shown in Table 1;
[0092] Table 1
[0093]
[0094] Step 4: For the LCCC-base and NaturalConv chat datasets, align all sample data into a common data structure. The aligned sample corpus is recorded as
[0095] In order to realize a universal composite task dialogue system, the present invention needs to align the data of the two chat corpora, LCCC-base and Neural-Conv, into a universal structure. The specific alignment process is as follows:
[0096] First, a single sample of chatty user input is used as an open domain topic, which does not provide or request any information related to the predefined domain;
[0097] Then, set both the Info Msg and Query Msg dimensions of NLU to empty;
[0098] Next, set the DB Status under Query to a special identifier <notquery>, indicating that the database will not be checked;
[0099] Finally, the Raw Response under NLG is directly copied from the sentence Final Response to simulate the diversity and randomness of the system responses in open domain dialogues.
[0100] Step 5: Convert the corpus of the RisaWOZ dataset The corpus aligned with the chat dataset Perform cross-fusion to build a general dataset of Encoder-Decoder.
[0101] Corpus cross-pollination is the most critical step in the process of building proprietary data. In real-life conversation scenarios, user speech is unpredictable, that is, users may not necessarily input their words according to a perfect and logical script. The user's behavioral characteristics are different from the agent played by the annotator. The model trained with the original corpus will inevitably be very non-robust. Therefore, it is necessary to enhance the original corpus in some way; such as Figure 4 As shown in the figure, the specific cross-fusion process is:
[0102] First, the corpus With Corpus The context part of each sample is separated from the input-output pair and then cross-grafted to each other;
[0103] The context part includes the conversation history and the status of the last conversation. For simplicity, the output fields except the system response are omitted.
[0104] Corpus The context fields of the sample are grafted onto the corpus using a method similar to the one in the figure. The sample input and output pairs are divided into some fields, and the following are obtained: The context is conditional The input and output pairs constitute the enhanced corpus
[0105] Corpus The context fields of the sample are grafted onto the corpus The sample input and output pairs are divided into some fields, and the following The context is conditional The input and output pairs constitute the enhanced corpus
[0106] For enhanced corpus Because the correlation between rounds in TOD is stronger, manual strategy judgment is required Is the sample legal?
[0107] If the user has not informed any new information in the previous round, and has not changed the entities in the locked domain in the previous round, then this round is considered The contextual part of the sample is suitable for grafting "language material", synthetic The sample is legal.
[0108] The general dataset for constructing the Encoder-Decoder is and Collection of .
[0109] Example:
[0110] In the prior art, controllable experiments, whether in strategy or style, face the problem of data imbalance (Data Bias). Since each dialogue context has only one reference response when collecting corpus in RisaWOZ, the data distribution of legal actions depends on their frequency of occurrence in the data set, which is extremely unbalanced, resulting in insufficient coverage of model learning for less common strategies and styles. This embodiment proposes a general controllable response data augmentation (CRDA) framework to achieve the above goals.
[0111] First, convert the system action into a sequence form, recorded as action span. For example, the action span corresponding to "return_one(domain=attraction, slot=name, value=Zhouzhuang Ancient Town)" is "attraction<return_one> Name: Zhouzhuang Ancient Town.<return_one> Is a special token representing the action name.
[0112] As for strategy enhancement, since Skeleton is directly generated from artificial templates, it is possible to deterministically construct a legal and diverse action set for each sample through predefined rules while keeping the context unchanged.
[0113] Define user input as U t , action span is A t , the reply sentence is R t , train a GPT2 generative model through the training set component sequence. Then, let the obtained GPT2 generate multiple response results. Since traditional beam search tends to generate sequences with the same root, this example refers to the diversified generation method and uses decoding techniques such as top-k sampling and kernel sampling to promote the diversity of generated results. Finally, the rule-based annotation method is used to perform style annotation for n generated results. After manual screening and cleaning, a personality dataset with balanced distribution in style types is achieved:
[0114]
[0115] Based on the open source of academic chat and task-based dialogue datasets, the present invention designs a simple and efficient hybrid task corpus construction framework. By defining new data structures, corpus alignment, and context crossover, the obtained hybrid dataset can be used to train dialogue models that support heterogeneous dialogue tasks such as chat and task-based dialogue at the same time, without the need to supervise intent classification. The traditional data annotation method is abandoned as a whole, which greatly reduces the difficulty of model parameters and supervised learning, and reduces the cost of model training and maintenance. When dealing with complex task challenges, chat is used to promote conversations, share semantic knowledge modeling, break through the disadvantage that academic algorithms are difficult to implement, and expand the dialogue system to more mixed and diverse application scenarios.< / notquery> < / notquery>
Claims
1. A method for constructing a general dialogue corpus framework, characterized in that: The specific steps are as follows: First, obtain round samples of the open source datasets RisaWOZ, LCCC-base, and NaturalConv respectively; For a single dialogue turn sample in the RisaWOZ dataset, the content of the turn sample is abstracted into inputs, NLU, Query, and NLG to form a complete data structure of the sample unit; The inputs include the state of the last dialogue Last BeliefState, the dialogue context Dialogue Context and the user input User Input; NLU is responsible for identifying the two dimensions of Info Msg and Req Msg, respectively modeling the information provided and requested in the user's intent; Query represents the matching result, including the matching entity table and the matching status; The matching entity table refers to the set of all specific entities that meet the conditions in the current domain. It only determines the slot value information that may appear in the response and has nothing to do with the system strategy; The matching status is an abstraction of the matching result, which directly affects the strategy selected by the system for interaction. NLG includes: the Raw Response of the sentence generated by the decoder and the Final Response of the sentence translated by the Rewriter; Then, using the heuristic method, we define the data segments of the RisaWOZ dataset: Query Msg, DB Status, Style, and Raw Resp, and obtain the classification results of all discrete data segments in the dataset as the corpus of the dataset. For the chat datasets LCCC-base and NaturalConv, all sample data are aligned into a common data structure. The aligned sample corpus is recorded as Finally, the corpus of the RisaWOZ dataset The corpus aligned with the chat dataset Perform cross-fusion and build a common data set.
2. A method for constructing a general dialogue corpus framework as claimed in claim 1, characterized in that: In the dataset RisaWOZ, a turn consists of three parts: utterance, action, and state. Utterances include user input statements and system response statements in natural language; The action is to abstract the utterance of two sides into a structured dictionary; State represents the state of the conversation, which is used to track the completion of the task and is a container for the attribute information of the locked entities in all fields involved so far.
3. The method for constructing a universal dialogue corpus framework according to claim 1, characterized in that: The corpus of the dataset The formation process is: First, for the above data fragments, a small number of samples are labeled through rule mapping, and the neural model is actively learned and reversely trained; Then, the trained neural model is used to predict all data segments of the dataset RisaWOZ to obtain the classification results of all discrete data segments.
4. The method for constructing a general dialogue corpus framework according to claim 1, characterized in that: The sample corpus is The specific alignment process is: First, the user input in a single sample is treated as an open domain topic, that is, no information related to the predefined domain is provided or requested; Then, set both the Info Msg and Query Msg dimensions of NLU to empty; Next, set the DB Status under Query to a special identifier <notquery> , indicating that the database will not be checked;< / notquery> Finally, the Raw Response under NLG is directly copied from the sentence Final Response to simulate the diversity and randomness of the system responses in open domain dialogues.
5. The method for constructing a universal dialogue corpus framework according to claim 1, characterized in that: The corpus With Corpus Carry out cross fusion, the specific process is: First, the corpus With Corpus The context part of each sample is separated from the input-output pair and then cross-grafted to each other; Corpus The context fields of the sample are grafted onto the corpus The input and output pairs of the sample form part of the enhanced corpus. Corpus The context fields of the sample are grafted onto the corpus The input and output pairs of the sample form part of the enhanced corpus. The context fields include the conversation history and the status of the last conversation. For enhanced corpus Manual strategy judgment is required Is the sample legal? If the user has not informed any new information in the previous round, and has not changed the entities in the locked domain in the previous round, then this round is considered The contextual part of the sample is suitable for grafting "language material", synthetic The sample is legal; The general dataset for constructing the Encoder-Decoder is and Collection of .
