A task-based dialogue system and method based on seq2seq architecture

Abstract

The present invention provides a task-based dialogue system and method based on seq2seq architecture, including four parts, namely, intention and slot filling network module, database operation module, trust tracking module and optimization and generation network module; including the following steps: the bert output of the classification task and the vector of the input label are mapped to the same vector space, and the slot value extraction task is trained through the CRF layer; the database operation module mainly performs other slot value queries based on the identified slot value, the trust tracking layer, takes the output of the intention and slot value network module and the database operation module as input, takes the output of this layer as input, takes the input of the intention and generation network, and the optimization and generation network module selects the highest score as the answer An of this round of dialogue, and predicts the answer later. The present invention combines the task-based dialogue system with the chat-type dialogue system, and internally combines the modules of the task-based dialogue system, so as to ensure the role of context and enable overall optimization.

Description

Technical Field

[0001] The present invention relates to a task-based dialogue system and method based on a seq2seq architecture, belonging to the technical field of intelligent question answering. Background Art

[0002] Traditional task-based dialogue systems use a pipeline structure. Because this structure is modular, errors in upstream task parameters will be transmitted to downstream. Chatty dialogue systems use a seq2seq structure to generate corresponding answers through the Encoder and Decoder modules. However, chatty dialogue systems do not consider the role of context, and the generated NLG answers are uncontrollable. Summary of the invention

[0003] The purpose of the present invention is to provide a task-based dialogue system and method based on a seq2seq architecture, which not only takes into account the impact of context and avoids generating uncertain answers, but also performs overall optimization.

[0004] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0005] A task-based dialogue system based on seq2seq architecture, including four parts: intent and slot filling network module, database operation module, trust tracking module, and optimization and generation network module;

[0006] The intent and slot value network module is built using the Bert+CRF structure. Bert is used for intent classification and CRF is used for slot value extraction. The module is pre-trained for three tasks, namely, intent classification task, slot value extraction task and Bert's own mask word prediction task.

[0007] The database operation module performs other slot value queries based on the identified slot value;

[0008] The Belief Tracker module uses the Roberta model and outputs a vector containing all information from the beginning to the current round. ;

[0009] The optimization and generation network module uses a classification algorithm to select the action with the highest score among the existing actions as the answer to this round of dialogue, and uses the answer as the input of the trust tracking module for subsequent predictions.

[0010] A task-based dialogue method based on a seq2seq architecture, comprising the following steps:

[0011] 1) Bert output [CLS]Token for classification task and a vector of input labels Mapped to the same vector space, by maximizing the similarity between the predicted value and the positive label And minimize the similarity between the predicted value and the negative label For optimization, the specific formula is as follows:

[0012]

[0013] 2) The slot value extraction task is trained through the CRF layer, and the formula is as follows:

[0014]

[0015] in is the hidden vector of the entity input label, a is the corresponding vector Token of the data input, is the negative similarity of CRF;

[0016] 3) The overall loss of the intent and slot filling layer is calculated as:

[0017]

[0018] In the formula Mask prediction for Bert.

[0019] 4) The database operation module searches for other slot values ​​based on the identified slot value, and searches for other slot values ​​through a select query statement with conditions based on the keywords proposed by the system;

[0020] 5) Trust tracking layer, which takes the output of the intent and slot value network module and the database operation module as input One of the output Sn of this layer is used as one of the inputs, and the input An of the intention and generation network is used as one of the inputs of Belief Tracker.

[0021]

[0022] in Represents the user intention and corresponding entity in this round, Sn represents all states from the beginning to the last round, and An represents the answer to the user's question in the last round;

[0023] 6) The optimization and generation network module uses a classification algorithm to obtain the final result. Among the existing actions, the one with the highest score is selected as the answer An for this round of dialogue, and the answer is used as the input of Belief Tracker for subsequent prediction.

[0024] The advantages of the present invention are that the present invention combines the task-based dialogue system with the chat-based dialogue system, adopts a seq2seq structure as a whole, and combines the modules of the task-based dialogue system internally, thereby ensuring the role of context and being able to perform overall optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0026] Figure 1 It is a schematic diagram of the overall structure of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The entry point of this model is the intent and slot value network. The network is built using the Bert+CRF structure. Bert is used for intent classification, and CRF is used for slot value extraction. When doing pre-training in this part, three tasks are considered: intent classification task, slot value extraction task, and Bert's own mask word prediction task.

[0029] Bert output [CLS]Token for classification task and a vector of input labels Mapped to the same vector space, by maximizing the similarity between the predicted value and the positive label And minimize the similarity between the predicted value and the negative label For optimization, the specific formula is as follows:

[0030]

[0031] The slot value extraction task is trained through the CRF layer, and the formula is as follows:

[0032]

[0033] in is the hidden vector of the entity input label, a is the corresponding vector Token of the data input, is the negative similarity of CRF;

[0034] The overall loss for calculating the intent and slot filling layer is:

[0035]

[0036] In the formula Mask prediction for Bert.

[0037] The database operation module performs other slot value queries based on the identified slot value for answer generation. For example:

[0038] S: Give me a drink.

[0039] System: What kind of drink would you like?

[0040] S: What do you have?

[0041] System: We have coffee, milk tea and juice.

[0042] In the above dialogue, after the system extracts the keyword "beverage", it can be searched through a select query statement with conditions.

[0043] Trust Tracking Layer,The trust tracking layer takes the output of the intent and slot value network module and the database operation module as input One of the output Sn of this layer is used as one of the inputs, and one of the inputs An of the intention and generation network is used as the final input of BeliefTracker

[0044]

[0045] in represents the user intention and corresponding entity in this round, Sn represents all states from the beginning to the last round, and An represents the answer to the user's question in the last round; the network considers both the historical information and the impact of the current round information. The network adopts the Roberta model. The output is a vector containing all the information from the beginning to this round .

[0046] The optimization and generation network module uses a classification algorithm. Since the model is used in a task-based dialogue system, the accuracy of the output result is the most important. The final result uses a classification algorithm. Among the existing actions, the one with the highest score is selected as the answer An for this round of dialogue, and the answer is used as the input of Belief Tracker for subsequent prediction.

Claims

1. A task-based dialogue method based on seq2seq architecture, characterized in that: A task-based dialogue system is included, wherein the system includes four parts, namely, an intent and slot filling network module, a database operation module, a trust tracking module, and an optimization and generation network module; The intent and slot value network module is built using the Bert+CRF structure. Bert is used for intent classification and CRF is used for slot value extraction. The module is pre-trained for three tasks, namely, intent classification task, slot value extraction task and Bert's own mask word prediction task. The database operation module performs other slot value queries based on the identified slot value; The trust tracking module uses the Roberta model and outputs a vector containing all information from the beginning to the current round. ; The optimization and generation network module uses a classification algorithm to select the action with the highest score from the existing actions as the answer to this round of dialogue, and uses the answer as the input of the trust tracking module for subsequent predictions; The method comprises the following steps: 1) Bert output [CLS]Token for classification tasks, , and the vector of input labels Mapped to the same vector space, by maximizing the similarity between the predicted value and the positive label And minimize the similarity between the predicted value and the negative label For optimization, the specific formula is as follows: ， in, represents a vector space; 2) The slot value extraction task is trained through the CRF layer, and the formula is as follows: in is the hidden vector of the entity input label, a is the corresponding vector Token of the data input, is the negative similarity of CRF; 3) The overall loss of the intent and slot value filling layer is calculated as: In the formula This is the Mask prediction that comes with Bert. represents the loss function; 4) The database operation module searches for other slot values ​​based on the identified slot value, and searches for other slot values ​​through a select query statement with conditions based on the keywords proposed by the system; 5) Belief Tracking Layer Module, which takes the output of the intent and slot value network module and the database operation module as one of its inputs, the output Sn of this layer as one of its inputs, and the input An of the intent and generation network as one of its inputs. The BeliefTracker finally inputs in Represents the output of the intent and slot value network module and the database operation module, Sn represents all states from the beginning to the last round, and An represents the answer to the user's question in the last round; 6) The optimization and generation network module uses a classification algorithm to select the action with the highest score from the existing actions as the answer An for this round of dialogue, and uses the answer as the input of Belief Tracker for subsequent predictions.

Images