Conversational content-driven question generation for dialogue systems

Abstract

The present invention discloses a conversation content-driven question generation method applicable to a dialogue system, step 1: annotate and extract keywords of the answer part as named entities from a data set containing "question + answer"; step 2: generate a sequence of sentences with keywords from the named entities using a constrained language model CLM; step 3: use an end-to-end model framework to generate questions, the end-to-end framework model includes an encoder module, a decoder module and an intermediate state vector connecting the two, the encoder module learns the input named entity, encodes it into a state vector of a fixed size, and then passes the vector to the decoder module, and the decoder module outputs a question sentence by learning the state vector; step 4: use a gated self-attention mechanism to model the dialogue flow. Compared with the prior art, the present invention has a simple process and is more efficient in automatically generating expected and meaningful questions.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a question generation method suitable for a dialogue system. Background Art

[0002] Dialogue is the main means for people to seek and exchange information, and asking useful questions is an important prerequisite for the effective exchange of knowledge. Question generation (QG) task refers to automatically generating questions given a variety of inputs (such as raw text, database, semantic representation). Although not as prominent as question answering (QA) task, it is still an important task in the field of natural language processing. Question generation can be effectively applied to many fields, including answering questions, dialogue systems, and education.

[0003] Traditional question generation methods rely on heuristic rules, that is, after grammatical and lexical analysis, sentences are transformed syntactically to generate corresponding questions. However, these methods require particularly well-designed transformation and generation rules, resulting in low generalization. With the continuous development of deep learning methods, more and more deep learning methods such as recurrent neural networks (RNN), long short-term memory networks (LSTM) and gated recurrent units (GRU) are used to solve technical problems of dialogue systems.

[0004] The task of question generation for conversations is more challenging than that for single-discourse questions, as questions themselves are often highly ambiguous in conversations, requiring the model to have a deeper understanding of the context surrounding the passage text and conversation history in order to ask meaningful questions. Most studies on question generation tasks only use passages and conversation histories as input (i.e., without knowing the answer) to generate questions, while answer-aware conversational question generation models generate questions based on the target answer, conversation history, and passage. Although answer-aware conversational question generation models seek to improve the generated conversation flow, current answer-aware question generation models have some problems, including inaccurate co-citation alignment, inconsistent conversations, incorrect grammar, and the inability to generate many different types of questions. The relevant prior art mainly focuses on the task of conversational question generation without knowing the answer.

[0005] In our daily lives, people gradually receive and transmit information through progressive question-and-answer sessions to complete the final information collection and understanding, but traditional machine reading comprehension does not have this function. Based on this functional requirement, a dataset CoQA was created to measure the ability of machines to conduct conversational question-and-answer sessions. In daily conversations, it is found that the main purpose of asking questions is to mention something or something through answers. In other words, in daily conversations, people do not pay attention to using the correct grammatical format to express their opinions, but rather simply express the parts they think are important. In the process of asking questions, not all the information of the entire answer is needed. Existing technologies all complete the task of generating conversational questions based on the entire answer and other auxiliary features. Summary of the invention

[0006] The present invention aims to propose a conversation content-driven question generation method suitable for a dialogue system, which achieves the task of automatically generating expected and meaningful questions by guiding the scenario of subsequent dialogue content.

[0007] The present invention provides a method for generating questions driven by conversation content applicable to a dialogue system, the method comprising the following steps:

[0008] Step 1: Label and extract the keywords of the answer part from the dataset containing "question + answer" as named entities, and use the named entities as input;

[0009] Step 2: Use the Constrained Language Model (CLM) to generate a sequence of sentences with keywords from the named entities as input, which is used as the input for step 3. The details are as follows:

[0010] The CLM model is used to generate the words before and after the current input word at the same time. On the other hand, in a sentence w=w1,…,w l , the input word w s Divide the sentence into two parts, including: the preceding sequence of the first s words is w s , w s-1 , w s-2 ,…,w1; the sequence of the last l-s+1 words is w s , w s+1 , w s+2 ,…,w l , that is, at each time step, the concatenation of the word embeddings is taken as input, the generated sequence is the word generated in the last time step, the embedding information is provided to a shared hidden layer, and then the two output layers predict the words in the forward and reverse sequences respectively, thus generating a network containing the word w s Sentences;

[0011] Step 3: Use the end-to-end model framework to generate questions. The end-to-end framework model includes an encoder module, a decoder module, and an intermediate state vector connecting the two. The encoder module learns the input name entity and encodes it into a state vector of a fixed size, and then passes the vector to the decoder module. The decoder module then outputs a question sentence by learning this state vector;

[0012] Step 3.1: Use the encoder module to jointly encode the information from the two sources through two parts: paragraph encoding and dialogue encoding:

[0013] A bidirectional LSTM network model is used for paragraph encoding, with word embedding representing w i and the answer position embedding representation a i The concatenation of i =[w i ; a i ], using the typical BIO tagging scheme to represent the answer range, and mapping each token in the paragraph to the corresponding answer position embedding representation B_ANS, I_ANS, O, B_ANS represents the beginning of the answer, I_ANS represents the middle part of the answer, and O represents the non-answer part; the entire paragraph is represented by the hidden state of the bidirectional LSTM encoder, that is, Among them, m represents the sequence length, and p represents the chapter;

[0014] Encode the conversation and convert the conversation history C i-1 Modeled as a "question-answer" sequence C i-1 ={(Q1, A1,), ..., (Q i-1 , A i-1 ,)}use <q> Label the questions and answers and connect them to form a token sequence ( <q>,q1,…q m ;< / q> ,a1,…a m ),in, <q>Indicates question tag;< / q> Indicates the answer sentence label;

[0015] Use a token-level bidirectional LSTM to obtain the contextual representation of the question-answer pair:

[0016]

[0017] Among them, i represents the round number, The latent vector representation of the question-answer pair obtained by the bidirectional LSTM at the token level, m and q1, ...q m The m in it indicates corresponding;

[0018] Context-level LSTM is used in different question-answering rounds to learn context dependencies:

[0019]

[0020] Generate a question Q for the next round of dialogue i , the expression is as follows:

[0021]

[0022] Among them, P is a given chapter, C i-1 ={(Q1, A1,), ..., (Q i-1 , A i-1 ,)} is the conversation history, A i is the answer to the current conversation;

[0023] Step 3.2: Decode using the LSTM network model with attention mechanism, and generate a new sequence as the generated question:

[0024] At each decoding time step t, the word embedding w is read t and the hidden state of the previous decoding time step t-1 To generate the current hidden state:

[0025]

[0026] At each decoding time step, the chapter attention (α1, ..., α m ) and dialogue attention (β 1，1 , …, β 1，m ;…;β i-1，1 , …, β i-1，m ):

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] Among them, W p , W ω , W c represents the learnable weights, Respectively represent the weights of the corresponding hidden states; α j , β i-k,j represents the normalized representation of the corresponding hidden state; e total represents the sum of weights;

[0033] Finally, we get the context vector and the final vocabulary distribution:

[0034]

[0035]

[0036] Among them, W v , W a represents the learnable weight, c t represents the context vector representation, P v represents the final vocabulary distribution, b v represents learnable parameters;

[0037] Step 4: Use the gated self-attention mechanism to model the dialogue flow:

[0038] A gated self-attention mechanism is applied to the answer position embedding to aggregate answer-related information from the entire paragraph to generate questions, and learn the latent alignment between the turn number embedding representation and the chunk embedding representation:

[0039]

[0040]

[0041]

[0042]

[0043] Among them, W s , W f , W g represents the learnable weight, ⊙ represents element-wise multiplication, and replace To calculate the chapter attention, Enhanced chapter representation With oneself Matching results, represents the result of the self-matching representation, express With the original representation The combined results, represents the vector representation obtained by gate summation, Enhanced representation for self-matching;

[0044] Get the dialogue flow loss function for:

[0045]

[0046] Among them, λ1, λ2 represent hyperparameters, ω j ∈CES / HES;

[0047] The total loss function for:

[0048]

[0049] in, represents the negative log-likelihood loss function in sequence generation, which is expressed as follows:

[0050]

[0051] Among them, Q i Indicates the currently generated question; P indicates the chapter; A i Indicates the current answer; C i-1 Represents the conversation history.

[0052] Compared with the prior art, the present invention is different from the traditional conversational question generation (CQG) task. It only takes a part of the answer sentence as input instead of the entire answer sentence to generate corresponding questions. Therefore, the process is simpler and the efficiency of automatically generating expected and meaningful questions is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is an overall flow chart of the conversation content-driven question generation method applicable to a dialogue system of the present invention;

[0054] Figure 2 A flow chart of an embodiment of a method for generating questions driven by conversation content applicable to a dialogue system of the present invention;

[0055] Figure 3 This is a model framework diagram of an embodiment of a method for generating conversation content-driven questions applicable to a dialogue system according to the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0057] like Figure 1 As shown in FIG. 1 , it is an overall flow chart of the conversation content driven question generation method applicable to the dialogue system of the present invention. Figure 2 As shown in FIG. 1 , it is a flow chart of an embodiment of a method for generating questions driven by conversation content applicable to a dialogue system of the present invention. Figure 3 As shown, it is a model framework diagram of an embodiment of the method for generating questions driven by conversation content applicable to a dialogue system of the present invention. In combination with the embodiment, the process specifically includes the following steps:

[0058] Step 1: Label and extract the keywords of the answer part from the dataset containing "question + answer" as named entities, and use the named entities as input for data processing:

[0059] The data comes from CoQA (Conversational Question Answering), a large dataset proposed by Stanford University for building a conversational question-answering system. The CoQA dataset is a dataset used to measure the ability of machines to conduct conversational question-answering. It contains 127k "question + answer" pairs from 8k conversations, similar to people's daily conversations; each data sample extracted from the dataset contains 4 parts, namely the chapter (conversation background), question, answer, and historical conversation (the first n rounds of question-answer pairs) (this step processes the answer sentence part in the dataset), and then filters out the "question + answer" pairs with the answer "unknown" (because the amount of information is too little to generate valid questions). Then the dataset is randomly divided into 80% training set, 10% validation set, and 10% test set.

[0060] Specific definition of extracted answer span: Abstract answers in CoQA are mostly small modifications that occur in the context. The maximum performance achievable by the model predicting span from the context is 97.8 in F1 value. Therefore, the present invention finds the extracted answer span from the paragraph with the highest answer F1 score and uses it as the answer embedded in the answer position.

[0061] Specific definition of the selection of the number of historical dialogue rounds: Most of the questions in previous related work dialogues have limited dependencies within the range of two rounds. Therefore, the present invention selects the number of historical rounds to be n=3 to ensure that the target question has sufficient conversation history information to generate and avoid introducing too much noise from the question-answer pairs of each round.

[0062] Step 2: Use the Constrained Language Model (CLM) to generate a sequence of sentences with keywords from the input. This sequence can briefly summarize the entire answer part and complete the incomplete information. As the input for the subsequent task (question generation task), it is necessary to ensure that the words in the input appear in the generated sequence. The details are as follows:

[0063] The CLM model consists of a shared hidden layer and two output layers. The input words are sent to the hidden layer, and then the two output layers predict the words in the forward and backward sequences respectively.

[0064] The CLM model is used to generate the words before and after the current input word at the same time. On the other hand, in a sentence w=w1,…,w l , the input word w s Divide the sentence into two parts, including: the preceding sequence of the first s words is w s , w s-1 , w s-2 ,…,w1; the sequence of the last l-s+1 words is w s , w s+1 , w s+2 ,…,w l , that is, at each time step, the concatenation of the word embeddings is taken as input, the generated sequence is the word generated in the last time step, the embedding information is provided to a shared hidden layer, and then the two output layers predict the words in the forward and reverse sequences respectively, thus generating a network containing the word w s Sentences;

[0065] The CLM model consists of a shared hidden layer and two output layers. The input word is sent to the hidden layer, and then the two output layers predict the words in the forward and backward sequences respectively; the concatenation of the word embeddings represents the input word sequence; the generation part represents the generated word with w s sequence;

[0066] Step 3: Use the end-to-end model framework to generate questions. The specific process is as follows: The end-to-end framework model mainly consists of three parts: the encoder module, the decoder module, and the intermediate state vector connecting the two. The encoder module learns the input name entity and encodes it into a state vector of a fixed size. This vector can be regarded as the semantics of the input name entity. The vector is then passed to the decoder module. The decoder module then outputs a question sentence by learning this state vector.

[0067] Step 3.1: Encode the above generated sequence using the encoder module of the end-to-end model framework. Since the conversational question depends on a part of the text and the conversation history, the text encoder and the conversation encoder are used to jointly encode the information from the two sources:

[0068] The text encoder is a bidirectional LSTM (Long Short-Term Memory Network) that represents word embeddings w i and the answer position embedding representation a i The concatenation of i =[w i ; a i ], using the typical BIO tagging scheme to represent the answer range, and mapping each token in the paragraph to the corresponding answer position embedding representation (i.e., B_ANS, I_ANS, O), where B_ANS represents the beginning of the answer, I_ANS represents the middle part of the answer, and O represents the non-answer part; the entire paragraph is represented by the hidden state of the bidirectional LSTM encoder, i.e., Where m is the sequence length and p represents the chapter, which is the context of the conversation.

[0069] The dialogue encoder is a hierarchical structure with two layers: a token-level bidirectional LSTM and a context-level bidirectional LSTM, which are used to model the dialogue history. i-1 Modeled as a "question-answer" sequence C i-1 ={(Q1, A1,), ..., (Q i-1 , A i-1 ,)}use <q> Label the questions and answers and connect them into a token sequence ( <q>,q1,…q m ;< / q> ,a1,…a m ),in, <q> Indicates question tag;< / q> Indicates the answer sentence label;

[0070] Use a token-level bidirectional LSTM to obtain the contextual representation of the question-answer pair:

[0071]

[0072] Where i represents the round number, and k∈[1,i), The latent vector representation of the question-answer pair obtained by the bidirectional LSTM at the token level, m and q1, ...q m The m in it indicates corresponding;

[0073] Since cross-turn dependencies appear in the conversation, context-level LSTM is used in different question-answer rounds to learn context dependencies:

[0074]

[0075] Regarding the problem definition:

[0076] Given a chapter P (conversation background), a conversation history C i-1 ={(Q1, A1,), ..., (Q i-1 , A i-1 ,)} and the answer A of the current dialogue i , generate a question Q for the next round of dialogue i , the expression is as follows:

[0077]

[0078] Step 3.2: Decode using the decoder module of the end-to-end model framework. The decoder module is an LSTM with an attention mechanism that predicts the probability distribution of words. The generated new sequence will be used as the generated question: The decoder in this step is an LSTM with an attention mechanism;

[0079] At each decoding time step t, the word embedding w is read t and the hidden state of the previous decoding time step t-1 To generate the current hidden state:

[0080]

[0081] To effectively generate a conversation question based on the context of the text, the decoder module should decide at each decoding time step how much to pay attention to the hidden state of the text. and the hidden state of the conversation history De-emphasize token-level hidden state of conversations Use the standard attention mechanism to calculate the chapter attention (α1, ..., α m ) and dialogue attention β 1，1 , …, β 1，m ;…;β i-1，1 , …, β i-1，m ):

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] Among them, W p , W ω , W c represents the learnable weights, Respectively represent the weights of the corresponding hidden states; α j , β i-k,j represents the normalized representation of the corresponding hidden state, e total represents the sum of weights;

[0088] Finally, we get the context vector and the final vocabulary distribution:

[0089]

[0090]

[0091] Among them, W v , W a represents the learnable weight, c t represents the context vector representation, P v represents the final vocabulary distribution, b v represents learnable parameters;

[0092] Step 4: Model the dialogue flow and add a gated self-attention mechanism for two purposes: first, perform self-attention on the answer position embedding to aggregate answer-related information from the entire paragraph to generate questions; second, learn the potential alignment between the turn number embedding representation and the chunk embedding representation:

[0093]

[0094]

[0095]

[0096]

[0097] Among them, W s , W f , W g represents the learnable weight, ⊙ represents element-wise multiplication, and replace To calculate the chapter attention, Enhanced chapter representation With oneself Matching results, represents the result of the self-matching representation, express With the original representation The combined results, represents the vector representation obtained by gate summation, Enhanced representation for self-matching;

[0098] Sentences that are helpful for the current question to be generated are called current evidence sentences (CES); sentences that are helpful for questions in the conversation history and irrelevant to the current question are called historical evidence sentences (HES). The dialogue flow model can pay more attention to CES and ignore HES through paragraph attention, and obtain the dialogue flow loss function for:

[0099]

[0100] Among them, λ1 and λ2 represent hyperparameters, ω j ∈CES / HES;

[0101] The total loss function for:

[0102]

[0103] in, represents the negative log-likelihood loss function in sequence generation, which is expressed as follows:

[0104]

[0105] Among them, Q i represents the currently generated question, P represents the chapter, A i Indicates the current answer, C i-1 Represents the conversation history.

[0106] The present invention mainly uses BLEU and ROUGE-L scores to evaluate the n-gram similarity between the generated questions and the ground truth.

[0107] The present invention conducts dialogue question generation task experiments on the CoQA dataset, and the experimental results are compared with multiple dialogue question generation models, namely: Seq2Seq Model, PGNet and NQG. Seq2Seq Model is a basic encoder-decoder sequence model that has been widely used in dialogue generation. The answer basis and dialogue history are connected as input sequences. PGNet is a pointer generator network that connects the passage, dialogue history and current answer as input sequences. NQG is a powerful neural network method for attention-based question generation tasks, and the input is the same as the Seq2Seq Model above.

[0108] Based on the model of the present invention, the present invention designs an ablation experiment to prove the effectiveness of the proposed dialogue question generation task model. The comparison settings of the ablation experiment are as follows:

[0109] -noCLM: Generate sentences without using CLM;

[0110] -allANS(noCLM): Take the complete answer sentence as input.

[0111] As shown in Table 1, the BLEU and ROUGE-L scores of the proposed method and three baseline models are compared on the CoQA dataset, as well as the results of ablation experiments.

[0112] Table 1

[0113] BLEU ROUGE-L Seq2Seq Model 7.38 25.84 PGNet 8.13 28.87 NQ 13.26 31.15 -noCLM 14.91 32.56 -allANS(noCLM) 16.20 34.05 The present invention* 15.87* 33.73*

[0114] By comparing the experimental results in Table 1, we find that the results of the present invention are improved compared with the baseline model. In addition, when there is no CLM model (noCLM) to complete the sentence, the experimental results are somewhat lower than those of the complete model of the present invention. This is because when the answer is not completed, less information can be obtained, so the quality of the generated questions will be reduced; in the case of allANS (noCLM), the results are the best, but compared with the results of the present invention, the difference between the two results is very small.

[0115] From the results and analysis of the comparative experiment and the ablation experiment, we can draw the beneficial effects of the present invention: Due to the work of the present invention, compared with the traditional dialogue question generation task, the input of the data intercepts the named entities in the answer sentence, and there is a certain deviation from the original answer sentence after the CLM generated sentence, so the result of the generated question is somewhat lower than the result of the complete answer input, but the difference between the two results is not big. Based on people's daily chatting habits, in daily conversations, when answering questions, people often only focus on expressing some important named entities, and will not answer questions raised by others in a rigorous grammatical format. According to the results and analysis, the present invention is sufficient to complete the key information expressed by people in daily conversations, and raise reasonable and effective questions to guide the entire conversation.

[0116] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. < / q> < / q>

Claims

1. A conversation content-driven question generation method suitable for a dialogue system, characterized in that: The method comprises the following steps: Step 1: Label and extract the keywords of the answer part from the dataset containing questions and answers as named entities, and use the named entities as input; Step 2: Use the Constrained Language Model (CLM) to generate a sequence of sentences with keywords from the named entities as input, which is used as the input for step 3. The details are as follows: The CLM model is used to generate the words before and after the current input word at the same time. On the other hand, in a sentence w=w1,…,w l , the input word w s Divide the sentence into two parts, including: the preceding sequence of the first s words is w s , w s-1 , w s-2 ,…,w1; the sequence of the last l-s+1 words is w s , w s+1 , w s+2 ,…,w l , that is, at each time step, the concatenation of the word embeddings is taken as input, the generated sequence is the word generated in the last time step, the embedding information is provided to a shared hidden layer, and then the two output layers predict the words in the forward and reverse sequences respectively, thus generating a network containing the word w s Sentences; Step 3: Use the end-to-end model framework to generate questions. The end-to-end framework model includes an encoder module, a decoder module, and an intermediate state vector connecting the two. The encoder module learns the input name entity and encodes it into a state vector of a fixed size, and then passes the vector to the decoder module. The decoder module then outputs a question sentence by learning this state vector; Step 3.1: Use the encoder module to jointly encode the information from the two sources through two parts: paragraph encoding and dialogue encoding: A bidirectional LSTM network model is used for paragraph encoding, with word embedding representing w i and the answer position embedding representation a i The concatenation of i =[w i ; a i ], using the typical BIO tagging scheme to indicate the answer range, and mapping each token in the paragraph to the corresponding answer position embedding representation B_ANS, I_ANS, O, B_ANS indicates the beginning of the answer, I_ANS indicates the middle part of the answer, and O indicates that it is not the answer part; the entire paragraph is represented by the hidden state of the bidirectional LSTM encoder Among them, m represents the sequence length, and p represents the chapter; Encode the conversation and convert the conversation history C i-1 Modeled as a "question-answer" sequence C i-1 ={(Q1,A1,),…,(Q i-1 ,A i-1 ,)}use <q> Label the questions and answers and connect them to form a token sequence ( <q>,q1,…q m ;< / q> ,a1,…a m ),in, <q> Indicates question tag;< / q> Indicates the answer sentence label; < / q> Use a token-level bidirectional LSTM to obtain the contextual representation of the question-answer pair: Among them, i represents the round number, The latent vector representation of the question-answer pair obtained by the bidirectional LSTM at the token level; Context-level LSTM is used in different question-answering rounds to learn context dependencies: Generate a question Q for the next round of dialogue i , the expression is as follows: Among them, P is a given chapter, C i-1 ={(Q1,A1,),…,(Q i-1 ,A i-1 ,)} is the conversation history, A i is the answer to the current conversation; Step 3.2: Decode using the LSTM network model with attention mechanism, and generate a new sequence as the generated question: At each decoding time step t, the word embedding w is read t and the hidden state at the previous decoding time step t-1 To generate the current hidden state: At each decoding time step, the standard attention mechanism is used to calculate the chapter attention (α1,…,α m ) and dialogue attention (β 1,1 ,…,β 1,m ;…;β i-1,1 ,…,β i-1,m ): Among them, W p , W ω , W c represents the learnable weights, Respectively represent the weights of the corresponding hidden states; α j , β i-k,j represents the normalized representation of the corresponding hidden state, e total represents the sum of weights; Finally, we get the context vector and the final vocabulary distribution: Among them, W v , W a represents the learnable weight, c t represents the context vector representation, P v represents the final vocabulary distribution, b v represents learnable parameters; Step 4: Use the gated self-attention mechanism to model the dialogue flow: A gated self-attention mechanism is applied to the answer position embedding to aggregate answer-related information from the entire paragraph to generate questions, and learn the latent alignment between the turn number embedding representation and the chunk embedding representation: Among them, W s , W f , W g represents the learnable weight, ⊙ represents element-wise multiplication, and replace To calculate the chapter attention, Enhanced chapter representation With oneself Matching results, represents the result of the self-matching representation, express With the original representation The combined results, represents the vector representation obtained by gate summation, Enhanced representation for self-matching; Get the dialogue flow loss function for: Among them, λ1, λ2 represent hyperparameters, ω j ∈CES / HES, CES represents the current evidence sentence, HES represents the historical evidence sentence; The total loss function for: in, represents the negative log-likelihood loss function in sequence generation, which is expressed as follows: Among them, Q i represents the currently generated question, P represents the chapter, A i Indicates the current answer, C i-1 Represents the conversation history.

Images