Task type question answering system based on pre-training autoregression language model

Abstract

The dialogue system can be divided into two types according to a modeling mode: a method based on an assembly line and a method based on end-to-end. However, in a method based on an assembly line, each module can only be optimized respectively, an error superposition problem exists, and system reply is directly generated by utilizing historical dialogues based on an end-to-end method. However, the dialogue system based on the end-to-end method does not comprehensively consider the information in the knowledge base, so that the effect of the task type dialogue system is limited. According to the task type question answering system based on the pre-training autoregression language model, all modules in a traditional conversation system based on an assembly line are integrated into the single autoregression language model, and results of all the modules are sequentially generated in an analysis mode of the autoregression language model till a final system reply is generated.

Description

Technical Field

[0001] The invention belongs to the field of natural language processing. Background Art

[0002] In the dialogue system, there are two types according to the modeling method: pipeline-based method and end-to-end method. However, in the pipeline-based method, each module can only be optimized separately, there is an error superposition problem, and the optimization scheme of each module cannot guarantee that the overall performance can be optimized. The end-to-end method uses historical dialogues to directly generate system responses, thereby avoiding the drawbacks of the pipeline-based method. On the other hand, since the dialogue system based on the end-to-end method usually only trains input and output discourse, without fully considering the information in the knowledge base. In the existing GPT-based task-based question-answering system, the input of the GPT-2 model needs to contain belief state information, so it can only generate responses for labeled data. The above problems limit the effectiveness of the task-based dialogue system. Summary of the invention

[0003] The present invention proposes a task-based question-answering system based on a pre-trained autoregressive language model, which integrates the various modules in the traditional pipeline-based dialogue system into a single autoregressive language model, and uses the parsing method of the autoregressive language model to generate the results of each module in turn until the final system response is generated. The steps of the AGE-TOD algorithm proposed in the present invention are as follows:

[0004] (1) Predict the involved areas and corresponding dialogue states based on the dialogue history.

[0005] (2) Predict system behavior in terms of symbolic tokens, conditioned on the conversation history and conversation state.

[0006] (3) If the system behavior requires obtaining external information from the database, the query module will retrieve the candidate information and return one of the results.

[0007] (4) When it is detected that the query result is empty, the system behavior part is updated;

[0008] (5) Based on the dialogue history, dialogue state, and system behavior, generate system responses represented by symbolic characters.

[0009] (6) Use the query results to update the symbolic tags in the system response.

[0010] Description of the Figures and Schedules

[0011] Figure 1 This is the overall architecture diagram of the algorithm of the present invention.

[0012] Figure 2It is the overall flow chart of the present invention.

[0013] Figure 3 It is the input representation of the present invention. DETAILED DESCRIPTION

[0014] According to the modeling method, the dialogue system can be divided into two types: pipeline-based method and end-to-end method. However, in the pipeline-based method, each module can only be optimized separately, which has the problem of error superposition, and the optimization scheme of each module cannot guarantee that the overall performance can be optimized. On the other hand, the dialogue system based on the end-to-end method usually only trains the input and output discourse without fully considering the information in the knowledge base. In the existing GPT-based task-based question-answering system, the input of the GPT-2 model needs to contain belief state information, so it can only generate responses for labeled data. The above problems limit the effect of the task-based dialogue system. Based on the above problems, this paper proposes AGE-TOD to use the GPT-2 model to complete the work of each module in the pipeline-based model at one time, including: dialogue state tracking (DST), prediction of system actions (POL), data query, and generation of system responses (NLG). The model will not only generate system responses, but also generate intermediate outputs such as dialogue states and system actions, which can improve the solvability of the dialogue system. The overall architecture diagram is shown in the figure below. Figure 1 This model integrates the various modules in the traditional pipeline-based dialogue system into a single autoregressive language model, and uses the parsing method of the autoregressive language model to generate the results of each module in turn until the final system response is generated.

[0015] The following will describe the idea of ​​the algorithm and give the specific steps of the algorithm. The algorithm flow chart is as follows: Figure 2 As shown, the specific steps are as follows:

[0016] Step 1: Construct input representation

[0017] First, the dialogue state and system behavior are converted into the word tokens required by GPT-2. The input representation used by AGE-TOD is as follows Figure 3 As shown, including word embedding, speaker embedding and position embedding, this paper introduces the separator <usr> 、 <sys> 、 <ds>and <sa>, respectively, indicate the start of the sequence representation of user utterance, system response, dialog state, and system operation. The involved fields and slots are also indicated by additional special markers, <nm>means "not mentioned", <dc>It means "it doesn't matter".

[0018] Step 2: Generate symbolic markup

[0019] Since each conversation in the dataset is generated based on the database query results, the slot values ​​to be filled, such as special information such as serial number and address, are only valid for specific conversation instances. In addition, in order to provide appropriate information according to the conversation context, the AGE-TOD model symbolizes the values ​​of all slots to be filled (serial number, name, zip code, phone number, address, etc.) and changes them to [DOMAIN SLOTNAME]. Then, the AGE-TOD model learns to generate symbolic system responses, and finally uses a post-processing code to replace the string with information from the database.

[0020] Step 3: Fine-tuning GPT-2

[0021] The AGE-TOD model fine-tunes the GPT-2 model and optimizes the weighted sum of the language model (LM) and the next sentence classification (NC) objectives. For the language model, we use a standard left-to-right language model, where the objective function of the language model calculates the probability of the next token given the previous token. The objective function of the language model is as follows:

[0022]

[0023] In the task of sentence classification, this paper randomly samples from the data set and introduces interference responses. What the AGE-TOD model needs to do is to distinguish the correct response (dialogue state + system action + correct system response) and the interference response (dialogue state + system action + interference system response) for NC. The linear classifier takes the hidden state of the last layer of the decoder module of GPT-2 as input, and then calculates the probability of the next sentence being a positive example or a negative example through the softmax layer. The objective function of sentence classification uses the cross entropy L between the positive example probability and the positive example. NC express.

[0024] For a given word sequence W = (w1,…,w i-1 ), the overall objective function is L LM and L NC The linear combination of is shown below:

[0025] L total (W) = α LM L LM (W)+α NC L NC (W)

[0026] where α LM and α NC is a hyperparameter of the model.

[0027] Step 4: Decoding strategy selection

[0028] When we generate a system response from the context history of the conversation, the final output is the probability distribution of the token at each position. For probability distribution, different search and decoding strategies for generating tokens have a great impact on the accuracy of the output. The most common ones are the traditional greedy search method and the beam search method. However, since the greedy search always selects the token with the highest probability at each position, it is easy to generate repeated sentences, and it may not be able to generate the system response with the highest global probability in the end; Beam Search is a heuristic graph search algorithm, which is usually used when the solution space of the graph is relatively large. In order to reduce the space and time occupied by the search, at each step of depth expansion, some nodes with poor quality are cut off and some nodes with higher quality are retained. This reduces space consumption and improves time efficiency, but the disadvantage is that there may be a potential optimal solution that is discarded, the generated words are prone to incoherence, the context is relatively contradictory, and it is easy to generate strange words and rare words. Top-k sampling means sorting the probabilities and then converting the probabilities after the kth position to 0. Obviously, this can increase the interference of the long-tail probability and reduce the possibility of deviation. However, in some examples, many words can indeed be sampled from a reasonable range (see the broad distribution in the figure below), but not for certain emotions (see the narrow distribution in the figure below). AGE-TOD uses the top-p sampling algorithm to evaluate the performance of our model. It accumulates the probability distribution, and then when the accumulated value exceeds the set threshold p, the subsequent probabilities are set to 0. In the broad distribution of top-k sampling, more tokens may be required to achieve a higher probability. In the narrow distribution, only two tokens, hot and warm, may be needed to have the same probability, avoiding interference from the long tail. At the same time, when the confidence of the top token is not high, diversity is still guaranteed.

[0029] Step 5: Handling empty query results

[0030] Although GPT-2 calls the query module to complete the interaction with the database, GPT-2 cannot know how many candidate answers satisfy the constraints. Therefore, there is a situation where there are no candidate objects that satisfy the constraints in a query, which we call an empty query result.

[0031] In this case, the dialogue system should generate a system response corresponding to the intent Empty-Query-Result. The AGE-TOD algorithm monitors the system actions generated by GPT-2 and replaces them with <eqr>, and feed the modified input to GPT-2 to generate the system response.

[0032] Although the above is a description of the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. Where equivalent substitution or equivalent substitution is adopted, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.< / eqr> < / dc> < / nm> < / sa> < / ds> < / sys> < / usr>

Claims

1. A task-based question answering system based on a pre-trained autoregressive language model, the algorithm includes the following steps: Step 1: Construct input representation First, the dialogue state and system behavior are converted into word tags required by GPT-2. The input representation used by AGE-TOD is shown in Figure 3, including word embedding, speaker embedding, and position embedding. This paper introduces the separator <usr> 、 <sys> 、 <ds>and <sa>, respectively, indicate the start of the sequence representation of user utterance, system response, dialog state, and system operation. The involved fields and slots are also indicated by additional special markers. <nm>means "not mentioned", <dc> It means "it doesn't matter".< / dc> < / nm> < / sa> < / ds> < / sys> < / usr> Step 2: Generate symbolic markup Since each conversation in the dataset is generated based on the results of a database query, the slot values ​​to be filled, such as special information such as serial numbers and addresses, are only valid for specific conversation instances. In addition, in order to provide appropriate information based on the conversation context, the AGE-TOD model symbolizes the values ​​of all slots to be filled (serial number, name, zip code, telephone number, address, etc.) and changes them to [DOMAIN SLOTNAME]. Then, the AGE-TOD model learns to generate symbolic system responses, and finally uses a post-processing code to replace the string with information from the database. Step 3: Fine-tuning GPT-2 The AGE-TOD model fine-tunes the GPT-2 model and optimizes the weighted sum of the language model (LM) and the next sentence classification (NC) objectives. For the language model, we use the standard left-to-right language model, where the objective function of the language model calculates the probability of the next token given the previous token. The objective function of the language model is as follows: In the sentence classification task, this paper randomly samples from the data set and introduces interference responses. What the AGE-TOD model needs to do is to distinguish the correct response (dialogue state + system action + correct system response) and the interference response (dialogue state + system action + interference system response) for NC. The linear classifier takes the hidden state of the last layer of the decoder module of GPT-2 as input, and then calculates the probability of the next sentence being a positive example or a negative example through the softmax layer. The objective function of sentence classification uses the cross entropy L between the positive example probability and the positive example. NC express. For a given word sequence W = (w1,…,w i-1 ), the overall objective function is L LM and L NC The linear combination of is shown below: L total (W)=a LM L LM (W)+a NC L NC (W) where α LM and α NC is a hyperparameter of the model. Step 4: Decoding strategy selection When we generate a system response from the context history of the conversation, the final output is the probability distribution of the token at each position. For the probability distribution, different search and decoding strategies for generating tokens have a great impact on the accuracy of the output. The most common ones are the traditional greedy search method and the beam search method. However, since the greedy search always selects the token with the highest probability at each position, it is easy to generate repeated sentences, and it may not be able to generate the system response with the highest global probability in the end. Beam Search is a heuristic graph search algorithm, which is usually used when the solution space of the graph is relatively large. In order to reduce the space and time occupied by the search, at each step of depth expansion, some nodes with poor quality are cut off and some nodes with higher quality are retained. This reduces space consumption and improves time efficiency, but the disadvantage is that there may be potential optimal solutions that are discarded, the generated words are prone to incoherence, the context is relatively contradictory, and it is easy to generate strange words and rare words. Top-k sampling means sorting the probabilities and then converting the probabilities after the kth position to 0. Obviously, this can increase the interference of the probability of the long tail and reduce the possibility of deviation. However, in some examples, many words can indeed be sampled from a reasonable range (see the broad figure below). distribution), but not for some emotions (see the narrow distribution in the figure below). AGE-TOD uses the top-p sampling algorithm to evaluate the performance of our model. It accumulates the probability distribution, and then when the accumulated value exceeds the set threshold p, the subsequent probability is set to 0. In the broad distribution of top-k sampling, more tokens may be required to achieve a higher probability. In the narrow distribution, only hot and warm tokens may be needed to achieve the same probability, avoiding the interference of the long tail part. At the same time, when the confidence of the top token is not high, diversity is still guaranteed. Step 5: Handling empty query results Although GPT-2 calls the query module to complete the interaction with the database, GPT-2 cannot know how many candidate answers meet the constraints. Therefore, there is a situation where a query does not have candidate objects that meet the constraints. We call it an empty query result (Empty-Query-Result).

2. The method according to claim 1, characterized in that: Introduce the separator in step 1 <usr> 、 <sys> 、 <ds>and <sa>, respectively, indicate the start of the sequence representation of user utterance, system response, dialog state, and system operation. The involved fields and slots are also indicated by additional special markers. <nm>means "not mentioned", <dc> It means "it doesn't matter".< / dc> < / nm> < / sa> < / ds> < / sys> < / usr> 3. The method according to claim 1, characterized in that: In step 2, the AGE-TOD model symbolizes the values ​​of all slots to be filled (serial number, name, zip code, phone number, address, etc.) and changes them to [DOMAIN SLOTNAME]. Then, the AGE-TOD model learns to generate symbolic system responses, and finally uses a post-processing code to replace the string with information from the database.

4. The method according to claim 1, characterized in that: In step 3, the word and sentence classification task is introduced. In the sentence classification task, this paper randomly samples from the dataset and introduces interference responses.

5. The method according to claim 1, characterized in that: In step 4, the top-p algorithm is used as the decoding strategy instead of greedy search or beam search, which ensures the diversity and accuracy of the output.

6. The method according to claim 1, characterized in that: In step 5, the AGE-TOD algorithm monitors the system operations generated by GPT-2 and replaces them with <eqr> , and feed the modified input to GPT-2 to generate the system response.< / eqr>

Images