A Pre-training Method for Retrieval Based on KL Regularization in Open-domain Question Answering

Abstract

The present invention discloses a retriever pre-training method based on KL regularization in open domain question answering. The present invention uses two forms of training, namely, rewriting the current question and splicing the historical question (which may contain the answer) with the current question, and uses KL divergence to pull in the retrieval results of the two question forms, which not only improves the stability of the training, but also improves the semantic understanding ability of the question encoder for the question. The present invention improves the training effect of the retriever by introducing difficult negative samples, that is, articles retrieved by TF-IDF but not containing the correct answer, so that the retriever can identify articles with stronger relevance to the current question from a group of articles with similar content.

Description

Technical Field

[0001] The present invention relates to the field of conversational open-domain question answering, and particularly to a retriever pre-training method based on KL regularization. Background Art

[0002] In the field of open-domain question answering, a two-stage system is mainly adopted, which consists of two components: a retriever and a reader. First, the retriever pre-finds N articles with relatively high relevance to the question from a large number of Wikipedia articles according to the conversational question. Second, the reader finds the correct answer from these N articles according to the conversational question.

[0003] The present invention focuses on the retriever. Traditional retrievers can be implemented by retrieval algorithms such as TF-IDF and BM25. These retrieval algorithms are based on the overlapping frequency of words, and they cannot handle the situation where the semantics are highly relevant but the lexical overlap is small, and they are not trainable. In recent years, with the gradual development of pre-trained models and the continuous optimization of vector retrieval tools such as faiss, in order to solve some deficiencies of traditional sparse retrievers, many researchers have carried out research on vector retrievers. Most vector retrievers adopt a dual-encoder architecture, which consists of a question encoder and an article encoder, and encode the question and the article into low-dimensional vectors respectively. Many similarity functions, such as inner product and Euclidean distance, can be used to measure the correlation degree between the question and the article. The current vector retriever used for retrieving relevant articles of conversational questions still has the following deficiencies:

[0004] (1) The understanding of conversational questions is not accurate enough. Since there are some demonstrative pronouns in the conversational questions, it is necessary to connect the context to understand the questions. In order to alleviate the impact brought by this problem, existing systems rewrite the current question and use real entities to replace pronouns to pre-train the retriever. However, in the subsequent inference stage, the representation of the question uses a combination of historical questions (which may include answers) and the current question. When training and inferring, the inconsistent question representations cause the model to not really understand the conversational questions.

[0005] (2) The semantics of positive and negative samples differ greatly, making it difficult to improve the training effect. The current system uses the positive samples of other questions in the same training batch as the negative samples of the current question. Since the content of the relevant articles of other questions in the same batch and the relevant articles of the current question differ too much, it is relatively simple to distinguish positive and negative samples during training. However, in the inference stage, it may be necessary to find the most relevant article among many articles with similar content. Insufficient difficulty during training will lead to poor training effects. Summary of the Invention

[0006] In view of the above problems, the present invention discloses a retriever pre-training method based on KL regularization, which can enable the retriever to retrieve a set of articles more relevant to the conversational question.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] Step 1, construct training data:

[0009] For each dialogue question, rewrite the current question q rw Concatenate it with the historical questions (which may include answers) and the current question q or These two forms, and give a relevant article and an irrelevant article (an article retrieved by TF-IDF but not containing the correct answer) for the corresponding question.

[0010] Step 2, construct a retriever model:

[0011] The retriever adopts a dual-encoder architecture, consisting of a question encoder and an article encoder. Both the question encoder and the article encoder can be implemented by the ALBERT model.

[0012] Step 3, input the training data set into the retriever for training:

[0013] q in the same training batch rw and q or are fed into the question encoder, and p in the same training batch + and p - are fed into the article encoder. The loss function during training consists of two parts, namely the cross-entropy loss and the KL divergence (Kullback-Leibler divergence) loss. Finally, the question encoder and the article encoder are trained.

[0014] Step 4, obtain the page data of Wikipedia and get the vector representation of Wikipedia paragraphs:

[0015] After obtaining the page data of Wikipedia from the web, they are segmented into article paragraphs with a length not exceeding 384 words. The segmented paragraphs are encoded by the article encoder in the retriever to obtain the vector representation of Wikipedia articles.

[0016] Step 5, use the faiss vector retrieval tool to construct an index of Wikipedia articles.

[0017] Step 6, encode the dialogue question using the question encoder, and query the top K most relevant articles from faiss according to the question representation vector.

[0018] The beneficial effects of the present invention are:

[0019] 1. The present invention conducts training in two forms: rewriting the current question and concatenating the historical question (which may include the answer) with the current question, and uses the KL divergence to narrow the retrieval results of the two question forms, not only improving the stability of training, but also enhancing the semantic understanding ability of the question encoder for questions.

[0020] 2. The present invention improves the training effect of the retriever by introducing hard negative samples, that is, articles retrieved by TF-IDF but not containing the correct answer, enabling the retriever to identify articles with a stronger association with the current question from a batch of articles with similar content. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of the pre-training method for the retriever based on KL regularization proposed by the present invention.

[0022] Figure 2 It is a model diagram of the retriever proposed by the present invention.

[0023] Figure 3 It is a training schematic diagram of the pre-training method for the retriever based on KL regularization proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] In order to clarify the technical solution and working principle of the present invention, the disclosed embodiments of the present invention will be further described in detail below with reference to the drawings. This embodiment provides a pre-training method for a retriever based on KL regularization in open-domain question answering, as Figure 1 shown, and the method mainly includes the following steps:

[0025] Step 1, constructing training data:

[0026] Obtain the current question q from the training dataset of OR-QuAC c and historical question and answer pairs Take the historical dialogue window size as w, and concatenate the question and the answer as the question, denoted as

[0027] q or =[CLS]q 1 [SEP]a 1 [SEP]q c-w [SEP]a c-w [SEP]…[SEP]q c-1 [SEP]a c-1 [SEP]q c [SEP].

[0028] The rewritten current question q rwIt is provided by the CANARD dataset, which replaces some demonstrative pronouns in the current question with real words. The relevant articles of the current question contain the answer to the question, while the irrelevant articles are retrieved by TF-IDF and do not contain the answer to the question.

[0029] Finally, the training data is denoted as where m represents the total number of groups of training data, and are two representation methods of the question, is the concatenation of the current question and the historical questions (which may include answers), is the rewrite of the current question, is the relevant article corresponding to the current question, are n irrelevant articles corresponding to the current question. Among these n irrelevant articles, one is the difficult negative sample corresponding to this question, that is, the article retrieved by TF-IDF but does not contain the correct answer, and the other n - 1 articles are the positive samples and difficult negative samples of other questions from the same training batch. For the specific acquisition of the difficult negative sample, please refer to Step 3.

[0030] Step 2, construct the retriever model: Figure 2 is the model diagram of the retriever. The retriever adopts a dual-encoder architecture, consisting of a question encoder and an article encoder. The internal implementations of the question encoder and the article encoder are the same, as described below:

[0031] (1) First, use the bidirectional Transformer encoder inside the ALBERT pre-trained language model to process the question / article input, and take the vector at the [CLS] position as the output, with a dimension of 768.

[0032] (2) Second, pass the vector at the [CLS] position of the previous layer into the Dropout layer to prevent the trained model from overfitting.

[0033] (3) Finally, through a Dense linear layer, map the 768-dimensional vector to 128 dimensions.

[0034] After obtaining the vector representations of the question and the article respectively, calculate the dot product of the vectors to represent the relevance between the question and the article.

[0035] Step 3, input the training dataset into the retriever for training: Figure 3 is the training schematic diagram of the model. The specific method combined with this diagram is:

[0036] (1) The total number of training rounds for the retriever is 12 rounds. After each training round, the retriever model R(E) of the current round will be obtained. <E≤12。在训练的第一轮,困难负样本采用的是使用TF-IDF检索到但不包含答案的文章。之后的每一轮训练采用的困难负样本,是上一轮检索器模型R(E)检索到但不包含答案的文章。随着检索器模型训练,检索器检索到的与当前问题相关的文章与真正的正样本的相似度是接近的,而使用该样本作为困难负样本,对检索器的考验更高,也更有利于区分真正的正样本和与正样本极相似的负样本。

[0037] (2) Rewrite the current problem And the splicing of current questions and historical questions (including answers) The question encoder is passed in separately to obtain vector representations of two sets of questions. The relevant articles and irrelevant articles of the current question are passed in separately to the article encoder to obtain vector representations of two sets of articles.

[0038] (3) Using the similarity function, that is, the dot product of the question vector representation and the article vector representation, we can obtain the distribution of the retrieval. Here we can get two sets of distributions, namely and The cross entropy function is used to calculate the retrieval loss of each group. The calculation formula is as follows:

[0039]

[0040]

[0041]

[0042]

[0043] Taking the average of the two, we get the retrieval loss in a training batch, which is calculated as follows:

[0044]

[0045] (4) In addition to calculating the cross entropy loss between the two groups, the KL divergence between the two groups of distributions is also calculated. KL divergence is a relative entropy that represents the degree of difference between the two groups of distributions. Using KL divergence as one of the losses can narrow the distance between the two groups of distributions. Narrowing the distance between the two groups of distributions is to ensure that different dialogue expressions have similar distributions, which improves the semantic understanding ability of the question encoder and makes the training effect more stable. The calculation formula of KL divergence is as follows:

[0046]

[0047] The final losses are as follows:

[0048]

[0049] Step 4: Obtain the page data of Wikipedia and get the vector representation of Wikipedia paragraphs:

[0050] The present invention uses the Wikipedia data storage on October 20, 2019. Then WikiExtractor is used to extract and clean the text from the data storage, generating more than 5.9 million Wikipedia articles. After that, the BERT tokenizer is also used to divide the articles into at most 384 words, and finally about 11 million Wikipedia article paragraphs are obtained.

[0051] Encode these more than 11 million Wikipedia article paragraphs respectively with the article encoder trained in Step 3 to obtain the vector representation of Wikipedia articles.

[0052] Step 5: Use the faiss vector retrieval tool to build an index of Wikipedia articles:

[0053] First, use faiss to create an index and specify the inner product as the query method; second, store the vector representation of the Wikipedia articles obtained in Step 4 into the index. The present invention also performs hardware acceleration through GPU, greatly improving the retrieval speed.

[0054] Step 6: Encode the dialogue question with the question encoder and query the top K most relevant articles from faiss according to the question representation vector:

[0055] (1) Obtain the current question, historical questions and their answers from the test dataset of OR - QuAC, construct the dialogue question in the same way as in Step 1, concatenate the historical questions (and their answers) and the current question, and take the dialogue window size as w to get q or .

[0056] (2) Pass q or into the question encoder to obtain the question representation vector, use the index constructed by faiss, and obtain the distances and indices of the top K closest to the retrieval vector through the search interface. Finally, the content of the Wikipedia article can be obtained according to the index value.

[0057] The above has made an exemplary description of the present invention in combination with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited by the above - mentioned methods. All kinds of non - substantial improvements made by adopting the method concept and technical solution of the present invention; or those directly applying the above - mentioned concept and technical solution of the present invention to other occasions without improvement or with equivalent replacement are within the protection scope of the present invention.

Claims

1. A pre-training method for a retriever based on KL regularization in open-domain question answering, characterized in that the method comprises the following steps: Step 1, construct training data: For each conversation question, provide two forms of the question, namely the rewritten current question q rw , the concatenation of the historical question and the current question q or ; Give a relevant article p for the corresponding problem + and an irrelevant article p - ; Step 2, construct a retriever model: The retriever adopts a dual-encoder architecture and consists of a question encoder and a passage encoder; Step 3, input the training data set into the retriever for training: q of the same training batch rw and q or are input into the question encoder, and p of the same training batch + and p - are input into the passage encoder, where the loss function during training consists of two parts, namely cross-entropy loss and KL divergence loss; Step 4, obtain the page data of Wikipedia and get the vector representation of Wikipedia articles: Step 5, use the faiss vector retrieval tool to construct an index of Wikipedia articles; Step 6, use the question encoder to encode the dialogue questions, and query the top K most relevant articles from the faiss vector retrieval tool according to the question representation vector; Rewriting q of the current problem in Step 1 rw Provided by the CANARD dataset, relevant articles of the current problem contain the answer to the problem, while irrelevant articles are retrieved by TF-IDF and do not contain the answer to the problem, forming difficult negative samples corresponding to the problem.

2. The pre-training method for a retriever based on KL regularization in open-domain question answering according to claim 1, characterized in that: The retriever in Step 3 needs to be trained for multiple rounds. After each round of training, the retriever model of the current round will be obtained; in the first round of training, the hard negative samples are the articles retrieved by TF-IDF but not containing the answer, and the hard negative samples used in each subsequent round of training are the articles retrieved by the retriever model of the previous round but not containing the answer.

3. The pre-training method for a retriever based on KL regularization in open-domain question answering according to claim 1, characterized in that: Both the question encoder and the passage encoder in Step 2 are implemented by the ALBERT model.

4. The pre-training method for a retriever based on KL regularization in open-domain question answering according to claim 1, characterized in that: Step 4 is specifically: after obtaining the page data of Wikipedia from the web, cut them into article paragraphs; encode the cut article paragraphs with the passage encoder in the retriever to obtain the vector representation of Wikipedia articles.

5. The pre-training method for a retriever based on KL regularization in open-domain question answering according to claim 4, characterized in that: The length of the cut article paragraphs in Step 4 does not exceed 384 words.

Images