Researcher pre-training method based on KL regularization under open domain questions and answers

Abstract

The invention discloses a searcher pre-training method based on KL regularization under open domain questions and answers. According to the method, training is carried out by using two forms of rewriting of the current question and splicing of the historical question (including answers) and the current question, and the retrieval results of the two question forms are pulled in by using KL divergence, so that the training stability is improved, and the semantic understanding capability of a question encoder on the questions is improved. According to the method, the difficult negative samples, namely the articles retrieved by the TF-IDF but not containing the correct answers, are introduced, so that the training effect of the retriever is improved, and the retriever can discriminate articles more strongly associated with the current question from a batch of articles with similar contents.

Description

technical field [0001] The invention relates to the field of conversational open domain question answering, in particular to a retrieval pre-training method based on KL regularization. Background technique [0002] In the field of open-domain question answering, a two-stage system is mainly used, which consists of two components: a retriever and a reader. First, the retriever finds N articles with high relevance to the question from a large number of Wikipedia articles in advance according to the dialogue question, and secondly, the reader finds the correct answer from these N articles according to the dialogue question. [0003] The present invention focuses on the retrieval device, and the traditional retrieval device can be realized by retrieval algorithms such as TF-IDF and BM25. These retrieval algorithms are based on the overlapping frequency of words, it cannot handle highly semantically related but little lexical overlap, and it is not trainable. In recent years, w...

AI Technical Summary

Problems solved by technology

However, in the subsequent reasoning stage, the representation of the question is a combination of historical questions (which can contain answers) and the current question. During training and reasoning, the inconsistent representation of the question makes the model unable to truly understand the dialogue question.

[0005] (2) The semantic difference between positive and negative samples is large, and it is difficult to improve the training effect

Due to the large gap between the related articles of other questions and the related articles of the current question in the same batch, it is relatively simple to distinguish positive and negative samples during training, while in the inference stage, it may be necessary to find the most relevant one among many articles with similar content. Article, not enough difficulty in training will lead to poor training effect

Images