Conversational open-domain question answering (openqa) method based on curriculum leanring
Patent Information
- Application Number
- GB2024002893
- Authority / Receiving Office
- GB · GB
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-02-29
- Publication Date
- 2024-11-27
Smart Images

Figure 00000000_0000_ABST
Abstract
Claims
1. A conversational open-domain question answering (OpenQA) method based on curriculum learning, comprising following steps:step 1: constructing a retriever model, wherein a retriever adopts a dual-encoder architecture, comprising a question encoder and an article encoder;step 2: constructing a sorter model, wherein a sorter adopts a simple linear layer network, re-sorts top K articles retrieved by the retriever, and retrieves top T articles from the top K articles and inputs the top T articles into a subsequent reader module, such that an article more highly related to a question is ranked at a more prominent position;step 3: constructing a reader model, wherein the reader model comprises a bidirectional encoder representations from transformers (BERT) model or a robustly optimized BERT approach (RoBERTa) model;step 4: performing joint training based on curriculum learning, whereina semi-automatic curriculum learning strategy is used for joint training of the retriever, the sorter, and a reader; the semi-automatic curriculum learning strategy comprises two core components: an automatic difficulty measurer and a discrete training scheduler, which are used to improve the joint training of the retriever, the sorter, and the reader; the semi-automatic curriculum learning strategy reduces a probability of manually adding a golden paragraph during the joint training, to help make training and reasoning settings more consistent; and the semi-automatic curriculum learning strategy effectively promotes the retriever to find a paragraph containing a true answer, and encourages the reader to find a relevant fragment of a correct answer without a correctly related article;step 5: using, by the automatic difficulty measurer, a loss value of article retrieval as a benchmark to measure a current training difficulty; and determining that a difficult mode in which the retriever is difficult to discover a positive sample article is available when the retriever has a large loss, or determining that a simple mode in which the retriever is easy to discover a positive sample article is available when the retriever has a small loss;step 6: when the automatic difficulty measurer determines that the difficult mode is available, if the retriever does not retrieve the positive sample article, automatically introducing, by the discrete training scheduler, the positive sample article into a retrieval result to make the training easier; or when the automatic difficulty measurer determines that the simple mode in which the retriever is easy to discover the positive sample article is available, no matter whether the retriever retrieves the positive sample article, skipping adding, by the discrete training scheduler, the positive sample article to a retrieval result; wherein the discretetraining scheduler is a baby step scheduler with a finer granularity; andstep 7: using a sum of scores from the sorter and the reader as a predicted score, and outputting, by a system, an answer span with a highest total score as a finally predicted answer.
2. The conversational OpenQA method based on curriculum learning according to claim 1, wherein in the step 2, the sorter module performs contrastive learning to enhance sorter training based on a neural network; and a key idea of comparison is to use a loss to compare distances between representations of question-article pairs, and force similar sample pairs to be close together in potential space while keeping dissimilar samples away.
3. The conversational OpenQA method based on curriculum learning according to claim 1 or claim 2, whereinin the step 1, both the question encoder and the article encoder are implemented by an acoustic and laryngeal biofeedback enhancement real time (ALBERT) model.
4. The conversational OpenQA method based on curriculum learning according to any one of claims 1 to 3, whereinthe semi-automatic curriculum learning strategy in the step 4 requires a plurality of rounds of training, and after each round of training, a model parameter for the current round of training is obtained;for each round of training, p® = {p®!^ is set to a set of the top K articles retrieved by the retriever for a question of a current batch;in the difficult mode, the discrete training scheduler adds a positive sample article of each question to the p® to form a set of retrieved articles with positive samples, which is denoted as p^ = {p® ufpHlQithen, a loss function to be finally optimized is defined as follows:LTinai = (Pkg) + (1 - (p®)wherein v® is determined by the automatic difficulty measurer based on a loss of the retriever in a previous iteration, wherein when the automatic difficulty measurer determines that the difficult mode is currently available, a value of the v® is 1, and the loss is calculated based on the article set p^2 to which the positive sample article is manually added; and conversely, the value of the v® is 0, and the loss is calculated based on the article set p®;in the difficult mode, the value of the v® is more likely to be set to 1, such that during the current round of training, there is a higher probability of adding an article containing the correct answer to the set of retrieved articles for training; and in the simple mode, a possibility of adding the article containing the correct answer to the set of retrieved articles is low.
5. The conversational OpenQA method based on curriculum learning according to any one of claims 1 to 4, wherein score calculation in the step 7 comprises:for each article pc,j in a set, obtaining the score Spost(qfn,pcj) from the sorter; andobtaining the score from the reader, comprising a score Sselect(qcd,pc j) for article selection and a score for an answer span, wherein the score for the answer span comprises a score for a start position and a score for an end position, and is calculated according to a following formula:^spanGfc <Pc,j<Sp) — m.GXSpgPcj{Ss(qc < Pc,j< D-s]) + $e (Qc 'Pc,j'[te])}wherein sp = ([ts], [te]) represents an answer span starting with a tag [ts] and ending with a tag [te]; andusing the sum of the scores from the sorter and the reader as a finally predicted score, as shown in a following formula:S(qc <Qc 'PcJ'Sp) ^post(Qi < Pcj) "T Sreader(Qc <Pc,j<Sp)wherein Sreader(q^d,pC;j,sp) = Sselect(q^d,pcj) + Sspan(q^d,pcj,sp) ; and for each question in a conversation, the system outputs the answer span sp with the highest total score S as the finally predicted answer.
