A method, device, and storage medium for implementing open-domain multi-answer question answering.

Abstract

A method, device, and storage medium for implementing open-domain multi-answer question answering includes three stages: a dense retrieval stage, a domain-specific pre-training stage, and a supervised multi-answer generation stage. First, a dense retrieval module finds relevant paragraphs from an encyclopedia corpus. Then, a multi-paragraph reader encodes the relevant paragraphs. Finally, an answer generator is trained based on the optimal generation order of the multi-answer set to obtain multiple answers to the question. Because the annotation cost of multi-answer datasets is too high, multi-answer datasets are generally small. This invention proposes domain-specific pre-training to improve the performance of multi-answer generation. The optimal generation order strategy for the multi-answer set alleviates the error bias caused by the forced specification of the generation order in the one-to-many generation paradigm in multi-answer solving. This invention has achieved good results on open-domain multi-answer solving datasets.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to intelligent question answering technology in natural language understanding. It is a method, device and storage medium for implementing open-domain multi-answer question answering. Background Technology

[0002] Open-domain question answering tasks require question answering systems to search knowledge document bases, find relevant paragraphs for the question, and then predict the answer after inputting the question and paragraphs into a reading comprehension model. Open-domain single-answer generation tasks involve providing a question and expecting the model to predict a single answer. Due to the complexity and diversity of questions, relevant paragraphs are typically retrieved from open corpora, and the answer is predicted based on the structure found in these paragraphs. Furthermore, open-domain multi-answer tasks involve questions corresponding to multiple answers, potentially distributed across multiple paragraphs, making them more difficult and challenging than single-answer generation. A representative dataset for multi-answer generation tasks is AmbigQA. AmbigQA is constructed based on the open-domain single-answer question answering dataset Natural Question, where questions are derived from real user queries in Google searches, and each question corresponds to an answer from Wikipedia. However, related work has found that many questions in Natural Question contain ambiguities such as unclear entity references and lack of time constraints, thus potentially leading to multiple answers. AmbigQA primarily focuses on ambiguous questions in Natural Question and strives to find all possible answers from Wikipedia for each question. For example, when asked when a movie will be released, this invention aims to find all possible answers from relevant paragraphs, such as release dates in different regions.

[0003] Relevant paragraph retrieval depends on the retrieval engine. Retrieval engines include dense and sparse retrieval architectures. Sparse retrieval, such as TF-IDF or BM25, uses inverted indexes to efficiently match keywords and can be viewed as representing the question and context using high-dimensional weighted sparse vectors. Conversely, dense retrieval complements sparse representations, allowing synonyms composed of completely different characters to still map to vector spaces that are close to each other. Furthermore, dense encoding can be learned by adjusting the embedding function, providing great flexibility for task-specific representations. Dense retrieval often employs a dual-tower architecture, typically based on pre-trained language models such as BERT or ROBERTA. The dual-tower architecture uses a Siamese network, inputting the question and paragraphs separately into the encoder. The output representations are then aggregated to obtain dense vector representations of the query and paragraphs. Finally, cosine similarity is used to calculate the similarity between the two. The advantage is that all paragraphs can be pre-encoded offline as vectors, and after encoding the user query online, the most relevant paragraphs are quickly retrieved using approximate nearest neighbor search, significantly improving efficiency compared to single-tower architecture models.

[0004] In OpenQA (Open Domain Question Answering) tasks, the answer can generally be obtained directly from the supporting paragraphs. That is, the answer is a continuous string of supporting paragraphs. Two common architectures are used to solve this task: extractive and generative architectures. Extractive architectures first input the question and paragraphs into an encoder such as BERT, then predict the start and end positions of the answer. There are pre-trained models specifically for answer extraction, such as SpanBERT and SPlinter. However, extractive models face two major challenges in open-domain multi-answer question answering: First, open-domain question answering is generally based on multiple supporting paragraphs. In this context, aggregating and combining evidence from multiple paragraphs using extractive models is not straightforward and requires additional techniques. Second, extractive architectures can currently only predict one start and end position, thus only suitable for single-answer question answering tasks. Generative architectures are based on sequence-to-sequence models, inputting the question and paragraphs into an encoder, and then generating the answer through a decoder. Generative architectures are based on pre-trained language models such as T5 and BART. T5, in particular, introduces a unified framework—a text-to-text architecture—to explore solutions for various downstream tasks in Natural Language Processing (NLP), including summarization, question answering, and text classification. T5 employs a pre-training method that corrupts the text at the input and restores it at the output. However, the encoders in generative pre-trained models generally limit the maximum input length, making it impossible to read large numbers of paragraphs. The FiD (Fusion-in-Decoder) architecture, building upon the T5 model, uses multiple encoders to encode multiple paragraphs separately, and then aggregates the encoded vectors at the decoder, achieving the reading of multiple paragraphs. FiD has achieved excellent results in open-domain single-answer generation.

[0005] Currently, there are two main approaches to achieve the transfer from single-answer question answering to multi-answer question answering tasks in open domains. The first approach is to model it as a one-to-one task. In this paradigm, the question and its corresponding multiple answers are split into multiple sample pairs, with each sample pair containing only one answer for each question. Because only one answer serves as supervision during training, beam search is needed to sample multiple output sequences if multiple answers need to be generated during testing. This paradigm has two major problems: First, the answer dependency problem. The multiple answers generated by beam search have no semantic dependency and are likely to generate semantically repetitive answers. Second, the dynamic number of answers is limited; the sampling number for beam search is often fixed, meaning it cannot generate a dynamic number of answers for different questions. The other approach is to model it as a one-to-many task, where multiple answers are output as a fixed sequence through the decoder, roughly achieving the transfer from single-answer generation to multi-answer generation. Because the decoder can see the previously generated sequences, it solves the problems of semantic dependency and repetitive generation. This architecture relies on the end marker predicted by the decoder to achieve the problem of generating a dynamic number of answers. However, this architecture also has a problem: it rigidly sets the dependency order between answers. However, some related work has found that different order functions can have a significant impact on the results. The randomly set answer generation order may not be the best choice, but how to learn the optimal order has not yet been fully studied.

[0006] Because constructing multi-answer generation datasets requires significant human annotation resources, they are generally small, leading to insufficient training samples and hindering model performance. Some work has shown that adding unannotated domain-specific text to an open-domain pre-trained language model for further training, followed by fine-tuning on downstream task datasets, can improve downstream task performance. However, this work primarily addresses the incompatibility issue caused by significant differences between general and specific domain corpora, without considering the impact of differences between the pre-training and downstream tasks. Taking the pre-training task of the T5 language model and the open-domain multi-answer generation task as examples, T5 is a pre-training task where text is corrupted at the input and restored at the output; the input is corrupted text, and the prediction target is the restored text. The open-domain multi-answer generation task takes a question and related paragraphs as input and predicts the answer. The two tasks differ significantly, therefore, a suitable transition phase addressing these differences is essential.

[0007] For research on open-domain multi-answer question answering, reference 1, "Answering Open-Domain Multi-AnswerQuestions via a Recall-then-Verify Framework" (Shao & Huang, ACL 2022), proposes a "recall-re-verify" scheme. This explicitly models the one-to-many task of multiple answers as multiple one-to-one tasks. That is, for the same question, different answers are generated based on different paragraphs (i.e., the recall phase), and then a verification (i.e., verifier) ​​is used to filter out the set of answers exceeding a certain threshold. The advantage of this approach is reduced memory usage and the ability to read multiple paragraphs, but it also has the following disadvantages:

[0008] 1. The number of answers generated for each sample is determined by the relative magnitude of the probability of the predicted answer and the threshold. However, due to various reasons such as different difficulty levels, the probability distribution of the predicted answers varies greatly among different samples. It is impossible to use a fixed threshold to verify, filter, and divide the set of predicted answers for the entire sample.

[0009] 2. This method fails to model dependencies between multiple answers. In open-domain multi-answer question-answering tasks, different answers may be related; for example, some answers may be synonyms or near-synonyms, or some answers may be related entities or concepts. The model needs to be able to identify and utilize these correlations to generate better answers. However, this method obtains multiple answers in parallel without utilizing the dependencies between them. Summary of the Invention

[0010] The problem this invention aims to solve is: in existing work, the fixed answer generation order in open-domain multi-answer generation architectures leads to error bias; in addition, the training samples used by the reader are much smaller in scale than the open-domain data, and how to set up the reader's training scheme to improve the answering performance on the relatively small multi-answer dataset is also a problem that needs to be solved. Here, the multi-answer dataset refers to the domain training dataset used to train the reader.

[0011] The technical solution of this invention is: an implementation method for open-domain multi-answer question answering, comprising a dense retrieval stage, a domain-specific pre-training stage, and a supervised multi-answer generation stage, for a given question... Firstly, based on a dense search engine, data is collected from encyclopedia corpora. The search yielded the most relevant Top results paragraph Then, based on the domain-specific corpus, a multi-paragraph reader is constructed and pre-trained within the domain. Finally, the multi-paragraph reader is fine-tuned on a supervised open-domain multi-answer question-answering dataset. The optimal generation order of the multi-answer set is defined to reduce the influence of the order on the generation of multiple answers. Multiple answers are obtained based on the given question and the retrieved paragraphs, as follows:

[0012] 1) Train a dense retrieval engine. The dense retrieval engine measures the relevance of paragraphs to questions by calculating the dot product of semantic vectors encoded by paragraph encoders and question encoders, and outputs a set of the most relevant paragraphs for a given question for subsequent reading stages.

[0013] 2) The domain-specific pre-training stage achieves a transition between the general domain pre-training stage and the multi-answer generation stage by synthesizing multi-paragraph, multi-answer question-and-answer data and training it on a multi-paragraph reader. This includes constructing self-supervised data and pre-training the multi-paragraph reader. The self-supervised data consists of three parts: questions... Related paragraphs , and the answer The process of constructing self-supervised data is as follows:

[0014] 2.1) Select a set of relevant search paragraphs from the domain corpus. The related search segments have similar topics, contain common entities, and the topic similarity is set by the user.

[0015] 2.2) Randomly from Select The paragraphs were used as the source for answer extraction. The remaining paragraphs are considered as related paragraphs. ;

[0016] 2.3) Identified based on Spacy tool Entities in the list, and filter out those not in the list. Entities appearing in;

[0017] 2.4) The filtered entities are divided into several groups according to the entity types identified by Spacy, and then a question-and-answer sample is constructed for each group of entities: the entities in each group are concatenated to form the answer. Each entity in The sentences are concatenated, and the entities are replaced with the [MASK] character to form the question. The corresponding relevant paragraphs are: ;

[0018] After constructing the self-supervised data, a generative multi-paragraph reader is obtained by modeling using the FiD architecture and then optimized and trained.

[0019] 3) In the supervised multi-answer generation stage, fine-tuning is performed on the multi-answer dataset. For a given question... The multi-paragraph reader obtains the Top results through a dense search engine. paragraph Information, predicting problems The corresponding answer set contains multiple different answers, which are separated by the delimiter [SEP]. Considering that multiple answers may have multiple different generation orders, all permutations of the answer set are enumerated to obtain the cross-entropy loss between the set and the prediction of the multi-paragraph reader. The sequence of answers with the minimum loss is the best answer sequence, which is used as the final multi-answer result.

[0020] The pre-training phase in this invention comprises two subtasks: self-supervised data construction and multi-paragraph reader training. The construction of self-supervised data is particularly challenging, namely, how to create question-answer pairs and related paragraphs from unlabeled data. This invention proposes utilizing a set of related paragraphs, randomly selecting some paragraphs to extract answers, constructing questions, and treating the remaining paragraphs as related paragraphs. Using a set of related paragraphs makes the connection between the constructed question-answer pairs and related paragraphs stronger, more closely approximating the downstream scenario. The set of related paragraphs can include paragraphs from the same article in Wikipedia, paragraphs corresponding to the same question in an open-domain supervised question-answering dataset, etc. This invention adopts the latter because it can be constructed based on a single-answer dataset, resulting in a larger scale.

[0021] The present invention also proposes an electronic device, including a storage medium and a processor, wherein the storage medium is used to store a computer program, and the processor is used to execute the computer program, wherein when the computer program is executed, the above-described method for implementing open-domain multi-answer question answering is implemented.

[0022] The present invention also proposes a computer-readable storage medium storing a computer program, which, when executed, implements the above-described method for implementing open-domain multi-answer question answering.

[0023] Compared to Reference 1, this invention models multi-answer question answering as a one-to-many task, i.e., inputting a question and directly predicting multiple answers. Furthermore, to reduce the impact of the answer generation order on training, the generation order with the minimum loss is adopted as the optimal order. Therefore, this invention has the following advantages: 1) It can model the dependencies between answers. During the decoding process, subsequent answers can use previously generated answers as references. 2) The number of answers generated for each sample is fixed, eliminating the need for additional threshold parameters to determine the number of answers generated for each sample.

[0024] Currently, most readers are based on open-domain pre-trained language models, such as BERT, T5, and BART. This invention introduces domain-specific pre-training on top of the pre-trained language model. Specifically, it performs a second stage of pre-training on domain-specific corpora using a pre-training method more similar to the downstream reading task. Then, based on the domain-specific pre-trained model, fine-tuning is performed on the downstream supervised task. This has the following advantages: 1) the corpus is more similar to the downstream task; 2) the task is more similar to the downstream task. It acts as a transitional stage between open-domain pre-trained models and fine-tuning on downstream supervised tasks. Domain-specific pre-training essentially builds a bridge between general-domain pre-training and downstream supervised tasks, serving a transitional role.

[0025] The beneficial effects of this invention are twofold. Firstly, it defines an optimal order—the order that minimizes prediction loss—for unordered sets of multiple answers. This allows the model to focus on answer prediction itself, ignoring training error bias caused by the answer order. Secondly, for open-domain multiple-answer generation tasks, this invention proposes a domain-specific pre-training scheme adapted to this task. This reduces the discrepancy between models pre-trained on general domains, general tasks, and general architectures and downstream tasks, thereby improving the model's performance on downstream tasks. Attached Figure Description

[0026] Figure 1 This is a flowchart of the overall processing of the present invention.

[0027] Figure 2 This invention relates to the process of constructing self-supervised data during pre-training. Detailed Implementation

[0028] This invention proposes an open-domain multi-answer question-answering method, comprising three stages: a dense retrieval stage, a domain-specific pre-training stage, and a supervised multi-answer generation stage. These are implemented by software programs as a retrieval module, a domain-specific pre-training module, and a multi-paragraph reading and multi-answer generation module trained based on optimal order. For a given question... First, based on the retrieval module, from the encyclopedia corpus... For example, find the most relevant Top on Wikipedia. paragraph Then, based on the domain-specific pre-training scheme designed in this invention, a pre-training method more similar to that used in downstream reading tasks is adopted based on the domain-specific corpus. Domain-specific pre-training is performed on the generative model to obtain a multi-paragraph reader, which is used to predict multiple answers across multiple paragraphs based on the question. Finally, the prediction results of the multi-paragraph reader are fine-tuned on a supervised open-domain multi-answer question-answering dataset. The optimal generation order of the multi-answer set is defined to reduce the influence of the order on multi-answer generation, thus obtaining the final multi-answer result. The three stages are as follows.

[0029] Phase 1: Dense Retrieval Phase. A dense retrieval engine is trained. This engine measures the relevance of paragraphs to the question by calculating the dot product of semantic vectors encoded by the paragraph encoder and the question encoder. For a given question, it retrieves a set of the most relevant paragraphs for subsequent reading phases.

[0030] Phase 2: Domain-Specific Pre-training Phase. Given the significant differences in corpora and tasks between the general domain pre-training phase and the third phase (supervised multiple-answer generation), this invention introduces a domain-specific pre-training phase to improve the model's performance in the third phase, i.e., supervised multiple-answer generation. Taking the pre-trained language model T5 as an example, in terms of corpora, T5's pre-training corpus is based on C4 (Colossal Clean Crawled Corpus), which is text data crawled from the internet. The encyclopedia corpus used in this invention is from Wikipedia, and the two have some differences in content coverage. In terms of tasks, T5's main pre-training task is to perform small-segment replacements on the input text, i.e., randomly selecting a span, replacing it with a special symbol such as [MASK], and predicting the span replaced by the special symbol at the output. For example, if the initial text is "Thank you for inviting me to your party last week," and the replaced text is "Thank you [MASK1] I attended your [MASK2] party," then the predicted label is "[MASK1] invited [MASK2] last week." The main differences between the open-domain multi-answer generation task and the T5 pre-training task are: (1) T5 fills in the [MASK] text mainly relies on local context and does not model multi-paragraph or long-distance context; (2) The multi-answer generation task includes questions and multiple paragraphs, relies on a multi-paragraph encoder to encode multiple paragraphs, and calculates multiple answers through the interaction of questions and paragraphs; (3) The labels generated by multi-answer generation are often meaningful entities, such as location and time, while the labels in the pre-training are randomly selected from the input text and may not have practical meaning. Based on this, this invention proposes to introduce an in-domain pre-training stage as a bridge between the T5 pre-training stage and the supervised multi-answer generation stage, reducing the corpus and task differences between the T5 pre-training stage and the supervised multi-answer generation stage.

[0031] To better align with the supervised multiple-answer generation stage in the third phase, the self-supervised data constructed during the domain-specific pre-training stage also comprises three main parts: questions... Related paragraphs ,Answer The process for constructing self-supervised data is as follows:

[0032] (1) Select a set of relevant search paragraphs from the expected range within the domain. Because related paragraphs have similar themes and contain many common entities, it is easier to synthesize question-and-answer data. Similar themes and common entities are in a progressive relationship. Because the themes of the paragraphs are similar, they tend to have the same entities. The degree of theme similarity can be set by the user.

[0033] (2) Randomly from Select The paragraphs were used as the source for answer extraction. The remaining paragraphs are considered as related paragraphs. .

[0034] (3) Identified based on Spacy tool Entities in the list, and filter out those not in the list. The entities that appear in, because these entities are in Their information is not present in the code, and if they are replaced by [MASK] in subsequent question-and-answer construction, they are difficult to restore.

[0035] (4) Divide the filtered entities into several groups according to the entity types identified by Spacy, and then construct a question-and-answer sample for each group. Specifically, each group of entities is concatenated to synthesize an answer. Each entity in The sentences are concatenated, and the entities are replaced with the [MASK] character to form the question. ;question The corresponding relevant paragraphs are The problem here is the construction. and related paragraphs All from Therefore, the problem can be considered as and related paragraphs They are related.

[0036] After constructing the self-supervised data, this invention uses the classic generative multi-paragraph reading architecture FiD for modeling and optimization training.

[0037] Phase 3: Supervised Multiple-Answer Generation. The challenge in this phase is defining the generation order for an unordered set of multiple answers. This invention proposes that the optimal generation order is the one that minimizes the reader's loss in predicting answers, as this minimizes the influence of order and avoids penalties for errors.

[0038] The implementation of this invention is described in detail below.

[0039] Dense retrieval phase. The retrieval machine employs a dense retrieval approach, based on the classic DPR (Deep Retrieval Process) architecture. Both questions and paragraphs in the corpus are encoded using dense encoders, referred to as the paragraph encoder and question encoder, respectively. Map paragraphs in the corpus to 3D real-valued vectors, using a problem encoder Map the problem to A dimensional vector, and retrieve the vector closest to the question vector. A paragraph vector. This invention uses the dot product of vectors to define the similarity between the question and the paragraph: that is... , To indicate a problem, Representing a paragraph. The dense encoder uses two independent BERT networks and outputs the paragraph vector as the representation of the [CLS] character. During the training phase, let... It is by The training data consists of instances. Each instance contains one question. and a related paragraph and n unrelated paragraphs , The loss function used is InfoNCE, i.e.

[0040]

[0041] The paragraphs in this example are from Wikipedia, which contains several million paragraphs, each consisting of a title and a body text.

[0042] In the training data of the dense retrieval system, the selection of relevant or irrelevant paragraphs is based on the dataset provided directly. For irrelevant paragraphs, two different construction methods are considered: (1) Using BM25 as the retrieval algorithm to search in the open domain based on the question, the paragraphs that are returned in the top few positions but do not contain the correct answer are considered as irrelevant paragraphs. This type of paragraph has many characters that are the same as the question, so it has a high BM25 score, but it does not contain the answer and is a relatively difficult-to-distinguish negative paragraph, which is very helpful for encoder training; (2) Relevant paragraphs paired with other questions appearing in the training set. Using BM25 alone to construct negative paragraphs may lead to the problem of popularity sampling bias, that is, the sampled paragraphs are a fixed subset of the corpus, and a large number of negative paragraphs in the corpus are not sampled. This invention uses the method of in-batch negative samples to construct random negative paragraphs, that is, positive paragraphs paired with other questions in the same batch can be used as the negative paragraphs corresponding to the current question. This allows for the reuse of the calculated encoding representation and reduces computational overhead.

[0043] Domain-specific pre-training phase. This phase involves pre-training in a general domain. It bridges the gap between the general domain pre-training phase and the third phase (multi-answer generation) by synthesizing multi-paragraph, multi-answer question-and-answer data and training on a multi-paragraph reader. The domain-specific pre-training phase includes constructing self-supervised data and pre-training the multi-paragraph reader. To better align with the supervised multi-answer generation phase in the third phase, the constructed self-supervised data also comprises three main parts: questions... Related paragraphs and the answer The process for constructing self-supervised data is as described above.

[0044] After constructing the self-supervised data, this invention employs the classic generative multi-paragraph reading architecture FiD modeling to obtain a multi-paragraph reader, which is then optimized and trained on the self-supervised data. To solve this task, the model first searches for context similar to the question within relevant paragraphs, then further reconstructs entities, learning the relationships between the question, relevant paragraphs, and answers. This task is very similar to the supervised multi-answer generation task in the third stage, and therefore can serve as a bridge between the general domain pre-training stage and the third stage.

[0045] The training for answer generation uses the cross-entropy loss function, i.e.

[0046]

[0047]

[0048] in Indicates the generation loss. To retrieve the number of paragraphs, It is a concatenation of multiple segment encoding representations. [;] indicates a concatenation operation. For the first The encoded vector of each search segment, where Y represents the answer sequence. Indicates the length of the answer sequence. This represents the parameters of the reader.

[0049] The final stage is the supervised multi-answer generation stage. The FiD architecture, by employing multiple encoders to encode multiple paragraphs, enables reading of multiple paragraphs. While FiD architecture performs well in training open-domain single-answer generation tasks, it introduces the problem of passively forcing the output order of answers when transferring from single-answer to multi-answer tasks. However, since multiple answers are inherently unordered, this introduces significant bias into the training phase, severely penalizing changes in the order of answers. To avoid bias in model training due to answer order, this invention proposes a training method independent of the order of supervised labels. Given that the optimal order among multiple answers cannot be obtained in advance, the optimal order can be considered to be the order that minimizes the loss of the sample. More precisely, given the set of supervised labels corresponding to a given sample... Its size is Then the set of all possible serialized texts corresponding to it is: , the size is ,enumerate Each sequence in To obtain the corresponding loss , This invention represents the parameters of a multi-paragraph reader. It combines the cross-entropy loss of the multi-paragraph reader to obtain the optimal answer sequence. The formula:

[0050]

[0051] The loss is defined here as

[0052]

[0053] in It is a concatenation of multiple segment encoding representations. This indicates reader parameters.

[0054] This invention presents a multi-paragraph reader employing the FiD architecture, based on a sequence-to-sequence network T5 pre-trained on unsupervised data. The FiD architecture takes a question as input and the Top K retrieved paragraphs D obtained by a dense retrieval system, outputting multiple answers in a fixed sequence. Each paragraph contains a title and body text. Each retrieved paragraph and its title are concatenated with the question and processed independently by the multi-paragraph reader's encoder, adding special markers before the question, title, and text of each paragraph: "Question:", "Title:", and "Context:". This helps the model better extract and understand the input information. Adding the special marker "Question" before the input question helps the model better identify which parts of the input text are the question; the same applies to "Refer to relevant paragraphs". Finally, the multi-paragraph reader's decoder learns relevant information based on an attention mechanism by concatenating the results of all retrieved paragraphs, completing multi-paragraph reading comprehension. Because the FiD architecture only fuses retrieved paragraphs in the decoder, it reduces interactions between paragraphs, lowers computational cost, and allows for scaling to a large amount of context.

[0055] The implementation of the present invention will be illustrated below with a specific example. The question comes from AmgigqaQA: Who won the final match show 20? The correct answers are A and B.

[0056] Step 101: First, train the open-domain retrieval engine. The retrieval engine is based on a dense retrieval architecture, fine-tuned from bert-base-uncased (https: / / huggingface.co / docs / transformers / model_doc / bert). The training phase uses a dual-tower architecture, encoding the question and paragraphs separately, maximizing the similarity between the question and positive paragraphs, and minimizing the similarity between the question and negative paragraphs. The negative paragraphs for each question are sampled from the first few paragraphs retrieved by BM25 (excluding the correct answer) and from the corresponding positive paragraphs of other samples. Given the sample "Who ultimately won the competition?", the corresponding paragraphs include: "...After 99 days in the match show venue, the September 26, 2018 finale saw A crowned the winner of Big Brother in a 5–4 vote over B. ", "On 25 August 2017, B was announced as the winner of theseries having received 35.33% of the final vote, with C as the runner-up after receiving 29.92%. Negative paragraphs include phrases like, "First evictee, D, later appeared as a guest for a two-day stint on 'Match Show 16.' E returned to the house for 'Celebrity Match Show 19' as an All-Star representing this series. She won this series," etc. After training, all paragraphs in Wikipedia are encoded using a paragraph encoder, and an index is built based on FAISS. During the inference phase, an approximate nearest neighbor search algorithm is used to obtain the top few paragraphs relevant to the question.

[0057] Step 102: Then, perform domain-specific pre-training. Construct self-supervised data for the open-domain multi-paragraph multi-answer question-answering task using a set of paragraphs with relevant relationships. Specifically, based on the paragraphs corresponding to the same question in the open-domain supervised single-answer question-answering dataset, this invention uses the Natural Question dataset because of its large data scale and the very simple construction method. Figure 2 An example of self-supervised data construction is shown. Figure 2 The left side is divided into two parts, upper and lower, which represent the sources of the answers. and the remaining related paragraphs These paragraphs all come from a collection of paragraphs corresponding to a question in the Natural Question dataset. It can be found that their topics are all related to the Arctic Circle, and there are many common entities among the paragraphs. (1) First, based on the Spacy tool, identification Figure 2 The answer extracts entities from the source text, including Honningsvoor, Honningsvoor town, Arctic Norway, Hegland, Norlland County, etc. (2) Then filter out the entities Honningsvoor and Honningsvoor town that do not exist in the relevant paragraphs, because these entities do not have their information in the relevant paragraphs and are difficult to restore after being replaced by [MASK]. (3) Next, divide them into several groups according to the type of entities identified by Spacy. Here, the entities are all location types, so they are all grouped together. (4) Extract the sentences where the entities are located and replace the entities with different [MASK] characters to construct Figure 2 The problem in the upper right corner For example, replace Arctic Norway with [MASK1], Hegelian with [MASK2], etc. (5) Figure 2 The question at the top right and the related paragraphs below The input is fed into the multi-paragraph reader FiD, and the prediction target is the entity replaced by [MASK]. To solve this task, the model first searches for context similar to the question in relevant paragraphs, and then further reconstructs the entity. The arrows indicate the information the model can refer to in relevant paragraphs to reconstruct each [MASK] character. It is thus found that the process of reconstructing the [MASK] character depends on the interaction between the question and relevant paragraphs. The model needs to find information related to the question from relevant paragraphs to solve this task. This task is very similar to the supervised multiple-answer generation task in the third stage. Therefore, the third stage of this invention can be fine-tuned based on the model trained in the second stage.

[0058] Step 103: Fine-tune on a supervised multi-paragraph, multi-answer question-and-answer dataset to define the optimal order for generating the multi-answer set. Fine-tune the multi-paragraph reader based on the model obtained in Step 102. For answer sets A and B, the size is 2. Then the size of all possible serialized text sets corresponding to them is 2, i.e., two different answer sequences, "A [SEP]B" and "B [SEP]A". For the model's predicted answer, enumerate each sequence in the set to obtain its corresponding loss. The best answer sequence is the answer sequence with the minimum loss.

[0059] In this embodiment, the maximum length of the encoder input sequence used by the reading model is set to 300. Any part exceeding this length will be removed, and any part shorter than the maximum length will be used... <pad>A padding operation was performed. The maximum decoding length was 40, and the batch size was set to 1. The learning rate was set to 0.00005, dropout to 0.1, the number of training epochs was 20, and the Adam optimizer used default parameters. The dataset used was AmbigQA, and the evaluation metric was F1. Compared with existing generative models, this invention has a superior answering metric. Table 1 shows the evaluation results of this invention and the solution model combining DPR retrieval and FiD reader only on the AmbigQA dataset, based on the T5-base model. As can be seen from Table 1, this invention has achieved a certain improvement compared to models with the same number of parameters.

[0060] Table 1

[0061] F1-all F1-multi DPR+FiD 40.8 27.5 Ours (This invention) 41.9 27.8 < / pad>

Claims

1. An implementation method for open-domain multi-answer question answering, characterized by: The process includes a dense retrieval phase, a domain-specific pre-training phase, and a supervised multi-answer generation phase. For a given question q, the system first retrieves the top K most relevant paragraphs D from the encyclopedia corpus C using a dense retrieval tool. Then, based on the domain-specific corpus, a multi-paragraph reader is constructed and pre-trained within the domain. Finally, the multi-paragraph reader is fine-tuned on a supervised open-domain multi-answer question-answering dataset. An optimal generation order for the multi-answer set is defined to reduce the impact of the order on multi-answer generation. Multiple answers are generated based on the given question and the retrieved paragraphs, as detailed below: 1) Train a dense retrieval engine. The dense retrieval engine measures the relevance of paragraphs to questions by calculating the dot product of semantic vectors encoded by paragraph encoders and question encoders, and retrieves a set of the most relevant paragraphs for a given question for subsequent reading stages. 2) The domain-specific pre-training stage achieves a transition between the general domain pre-training stage and the multi-answer generation stage by synthesizing multi-paragraph, multi-answer question-and-answer data and training it on a multi-paragraph reader. This includes constructing self-supervised data and pre-training the multi-paragraph reader. The self-supervised data consists of three parts: questions... Related paragraphs and answer The process for constructing self-supervised data is as follows: 2.1) Select a set of relevant retrieval paragraphs D from the domain corpus. rel The related search segments have similar topics, contain common entities, and the topic similarity is set by the user. 2.2) Randomly select from D rel Select k0 paragraphs as the source A for answer extraction. source The remaining paragraphs are considered as related paragraphs. 2.3) Identify A based on the Spacy tool source Entities in the list, and filter out those not in the list. Entities appearing in; 2.4) The filtered entities are divided into several groups according to the entity types identified by Spacy, and then a question-and-answer sample is constructed for each group of entities: the entities in each group are concatenated to form the answer. Each entity in A source The sentences are concatenated, and the entities are replaced with the [MASK] character to form the question. The corresponding relevant paragraphs are After constructing the self-supervised data, a generative multi-paragraph reader is obtained by modeling using the FiD architecture and then optimized and trained. 3) In the supervised multi-answer generation stage, fine-tuning of the multi-answer dataset is performed. For a given question q, the multi-paragraph reader obtains the information of the Top K paragraphs D through the dense retrieval device and predicts the answer set corresponding to question q. The answer set contains multiple different answers, which are separated by the delimiter [SEP]. Considering that there are multiple different generation orders for multiple answers, all permutations corresponding to the answer set are enumerated, and the cross-entropy loss between the answer set and the prediction of the multi-paragraph reader is obtained. The sequence of answers with the minimum loss is the best answer sequence, which is used as the final multi-answer result.

2. The method for implementing open-domain multi-answer question answering according to claim 1, characterized in that it is dense. The search engine is based on the DPR search architecture and uses the paragraph encoder E. P (·) Map paragraphs in the corpus to d-dimensional real-valued vectors, using the question encoder E Q (·) Map the question to a d-dimensional vector, and use the dot product of the vectors to define the similarity between the question and the article: i.e., sim(q,p) = q represents the question, p represents the paragraph, and the K paragraph vectors that are closest to the question vector are retrieved based on similarity. When training a dense search engine, assume The training data consists of m instances, each instance containing a question q. i and a related paragraph and n unrelated paragraphs The loss function uses InfoNCE, that is:

3. The method for implementing open-domain multi-answer question answering according to claim 2, characterized in that: In the training data of the dense retrieval system, the selection of relevant or irrelevant paragraphs is based on the dataset provided directly. For irrelevant paragraphs, two different construction methods are considered: (1) Using BM25 as the retrieval algorithm, the paragraphs that do not contain the correct answer in the open domain are used as irrelevant paragraphs; (2) Using the relevant paragraphs of other questions in the training data, the random negative paragraphs of the questions are constructed by using the negative sample method within the batch. That is, the positive paragraphs of other questions in the same batch are used as the negative paragraphs corresponding to the current question.

4. The method for implementing open-domain multi-answer question answering according to claim 1, characterized in that: The multi-paragraph reader employs a FiD architecture. It takes a question as input and the Top K retrieved paragraphs D obtained by a dense retrieval system as input. It outputs multiple answers in a fixed sequence. Each paragraph contains a title and body text. Each retrieved paragraph and its title are concatenated with the question and processed independently by the multi-paragraph reader's encoder, adding special markers before the question, title, and text of each paragraph: "Question":, "Title":, and "Context":. Finally, the multi-paragraph reader's decoder learns relevant information based on an attention mechanism by concatenating the results of all retrieved paragraphs, thus completing multi-paragraph reading comprehension.

5. The method for implementing open-domain multi-answer question answering according to claim 1, characterized in that: The determination of the optimal answer sequence is as follows: the optimal order is the order that minimizes the loss in generating the answer. Given a set of supervision labels A corresponding to a sample, with a size of |A|, then the set of all possible serialized texts corresponding to it is A. seq The size is |A|!, and A is enumerated. seq Each sequence Y in i To obtain the corresponding loss Then the optimal answer sequence Y best It is derived from the following formula: X cat =[X1;X2;…X K ] The loss is defined here as in This represents the generation loss, where K is the number of retrieved paragraphs, and X is the loss. cat It is a concatenation of multiple segment encoding representations, [;] indicates a concatenation operation, X j Let θ be the encoding vector for the j-th retrieved paragraph. multi This represents the parameters of a multi-paragraph reader.

6. An electronic device, characterized in that: It includes a storage medium and a processor, wherein the storage medium is used to store a computer program and the processor is used to execute the computer program, wherein when the computer program is executed, it implements the open-domain multi-answer question-answering method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores a computer program, which, when executed, implements the open-domain multi-answer question-answering method according to any one of claims 1-5.

Images