A machine reading comprehension method based on iterative screening and pre-training enhancement

Abstract

The present invention discloses a machine reading comprehension method based on iterative screening and pre-training enhancement, including: constructing an iterative screening network: training the initial screening network with the question and document pairs of the HotpotQA data set; finding the most relevant document from the original data containing the interference documents; iterative screening process, based on the documents obtained by the initial network and the original questions, screening out the supporting documents most relevant to the initial screening network and documents from the current document set, and obtaining two-step supporting documents related to the questions. Constructing a reader network: fine-tuning the SQuAD data with a pre-trained model to obtain a single-step reading comprehension model; performing two joint trainings on a multi-step reading comprehension data set to obtain answers and supporting facts about the questions and documents. The present invention can achieve better multi-step machine reading comprehension effects without constructing a complex graph neural network and decomposing multi-step problems.

Description

Technical Field

[0001] The invention relates to a machine reading comprehension method, in particular to a machine reading comprehension method based on iterative screening and pre-training enhancement. Background Art

[0002] In recent years, pre-training models based on large-scale unsupervised data sets have developed rapidly, greatly improving the results of many natural language processing tasks and providing sufficient source power for the development of natural language processing. At the same time, with the rapid development of the Internet, information is growing at an explosive rate. Electronic information such as encyclopedias, news, life, and books has gradually become an indispensable part of our lives. We contact the world, know the world, and understand the world from these electronic information. However, due to the rapid growth of information, it has become increasingly difficult to quickly and efficiently obtain the information we want from complex data. The difficulty of querying not only takes us a lot of time, but also may cause us to make wrong judgments due to the existence of some wrong information. Therefore, it is particularly important to use automated machine reading comprehension methods to help us automatically find the answers to the questions we want from authoritative encyclopedia electronic information.

[0003] Some existing methods are mainly aimed at single-step reading comprehension problems, that is, the questions can be answered based on a single document, but these methods are difficult to answer multi-step reasoning questions such as "the main literary works of Nobel Prize winners who once taught at Nanjing University" that require multiple documents. For multi-step reasoning question answering, the main method currently uses a document filter to filter out documents related to the current question, and then trains a reader to find answers and supporting sentences from related documents. However, this method ignores the reasoning properties between multi-step question documents and cannot fully utilize the results of single-step reasoning. At the same time, when solving multi-step reasoning problems, most methods directly use pre-trained models for training, and do not train on simple single-step reading comprehension datasets, which makes it impossible for pre-trained models to enhance their performance in the field of machine reading comprehension. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a machine reading comprehension method based on iterative screening and pre-training enhancement in view of the deficiencies in the prior art.

[0005] In order to solve the above technical problems, the present invention discloses a machine reading comprehension method based on iterative screening and pre-training enhancement, comprising the following steps:

[0006] Step 1: Use the HotpotQA dataset (a multi-step reading comprehension dataset based on Wikipedia collected by crowdsourcing) to train the relevant document model, using whether it is a supporting document as a label and using a geometric sampling strategy to obtain training positive and negative samples;

[0007] Step 2: Use the pre-trained model to fine-tune the classification network and identify the document relevance by concatenating the question and candidate documents to obtain a single-step relevant document filter.

[0008] Step 3: Use the relevant documents in the HotpotQA dataset that are not marked by the single-step relevant document filter as positive samples, and sample to obtain balanced positive and negative samples;

[0009] Step 4: Train a new pre-trained model classification network to discriminate document relevance in the form of questions, single-step related documents, and candidate documents to obtain a multi-step related document filter;

[0010] Step 5: Use the single-step extractive reading comprehension dataset SQuAD (released by Stanford University) as enhanced data for reader training to obtain a pre-trained enhanced reader;

[0011] Step 6: Use the single-step relevant document filter obtained in step 2 and the multi-step relevant document filter obtained in step 4 to obtain predicted candidate documents from the HotpotQA dataset, and then concatenate them with the questions to obtain multi-step reading comprehension training samples;

[0012] Step 7: Use the reader obtained in step 5 to perform answer extraction and supporting sentence identification multi-task learning on multi-step reading comprehension training samples to improve the efficiency and effectiveness of the reader model.

[0013] In the present invention, step 1 comprises:

[0014] According to the given data set, documents containing supporting sentences are marked as supporting documents, and documents without supporting sentences are marked as irrelevant documents; this data set is a two-step reasoning question-answering task, and the original positive and negative sample ratio is 4:1. This distribution is not conducive to model training, that is, there is a sample imbalance problem. 3 / 4 of the negative samples are randomly discarded to obtain balanced training samples.

[0015] In the present invention, step 2 comprises:

[0016] Step 2-1, vectorize the question document. During the training process, the maximum length of the question document is set to 512. Then the question document is encoded into the pre-trained model ELECTRA-large based on the generative discriminant network training (a pre-trained model based on the generative discriminant network training jointly released by Stanford University and Google), and the vector representation of the question document encoding dimension is [12,512,1024], where 12 is the batch processing size of the small batch training, 512 is the set maximum length of the training data, and 1024 is the representation dimension of a single word;

[0017] Step 2-2, use the [CLS]+question+[SEP]+document+[SEP] method, where [CLS] is the start marker of the sentence pair representation, question represents the question, [SEP] is the separator or end marker between the question and the document, and document is the document content; concatenate the input question document, and then calculate the correlation P(d|q) between each candidate document d and question q, and the loss function of the training process is The calculation process is as follows:

[0018]

[0019] Among them, q i is the i-th question in the dataset, d ij is the jth candidate document for the i-th question, l ij is the question document pair (q i ,d ij ), N is the number of questions, and M is the number of candidate documents;

[0020] Step 2-3: For a given question, obtain the single-step relevant document p according to the maximum relevant probability value i1 , the calculation process is p i1 = arg max j P(d ij |q i ).

[0021] In the present invention, step 3 comprises:

[0022] The single-step related document filter obtained in step 2 is used to filter the single-step related questions, and the related documents of non-single-step related documents are selected from the HotpotQA dataset as positive samples. The positive and negative samples are sampled to 1:1 to obtain the training samples required for training multi-step related documents.

[0023] In the present invention, step 4 comprises:

[0024] Step 4-1: Substitute the problem and the p obtained in step 2 into i1Join with candidate documents according to [CLS]+question+[SEP]+document 1 +[SEP]+document 2 +[SEP] are sequentially input into the pre-trained model to obtain a vector representation, where [CLS] is the start mark of the sentence pair representation, question represents the question, [SEP] is the separator or end mark between the question and the document, and document 1 and document 2 is the document content; where document 1 The document filtered by the single-step document filter, document 2 is a candidate document; [CLS] and [SEP] are both special connectives. In this process, ELECTRA-large is used as a pre-trained model to obtain a document question representation with an encoding dimension of [12,512,1024];

[0025] Step 4-2, use the [CLS] sentence pair representation of the pre-trained model in step 4-1 to start the mark c as the overall representation of the question and document, the representation dimension is [12,1024], where 12 is the batch size and 1024 is the vector representation dimension; input the representation c into a linear mapping layer, mapped to a binary representation logit, the representation dimension is [12,2], where 2 represents the probability value of the relevant probability and the irrelevant probability, respectively, and compress any dimensional vector into another vector of the same dimension through the normalized exponential function softmax, and each element ranges between 0 and 1, and the sum is 1;

[0026] Step 4-3, during the training process, calculate the loss function L based on whether the candidate document is relevant s2 , the calculation method is as follows:

[0027]

[0028] Among them, N is the number of questions, M is the number of candidate documents, and I is the indicator function, which determines whether the current document is the same as the single-step document screening result. ij is the label of the jth candidate document for the i-th question, d ij is the jth candidate document corresponding to the i-th question, p i1 For the single-step document screening results, the whole process is to calculate the cross entropy loss;

[0029] Step 4-4: For a given question, obtain the single-step relevant document p according to the maximum relevant probability value i2 , the calculation process is p i2 = arg max j P(dij |q i ).

[0030] In the present invention, step 5 comprises the following steps:

[0031] Step 5-1, filter out data with answers in the training set of the single-step reading comprehension dataset SQuAD to obtain single-step reading comprehension data; at the same time, consider the sentences containing the start and end positions of the answers as supporting sentences;

[0032] Step 5-2, concatenate the question document in the form of [CLS]+yes+no+context+[SEP]+question+[SEP], where [CLS] is the start mark of the sentence pair output, context is the document content, yes and no are the answer marks to the yes / no question, question represents the question, and [SEP] is the separator or end mark between the question and the document; at the same time, add [UNK] before each sentence to represent the subsequent sentences; input the concatenated question document into the pre-trained model Albert-xxlarge-v2 (a pre-trained model trained using matrix decomposition, parameter sharing, and sentence pair order jointly released by Google and Toyota Technological University Chicago) to obtain the overall vector representation H, the dimension of H is [12,512,4096], where 12 is the batch size, 512 is the input sequence length, and 4096 is the vector representation dimension;

[0033] Step 5-3, the obtained overall vector representation H is divided into question representation and document representation, and input into two cross-attention modules for feature enhancement to obtain the representation of the document and question; the calculation method is as follows:

[0034] Q=H c W Q

[0035] K=H q W K

[0036] V=H q W V

[0037] CrissAttebtion=Attention(Q,K,V)

[0038]

[0039] Among them, Q, K, V are matrices of attention operation, H c is the document representation, H q is the problem representation, W Q ,W K , W Vis a trainable parameter, CrossAttention is the calculation method used in the present invention, Attention is a matrix calculation formula, softmax is a normalized exponential function, which compresses a vector of any dimension into another vector of the same dimension, and each element ranges from 0 to 1, and the sum is 1, d k is the hidden layer vector dimension of the document and question representation, and T is the transposition operation; two different cross-attention models are used in the calculation process, one of which uses the layer normalization layer and the other does not; the two representations are concatenated by adding and summing to obtain the final document representation;

[0040] Step 5-4: Input the document representation into a linear layer to obtain the probability of the start and end positions of the answer. At the same time, use the [UNK] representation before each sentence to obtain the probability that each sentence is a supporting sentence. start 、The answer ends at the position loss L end and support sentence loss L sf The single-step reading comprehension model L is obtained by joint training as follows:

[0041] L=γ 1 L start +γ 2 L end +γ 3 L sf

[0042] Among them, γ 1 , γ 2 , γ 3 are pre-set hyperparameters.

[0043] In the present invention, step 6 comprises:

[0044] Use the single-step relevant document filter obtained in step 2 and the multi-step relevant document filter obtained in step 4 to obtain predicted candidate documents from the HotpotQA dataset, concatenate them with the questions to obtain multi-step reading comprehension training samples, remove other irrelevant candidate documents, obtain noise-free multi-hop documents, and reduce the interference of irrelevant documents on subsequent answer extraction and supporting sentence extraction.

[0045] In the present invention, step 7 comprises the following steps:

[0046] Step 7-1, concatenate the candidate documents and questions obtained in step 6 in the form of [CLS]+yes+no+context+[SEP]+question+[SEP], and add [UNK] before each sentence for subsequent representation; the context includes the relevant documents filtered out by the single-step machine reading comprehension filter and the multi-step reading comprehension filter; input into the single-step reading comprehension model obtained in step 5 to obtain the vector representation H, the dimension of H is [12,512,4096], where 12 is the batch size, 512 is the input sequence length, and 4096 is the vector representation dimension. In this process, the multi-step reading comprehension data set contains yes or no answers, and the yes-or-no questions can be directly answered based on the special words yes and no;

[0047] Step 7-2: The obtained overall vector representation H is divided into question representation and document representation, and input into two cross-attention modules for feature enhancement using the same processing method as the pre-trained enhanced model; the calculation method is as follows:

[0048] Q=H c W Q

[0049] K=H q W K

[0050] V=H q W V

[0051] CrossAttention=Attention(Q,K,V)

[0052]

[0053] Among them, Q, K, V are matrices of attention operation, H c is the document representation, H q is the problem representation, W Q ,W K , W V is a trainable parameter, CrossAttention is the calculation method used in the present invention, Attention is a matrix calculation formula, softmax is a normalized exponential function, which compresses a vector of any dimension into another vector of the same dimension, and each element ranges from 0 to 1, and the sum is 1, d kis the hidden layer vector dimension of the document and question representation, and T is the transposition operation. Two different cross-attention models are used in this calculation process, one of which uses a layer normalization layer and the other does not. The two representations are concatenated by summing, and the fast convergence of the layer normalization layer and the randomness of the layer-free normalization layer are used to converge to the global optimal point to obtain the final document representation.

[0054] Step 7-3, the method of extracting answers and supporting sentences includes: inputting the document representation into a linear layer, obtaining the probability of the start position and the end position of the answer, and using the [UNK] representation before each sentence to obtain the probability that each sentence is a supporting sentence, and losing the answer start position L start 、The answer ends at the position loss L end and support sentence loss L sf The multi-step reading comprehension model L is obtained by joint training as follows:

[0055] L=γ 1 L start +γ 2 L end +γ 3 L sf

[0056] Among them, γ 1 , γ 2 , γ 3 are pre-set hyperparameters.

[0057] The HotpotQA dataset described in step 1 of the present invention includes a question-answering dataset based on Wikipedia collected by crowdsourcing; each sample in the dataset provides a question and 10 documents with interference, and the training set and the validation set provide answers and supporting sentences supporting multi-step reasoning.

[0058] The pre-set hyperparameters in step 5-4 and step 7-3 of the present invention, γ 1 is 1, γ 2 is 1, γ 3 is 0.2.

[0059] Beneficial effects:

[0060] Using machine reading comprehension models, once trained on a large data set, they can automatically help people find appropriate information from massive amounts of information quickly and efficiently, intelligently solve question-and-answer problems, make rational use of massive amounts of information, and improve each user's experience in using electronic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0062] Figure 1 It is a schematic diagram of the workflow of the present invention.

[0063] Figure 2 Schematic diagram of a multi-hop understanding data instance in HotpotQA of the method of the present invention.

[0064] Figure 3 It is a schematic diagram of the overall model processing flow of the method of the present invention.

[0065] Figure 4 It is a network structure diagram of the single-step related document filter in the present invention.

[0066] Figure 5 It is a network structure diagram of the multi-step related document filter in the present invention.

[0067] Figure 6 It is a network structure diagram of the multi-step reader in the present invention.

[0068] Figure 7 It is a schematic diagram of the automated prediction results of the method of the present invention. DETAILED DESCRIPTION

[0069] The present invention discloses a machine reading comprehension method based on iterative screening and pre-training enhancement. The method is applied to the need to use automated means to improve the question-answering effect and enhance the user's search or language question-answering experience.

[0070] like Figure 1 As shown, the present invention provides a machine reading comprehension method based on iterative screening and pre-training enhancement, comprising the following steps:

[0071] Step 1: Use the HotpotQA dataset (reference: Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering [C]. In EMNLP, 2018) to train the relevant document model, using whether it is a supporting document as a label and using geometric sampling to obtain training positive and negative samples;

[0072] Step 2: Use the pre-trained model to fine-tune the classification network and identify the document relevance by concatenating the question and candidate documents to obtain a single-step relevant document filter.

[0073] Step 3: Use the relevant documents in the HotpotQA dataset that are not marked by the single-step relevant document filter as positive samples, and sample to obtain balanced positive and negative samples;

[0074] Step 4: Train a new pre-trained model classification network to discriminate document relevance in the form of questions, single-step related documents, and candidate documents to obtain a multi-step related document filter;

[0075] Step 5: Use the single-step reading comprehension dataset SQuAD (reference: Pranav Rajpurkar, Jian Zhang, Kon-stantin Lopyrev and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text [C]. In EMNLP, 2016.) as enhanced data for reader training to obtain a pre-trained enhanced single-step reading reader with excellent performance;

[0076] Step 6: Use the document filter obtained in Step 2 and Step 4 to obtain the predicted candidate documents from the HotpotQA dataset, and then concatenate them with the questions to obtain multi-step reading comprehension training samples;

[0077] Step 7: Use the reader obtained in step 5 to perform answer extraction and supporting sentence identification multi-task learning on multi-step reading comprehension training samples to improve model efficiency and effectiveness.

[0078] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0079] In one embodiment of the present invention, according to Figure 3 As shown, it is divided into three stages:

[0080] The first stage is the initial screening stage. The single-step document filter is trained, which includes marking documents as relevant documents and irrelevant documents from the HotpotQA dataset based on whether they contain supporting sentences, and then obtaining a balanced sample set by geometric sampling, such as Figure 2The reading comprehension question shown in the figure aims to find the supporting facts and the final answer related to the question "In which year did the Italian physicist who wrote "Seven Brief Lessons on Physics" start working in France" from the document shown in the figure. Seven Brief Lesson On Physics and Carlo Rovelli are the relevant documents containing supporting facts, among which Seven Brief Lesson On Physics is a descriptive article of "Seven Brief Lessons on Physics", which can help find the author of the book; Carlo Rovelli is a descriptive article of the author of "Seven Brief Lessons on Physics", introducing the author's work experience. Then enter the question documents in the order of "[CLS]+question+[SEP]+document+[SEP]" into the following table: Figure 4 The sentence pair classification network model based on the pre-trained model is trained, and the entire sentence pair is represented by [CLS] and then used to determine whether the document is related to the question. During the training process, the learning rate is set to 1e-5, the batch size is set to 12, and then the BERT-Adam optimizer (reference: Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding [C]. In NAACL-HLT, 2019) is used for optimization. At the same time, the cross entropy is used as the loss function to train the model. The whole process only needs two epochs to finally obtain the single-step document filter network S1.

[0081] The second stage, the iterative screening stage, trains a multi-step document filter, including using different supporting documents from HotpotQA as positive samples from the first stage single-step document filter, and then other documents as negative samples. Next, follow the “[CLS]+question+[SEP]+document 1 +[SEP]+document 2 +[SEP]" in the order of Figure 5 The sentence pair classification network model based on the pre-trained model is trained, and [CLS] is used to represent the entire sentence pair and then used to determine whether the document is related to the question and the single-step related document. The training parameter configuration is the same as the single-step document filter in the first stage, and finally a multi-step document filter network S2 is obtained.

[0082] The third stage is the reader training stage, which involves training a multi-step document reader. This includes constructing and training a single-step reading comprehension dataset from the single-step reading comprehension dataset SQuAD. Then, based on the single-step document filter network S1 and the multi-step document filter network S2, relevant paragraphs are filtered out from HotpotQA to construct a multi-step document reading comprehension dataset. Then, documents and questions are input in the order of “[CLS]+yes+no+context+[SEP]+question+[SEP]” as follows: Figure 6 The training is performed in the pointer network model based on the pre-trained model shown in the figure. Specifically, the pre-trained model is first used to obtain the representation of the question and the document, and then it is input into the cross-attention module to better represent the interaction between the document and the question. Finally, the document representation is extracted to extract the answer and supporting facts, where the answer is extracted using the probability values ​​of the start and end positions predicted by the model, and the supporting facts are extracted based on the representation probability values ​​of each sentence. The training parameter configuration is the same as the first-stage single-step document filter, but the pre-trained model used in the training process is Albert-xxlarge-v2 (reference: Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations [C]. In ICLR 2020), then we can get the multi-step reader R, and finally using this multi-step reading comprehension, we can automatically get the following Figure 7 The machine reading comprehension results shown in the figure, where ID represents the unique identifier of the data in the HotpotQA dataset, the predicted answer and predicted supporting sentence are the output results of the method of this embodiment, and the actual answer and actual supporting sentence are the annotation results of the original dataset. The answer content includes the yes and no type answers as shown in the figure and the result "Animorphs" extracted from the document, etc. The supporting sentence content format is a combination of document name and sentence index as shown in ["Animorphs", 0].

[0083] In the machine reading comprehension method based on iterative screening and pre-training enhancement described in this embodiment, step 1 includes:

[0084] The training dataset described in the present invention is HotpotQA, which includes 11.3w question-answering datasets based on Wikipedia collected by crowdsourcing. It contains 90,564 training samples, 7,405 verification samples and 7,405 test samples. In this paper, experiments are mainly conducted on the disturbed item dataset in HotpotQA. Each sample in the dataset provides a question and 10 documents with interference, and the training set and verification set provide answers and supporting sentences for multi-step reasoning.

[0085] According to the given data set, documents containing supporting sentences are marked as supporting documents, and documents without supporting sentences are marked as irrelevant documents. Since the data set is a two-step reasoning question-answering task, the original positive and negative sample ratio is 4:1. This distribution is not conducive to model training, so 3 / 4 of the negative samples are randomly discarded to obtain balanced training samples;

[0086] In the machine reading comprehension method based on iterative screening and pre-training enhancement described in this embodiment, step 2 includes:

[0087] Step 2-1, vectorize the question document. During the training process, the maximum length of the question document is set to 512, and then the question document pair is input into ELECTRA-large (reference: Kevin Clark, Minh-ThangLuong, Quoc V.Le and Christopher D.Manning. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators [C]. In ICLR2020.) for encoding to obtain a vector representation of the question document encoding dimension of [12, 512, 1024], where 12 is the batch processing size of the small batch training, 512 is the set maximum length of the training data, and 1024 is the representation dimension of a single word;

[0088] Step 2-2, use the method of "[CLS]+question+[SEP]+document+[SEP]" to concatenate the input question documents, and then calculate the correlation P(d|q) between each candidate document d and question q. The calculation process of the loss function in the training process is as follows:

[0089]

[0090] Among them, q i is the i-th question in the dataset, d ij is the jth candidate document for the i-th question, l ij is the question document pair (q i ,dij ), N is the number of questions, and M is the number of candidate documents;

[0091] Step 2-3: For a given question, obtain the single-step relevant document p according to the maximum relevant probability value i1 , the calculation process is p i1 = arg maxP(d ij |q i ) j .

[0092] In the machine reading comprehension method based on iterative screening and pre-training enhancement described in this embodiment, step 3 includes:

[0093] The single-step related document filter obtained in step 2 is used to filter the single-step related questions, and then the related documents of non-single-step related documents are selected from the HotpotQA dataset as positive samples. Then the positive and negative samples are sampled to 1:1 to obtain the training samples required for training multi-step related documents.

[0094] In the machine reading comprehension method based on iterative screening and pre-training enhancement described in this embodiment, step 4 includes:

[0095] Step 4-1: Substitute the problem and the p obtained in step 2 i1 Join with candidate documents according to "[CLS]+question+[SEP]+document 1 +[SEP]+document 2 +[SEP]" is sequentially input into the pre-trained model to obtain a vector representation, where document 1 The document filtered by the single-step document filter, document 2 is a candidate document. [CLS] and [SEP] are both special conjunctions. In this process, ELECTRA-large is used as a pre-trained model to obtain the document question representation of [12,512,1024];

[0096] Step 4-2: Use the [CLS] representation c obtained in step 4-1 as the overall representation of the question and document. The representation dimension is [12, 1024], where 12 is the batch size and 1024 is the vector representation dimension. Then input it to a linear mapping layer and map it to a binary representation logit. The representation dimension is [12, 2], where 2 represents the probability value of the relevant probability and the irrelevant probability respectively. Then, the probability value with the sum of 1 is obtained through softmax.

[0097] Step 4-3, during the training process, the loss function is calculated based on whether the candidate documents are relevant. The specific calculation formula is as follows:

[0098]

[0099] Where N is the number of questions, M is the number of candidate documents, and I is the indicator function, which determines whether the current document is the same as the single-step document screening result. ij is the label of the jth candidate document for the i-th question, d ij is the jth candidate document corresponding to the i-th question, p i1 For the single-step document screening results, the whole process is to calculate the cross entropy loss.

[0100] Step 4-4: For a given question, obtain the single-step relevant document p according to the maximum relevant probability value i2 , the calculation process is p i2 = arg max j P(d ij |q i ).

[0101] In the machine reading comprehension method based on iterative screening and pre-training enhancement described in this embodiment, step 5 includes:

[0102] Step 5-1: Filter out data with answers in the training set of the single-step reading comprehension dataset SQuAD, and obtain 86,821 single-step reading comprehension data. At the same time, the sentences containing the start and end positions of the answers are regarded as supporting sentences;

[0103] Step 5-2, concatenate the question document in the form of "[CLS]+yes+no+context+[SEP]+question+[SEP]", and add [UNK] before each sentence for subsequent representation. Then input it into the pre-trained model Albert-xxlarge-v2 to obtain the vector representation H, the dimension of H is [12,512,4096], where 12 is the batch size, 512 is the input sequence length, and 4096 is the vector representation dimension.

[0104] Step 5-3: In order to better obtain the representation of documents and questions, the overall vector representation is divided into question representation and document representation, and then input into two cross-attention modules for feature enhancement. The calculation method is as follows:

[0105] Q=H c W Q

[0106] K=H q W K

[0107] V=H q W V

[0108] CrossAttenttion=Attention(Q,K,V)

[0109]

[0110] Among them, Q, K, V are matrices of attention operation, H c is the document representation, H q is the problem representation, W Q ,W K , W V is a trainable parameter, CrossAttention is the calculation method used in this embodiment, Attention is a matrix calculation formula, softmax is a normalized exponential function, which compresses a vector of any dimension into another vector of the same dimension, and each element ranges from 0 to 1, and the sum is 1, d k is the hidden layer vector dimension of the document and question representation, and T is the transposition operation. In this process, in order to obtain better results, two different cross-attention models are used, one of which uses the layer normalization layer and the other does not, and then the two representations are concatenated by adding and summing to obtain the final document representation;

[0111] Step 5-4: In order to extract the answer and supporting sentences, the document representation is input into a linear layer to obtain the probability of the start and end positions of the answer. At the same time, the [UNK] representation before each sentence is used to obtain the probability that each sentence is a supporting sentence. Then the answer start position loss L is used. start 、The answer ends at the position loss L end and support sentence loss L sf The single-step reading comprehension model is obtained by joint training according to the following formula:

[0112] L=γ 1 L start +γ 2 L end +γ 3 L sf

[0113] where γ 1 , γ 2 , γ 3 is a pre-set hyperparameter, which is set to 1, 1, and 0.2 in this method.

[0114] In the machine reading comprehension method based on iterative screening and pre-training enhancement described in this embodiment, step 6 includes:

[0115] Use the document filter obtained in steps 2 and 4 to obtain predicted candidate documents from the HotpotQA dataset, and then concatenate them with the questions to obtain multi-step reading comprehension training samples. Remove other irrelevant candidate documents to obtain noise-free multi-hop documents, reducing the interference of irrelevant documents on subsequent answer extraction and supporting sentence extraction.

[0116] In the machine reading comprehension method based on iterative screening and pre-training enhancement described in this embodiment, step 7 includes:

[0117] Step 7-1, concatenate the relevant documents and questions obtained in step 6 in the manner of "[CLS]+yes+no+context+[SEP]+question+[SEP]", and add [UNK] before each sentence for subsequent representation. The context contains the relevant documents filtered out by the single-step machine reading comprehension filter and the multi-step reading comprehension filter. Then input it into the single-step reading comprehension model obtained in step 5 to obtain the vector representation H. The dimension of H is [12,512,4096], where 12 is the batch size, 512 is the input sequence length, and 4096 is the vector representation dimension. The difference between this process and step 5-2 is that the multi-step reading comprehension dataset contains yes / no answers, and the yes-no questions can be answered directly based on the special words yes and no.

[0118] Step 7-2 is the same as step 5-3. In order to better obtain the representation of documents and questions, the overall vector representation is divided into question representation and document representation. The same processing method as the pre-trained enhanced model is used, specifically input into two cross-attention modules for feature enhancement. The calculation method is as follows:

[0119] Q=H c W Q

[0120] K=H q W K

[0121] V=H q W V

[0122] CrossAttention=Attention(Q,K,V)

[0123]

[0124] Among them, Q, K, V are matrices of attention operation, H c is the document representation, H q is the problem representation, W Q ,W K , WV is a trainable parameter, CrossAttention is the calculation method used in this embodiment, Attention is a matrix calculation formula, softmax is a normalized exponential function, which compresses a vector of any dimension into another vector of the same dimension, and each element ranges from 0 to 1, and the sum is 1, d k is the hidden layer vector dimension of the document and question representation, and T is the transposition operation. In this process, in order to obtain better results, two different cross-attention models are used, one of which uses the layer normalization layer and the other does not, and then the two representations are concatenated by summing. In this way, the fast convergence of the layer normalization layer and the randomness of the layer-free normalization layer can be used to better converge to the global optimal point to obtain the final document representation;

[0125] Step 7-3: In order to extract the answer and supporting sentences, the document representation is input into a linear layer to obtain the probability of the start and end positions of the answer. At the same time, the [UNK] representation before each sentence is used to obtain the probability that each sentence is a supporting sentence. Then the answer start position loss L is used. start 、The answer ends at the position loss L end and support sentence loss L sf The multi-step reading comprehension model is obtained by joint training according to the following formula:

[0126] L=γ 1 L start +γ 2 L end +γ 3 L sf

[0127] where γ 1 , γ 2 , γ 3 is a pre-set hyperparameter. In this method, the settings are the same as step 5-4, which are 1, 1, and 0.2. After using the machine reading comprehension method, we can automatically and efficiently obtain the information we want.

[0128] With the rapid development of the Internet, information is growing at an explosive rate. Electronic information such as encyclopedias, news, life, and books have become an indispensable part of our lives. We contact the world, know the world, and understand the world from these electronic information. However, due to the rapid growth of information, it has become increasingly difficult to quickly and efficiently obtain the information we want from complex data. The difficulty of querying not only takes us a lot of time, but also may cause us to make wrong judgments due to the existence of some wrong information. Therefore, it is particularly important to use automated machine reading comprehension methods to help us automatically find the answers to the questions we want from authoritative encyclopedia electronic information. Some existing methods are mainly aimed at single-step reading comprehension problems, that is, the questions can be answered based on only a single document, but these methods are difficult to answer multi-step reasoning problems that require multiple documents, such as "The main literary works of Nobel Prize winners who taught at Nanjing University". For multi-step reasoning question answering, the current main method uses a document filter to filter out documents related to the current question, and then trains a reader to find answers and supporting sentences from the relevant documents. However, this method ignores the reasoning properties between multi-step question documents and cannot fully utilize the results of single-step reasoning. At the same time, when solving multi-step reasoning problems, most methods directly use pre-trained models for training, and do not train on simple single-step reading comprehension datasets, which makes it impossible for the pre-trained models to enhance their performance in the field of machine reading comprehension.

[0129] In a specific implementation, the present invention further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of a machine reading comprehension method based on iterative screening and pre-training enhancement provided by the present invention. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0130] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0131] The same and similar parts between the various embodiments in this specification can be referred to each other. The above-mentioned embodiments of the present invention do not constitute a limitation on the protection scope of the present invention.

[0132] The present invention provides a method and idea of ​​a method for machine reading comprehension based on iterative screening and pre-training enhancement. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A machine reading comprehension method based on iterative screening and pre-training enhancement, characterized in that: The following steps are involved: Step 1: Use the HotpotQA dataset to train the relevant document model, use whether it is a supporting document as a label and use a geometric sampling strategy to obtain training positive and negative samples; Step 2: Use the pre-trained model to fine-tune the classification network and identify the document relevance by concatenating the question and candidate documents to obtain a single-step relevant document filter. Step 3: Use the relevant documents in the HotpotQA dataset that are not marked by the single-step relevant document filter as positive samples, and sample to obtain balanced positive and negative samples; Step 4: Train a new pre-trained model classification network to discriminate document relevance in the form of questions, single-step related documents, and candidate documents to obtain a multi-step related document filter; Step 5, use the single-step extractive reading comprehension dataset SQuAD as enhanced data to train the reader and obtain a pre-trained enhanced reader; Step 6: Use the single-step relevant document filter obtained in step 2 and the multi-step relevant document filter obtained in step 4 to obtain predicted candidate documents from the HotpotQA dataset, and then concatenate them with the questions to obtain multi-step reading comprehension training samples; Step 7: Use the reader obtained in step 5 to perform answer extraction and supporting sentence discrimination multi-task learning on multi-step reading comprehension training samples to improve the efficiency and effect of the reader model; Wherein, step 2 comprises: Step 2-1, vectorize the question document. During the training process, the maximum length of the question document is set to 512. Then, the question document is input into the pre-trained model ELECTRA-large based on the generative discriminant network training to encode it, and the vector representation of the question document encoding dimension is [12,512,1024], where 12 is the batch processing size of the small batch training, 512 is the set maximum length of the training data, and 1024 is the representation dimension of a single word; Step 2-2, use the [CLS]+question+[SEP]+document+[SEP] method, where [CLS] is the start marker of the sentence pair representation, question represents the question, [SEP] is the separator or end marker between the question and the document, and document is the document content; concatenate the input question document, and then calculate the correlation P(d|q) between each candidate document d and question q, and the training process loss function L s1 , the calculation process is as follows: Among them, q i is the i-th question in the dataset, d ij is the jth candidate document for the i-th question, l ij is the question document pair (q i ,d ij ), N is the number of questions, and M is the number of candidate documents; Step 2-3: For a given question, obtain the single-step relevant document p according to the maximum relevant probability value i1 , the calculation process is p i1 = argmax j P(d ij |q i ); Step 4 includes: Step 4-1: Substitute the problem and the p obtained in step 2 into i1 The candidate documents are concatenated and input into the pre-trained model in the order of [CLS]+question+[SEP]+document1+[SEP]+document2+[SEP] to obtain vector representation, where [CLS] is the start mark of the sentence pair representation, question represents the question, [SEP] is the separator or end mark between the question and the document, and document1 and document2 are the document contents; document1 is the document filtered out by the single-step document filter, and document2 is the candidate document; [CLS] and [SEP] are both special conjunctions. In this process, ELECTRA-large is used as the pre-trained model to obtain the document question representation with an encoding dimension of [12,512,1024]; Step 4-2, use the [CLS] sentence pair representation of the pre-trained model in step 4-1 to start the mark c as the overall representation of the question and document, the representation dimension is [12,1024], where 12 is the batch size and 1024 is the vector representation dimension; input the representation c into a linear mapping layer, mapped to a binary representation logit, the representation dimension is [12,2], where 2 represents the probability value of the relevant probability and the irrelevant probability, respectively, and compress any dimensional vector into another vector of the same dimension through the normalized exponential function softmax, and each element ranges between 0 and 1, and the sum is 1; Step 4-3, during the training process, calculate the loss function L based on whether the candidate document is relevant s2 , the calculation method is as follows: Among them, N is the number of questions, M is the number of candidate documents, and I is the indicator function, which determines whether the current document is the same as the single-step document screening result. ij is the label of the jth candidate document for the i-th question, d ij is the jth candidate document corresponding to the i-th question, p i1 For the single-step document screening results, the whole process is to calculate the cross entropy loss; Step 4-4: For a given question, obtain the single-step relevant document p according to the maximum relevant probability value i2 , the calculation process is p i2 = argmax j P(d ij |q i ).

2. A machine reading comprehension method based on iterative screening and pre-training enhancement according to claim 1, characterized in that: Step 1 includes: According to the given data set, documents containing supporting sentences are marked as supporting documents, and documents without supporting sentences are marked as irrelevant documents; this data set is a two-step reasoning question-answering task, and the original positive and negative sample ratio is 4:

1. This distribution is not conducive to model training, that is, there is a sample imbalance problem. 3 / 4 of the negative samples are randomly discarded to obtain balanced training samples.

3. A machine reading comprehension method based on iterative screening and pre-training enhancement according to claim 2, characterized in that: Step 3 includes: The single-step related document filter obtained in step 2 is used to filter the single-step related questions, and the related documents of non-single-step related documents are selected from the HotpotQA dataset as positive samples. The positive and negative samples are sampled to 1:1 to obtain the training samples required for training multi-step related documents.

4. A machine reading comprehension method based on iterative screening and pre-training enhancement according to claim 3, characterized in that: Step 5 includes the following steps: Step 5-1, filter out data with answers in the training set of the single-step reading comprehension dataset SQuAD to obtain single-step reading comprehension data; at the same time, consider the sentences containing the start and end positions of the answers as supporting sentences; Step 5-2, concatenate the question document in the form of [CLS]+yes+no+context+[SEP]+question+[SEP], where [CLS] is the start mark of the sentence pair output, context is the document content, yes and no are the answer marks to the yes / no question, question represents the question, and [SEP] is the separator or end mark between the question and the document; at the same time, add [UNK] before each sentence to represent the subsequent sentences; input the concatenated question document into the pre-trained model Albert-xxlarge-v2 to obtain the overall vector representation H, the dimension of H is [12,512,4096], where 12 is the batch size, 512 is the input sequence length, and 4096 is the vector representation dimension; Step 5-3, the obtained overall vector representation H is divided into question representation and document representation, and input into two cross-attention modules for feature enhancement to obtain the representation of the document and question; the calculation method is as follows: Q=H c W Q K=H q W K V=H q W V CrossAttention=Attention(Q,K,V) Among them, Q, K, V are matrices of attention operation, H c is the document representation, H q is the problem representation, W Q ,W K , W V is a trainable parameter, CrossAttention is the calculation method used in the present invention, Attention is a matrix calculation formula, softmax is a normalized exponential function, which compresses a vector of any dimension into another vector of the same dimension, and each element ranges from 0 to 1, and the sum is 1, d k is the hidden layer vector dimension of the document and question representation, and T is the transposition operation; two different cross-attention models are used in the calculation process, one of which uses the layer normalization layer and the other does not; the two representations are concatenated by adding and summing to obtain the final document representation; Step 5-4: Input the document representation into a linear layer to obtain the probability of the start and end positions of the answer. At the same time, use the [UNK] representation before each sentence to obtain the probability that each sentence is a supporting sentence. start 、The answer ends at the position loss L end and support sentence loss L sf The single-step reading comprehension model L is obtained by joint training as follows: L=γ1L start +γ2L end +γ3L sf Among them, γ1, γ2, and γ3 are pre-set hyperparameters.

5. A machine reading comprehension method based on iterative screening and pre-training enhancement according to claim 4, characterized in that: Step 6 includes: Use the single-step relevant document filter obtained in step 2 and the multi-step relevant document filter obtained in step 4 to obtain predicted candidate documents from the HotpotQA dataset, concatenate them with the questions to obtain multi-step reading comprehension training samples, remove other irrelevant candidate documents, obtain noise-free multi-hop documents, and reduce the interference of irrelevant documents on subsequent answer extraction and supporting sentence extraction.

6. A machine reading comprehension method based on iterative screening and pre-training enhancement according to claim 5, characterized in that: The step 7 comprises the following steps: Step 7-1, concatenate the candidate documents and questions obtained in step 6 in the form of [CLS]+yes+no+context+[SEP]+question+[SEP], and add [UNK] before each sentence for subsequent representation; the context includes the relevant documents filtered out by the single-step machine reading comprehension filter and the multi-step reading comprehension filter; input into the single-step reading comprehension model obtained in step 5 to obtain the vector representation H, the dimension of H is [12,512,4096], where 12 is the batch size, 512 is the input sequence length, and 4096 is the vector representation dimension. In this process, the multi-step reading comprehension data set contains yes or no answers, and the yes-or-no questions can be directly answered based on the special words yes and no; Step 7-2: The obtained overall vector representation H is divided into question representation and document representation, and input into two cross-attention modules for feature enhancement using the same processing method as the pre-trained enhanced model; The calculation method is as follows: Q=H c W Q K=H q W K V=H q W V CrossAttention=Attention(Q,K,V) Among them, Q, K, V are matrices of attention operation, H c is the document representation, H q is the problem representation, W Q ,W K , W V is a trainable parameter, CrossAttention is the calculation method used in the present invention, Attention is a matrix calculation formula, softmax is a normalized exponential function, which compresses a vector of any dimension into another vector of the same dimension, and each element ranges from 0 to 1, and the sum is 1, d k is the hidden layer vector dimension of the document and question representation, and T is the transposition operation. Two different cross-attention models are used in this calculation process, one of which uses a layer normalization layer and the other does not. The two representations are concatenated by summing, and the fast convergence of the layer normalization layer and the randomness of the layer-free normalization layer are used to converge to the global optimal point to obtain the final document representation. Step 7-3, the method of extracting answers and supporting sentences includes: inputting the document representation into a linear layer, obtaining the probability of the start position and the end position of the answer, and using the [UNK] representation before each sentence to obtain the probability that each sentence is a supporting sentence, and losing the answer start position L start 、The answer ends at the position loss L end and support sentence loss L sf The multi-step reading comprehension model L is obtained by joint training as follows: L=γ1L start +γ2L end +γ3L sf Among them, γ1, γ2, and γ3 are pre-set hyperparameters.

7. A machine reading comprehension method based on iterative screening and pre-training enhancement according to claim 6, characterized in that: The HotpotQA dataset described in step 1 includes a question-answering dataset based on Wikipedia collected by crowdsourcing; each sample in the dataset provides a question and 10 documents with interference, while the training set and the validation set provide answers and supporting sentences supporting multi-step reasoning.

8. A machine reading comprehension method based on iterative screening and pre-training enhancement according to claim 7, characterized in that: The pre-set hyperparameters in step 5-4 and step 7-3 are γ1, γ2, and γ3, which are 1, 1, and 0.2.

Images