A method for automatic extraction of evidence sentences based on meta-erasure

Abstract

The present invention relates to an automatic extraction method of evidence sentences based on meta-erasure, and belongs to the technical field of machine reading comprehension in natural language processing. This method does not require manual annotation. By erasing at the sentence level, the evidence sentences are selected according to the contribution of each sentence to the answer, and the automatic selection of evidence sentences is realized. The selection of evidence sentences can be evaluated according to the magnitude of the improvement effect, so that the multiple-choice reading comprehension data set can explain the model behavior based on the evidence sentences. Compared with the existing technology, this method can save a lot of time and manpower in the extraction and annotation of evidence sentences, and obtain good results.

Description

Technical Field

[0001] The present invention relates to a method for automatically extracting evidence sentences, and in particular to an automatic extraction method based on meta-erasure for selecting evidence sentences and extracting multiple evidence sentences for reading comprehension, belonging to the technical field of machine reading comprehension in natural language processing. Background Art

[0002] In recent years, with the introduction of deep neural network models, machine reading comprehension technology has made significant progress and has been widely used in text question and answer, information extraction, machine translation and many other aspects.

[0003] Machine reading comprehension can be divided into four types according to the task form: cloze, multiple choice, span prediction, and free question-answering. Among them, multiple choice machine reading comprehension is to select the correct answer from multiple candidate answers given the context and question, which requires the model to have a deep understanding of the text content and make inferences. Compared with cloze and span prediction types, multiple choice has higher requirements for model reasoning and can better measure the model's understanding of the text.

[0004] At the same time, the opacity of deep models grows as their parameters and layers increase, which has prompted people to work hard to explain how these "black box" models work. Therefore, in order to explain questions such as "how these models select the correct answer" and "whether the correct answer is selected based on the correct basis", some works manually annotate the evidence sentences in the article so that the model can predict the corresponding evidence sentence while predicting the correct answer.

[0005] Although manually annotated evidence sentences have helped to improve model interpretability, manual data annotation consumes a lot of time and manpower, resulting in few datasets with fully annotated evidence sentences. Although in span prediction tasks, evidence sentences can be annotated according to the position of the answer in the article, this approach is obviously not feasible for tasks based on multi-sentence reasoning and multiple-choice tasks. How to automatically annotate evidence sentences or automatically select evidence sentences has become a technical problem that urgently needs to be solved. Summary of the invention

[0006] The purpose of the present invention is to address the problem that the existing manual method of annotating evidence sentences is time-consuming and labor-intensive, resulting in the inability to annotate evidence sentences for most existing multiple-choice data sets. In order to achieve automatic annotation of evidence sentences so that the multiple-choice reading comprehension data set can explain the model behavior based on evidence sentences, a method for automatic extraction of evidence sentences based on meta-erasure is creatively proposed.

[0007] The innovation of this method is that it does not require manual labeling. By erasing at the sentence level, the evidence sentences are selected according to the contribution of each sentence to the answer, and the selection of evidence sentences is realized automatically. The selection of evidence sentences can be evaluated according to the extent of the improvement effect.

[0008] The present invention is implemented by adopting the following technical solutions.

[0009] First, let’s explain the relevant concepts:

[0010] Definition 1: Chapter sequence d

[0011] Its expression is: d={w1,w2,…,w N}, indicating a passage in reading comprehension. There are N words w1, w2, ..., w N , subscript N is the sequence length, and w represents the word.

[0012] Definition 2: Sentence sequence s

[0013] Its expression is: represents the i-th sentence in the chapter. There are K i Words Subscript K i is the sequence length, and w represents the word.

[0014] Definition 3: Question sequence q

[0015] Its expression is: q={w1,w2,…,w M}, indicating the question corresponding to the passage. The question has M words w1,w2,...,w M , subscript M is the sequence length, and w represents the word.

[0016] Definition 4: Answer sequence a

[0017] Its expression is: represents the candidate answer sequence corresponding to the question, a j represents the jth candidate answer, with a total of L j Words Subscript L j Sequence length, w represents word.

[0018] Definition 5: Meta-erasure method

[0019] Erasure is a method in the field of interpretability in natural language processing that analyzes how the model gets the correct answer. First, a model is trained, then part of the input is masked, and the change in the output is used to measure the input properties. Meta-erasure means masking the article sentence by sentence, specifically taking each sentence out with replacement, and then observing the change in the output to measure the importance of the sentence.

[0020] Definition 6: Evidence Sentence

[0021] A description used to explain where the article supports the answer, indicating the basis of the answer in the article. Usually one or more sentences to prove the accuracy of the answer, used to study the interpretability of the model.

[0022] A method for automatically extracting evidence sentences based on meta-erasure includes the following steps:

[0023] Step 1: Preprocess the text sequence d and obtain the list sents_list of all sentences in d.

[0024] Its expression is: sents_list=[s1,…,s x ], where s x represents the xth sentence, and x represents the number of sentences in d.

[0025] Step 2: Encode all the sentences in the options and sentence list sents_list.

[0026] Specifically, first, we use the pre-trained word vector to train option a (a is the answer sequence) and sentence s in the sentence list sents_list. x Then perform average pooling on each sequence to obtain a one-dimensional vector representation v for option a. a and sents_dense_list = [v s1 ,…,v sx ], where v sx It represents the one-dimensional vector representation of the x-th sentence after average pooling, and sents_dense_list is the one-dimensional vector representation of all sentences.

[0027] Step 3: Select the most relevant k sentences for each option as the new chapter, where k is a hyperparameter.

[0028] Specifically, we use option a to represent v as a one-dimensional vector. a Perform cosine similarity with the one-dimensional vector representation of each sentence in sents_dense_list to obtain a list of similarity scores for each sentence: sents_sim = [(1,sim1),…,(x,sim x )], where sim x It represents the similarity score of the xth sentence. After sorting it in descending order, select the first k sentences in sents_list according to the value of x. In order to maintain the semantic continuity of the document, arrange them in the order of the original list sents_list, and get new_sents_list = [s1,…,s k ],s kIt indicates the kth sentence selected, and new_sents_list is the new sentence list.

[0029] Step 4: Train model M0.

[0030] Specifically, the new chapter composed of option a and the new sentence list new_sents_list is concatenated and input into the pre-trained model (such as the chinese-roberta-wwm-ext model), and a linear layer is added as the top classifier. The loss function is the cross entropy loss, and the classifier prediction method is as follows:

[0031] p 2 =softmax(W T ×U0)

[0032] Among them, p 2 It represents the probability distribution of whether the option is correct or not, and whether the passage supports the description of the option. The superscript 2 represents binary classification. U represents the vector representation of the fused context information obtained after the text sequence is input into the model. The subscript 0 represents the one-dimensional vector representation corresponding to the cls label of the pre-trained model. W represents the weight matrix for calculating the binary classification probability value, and T represents the matrix transpose.

[0033] After multiplying the vector representation and the corresponding weight matrix, a softmax operation is performed to obtain the correct and incorrect probabilities of each option, and finally the accuracy Acc0 of the data set is obtained.

[0034] Step 5: Erase each sentence with replacement, and obtain the degree of change in the predicted probability value after erasure of each sentence according to model M0, and select the top t sentences with the largest change as evidence sentences.

[0035] Specifically, for option a, each sentence in new_sents_list is erased once, and the remaining k-1 sentences are merged to obtain k new chapter lists: erased_list = [d1,…,d k ], where d k It means the new chapter after erasing the kth sentence and merging the other sentences. x Splice and input into M0 for verification to get the probability after erasing the xth sentence To get the importance of the xth sentence to the option, use Where c represents the change, abs represents the absolute value; p 2 Indicates the probability distribution of whether the option is correct or not; the y value is 0 or 1, 0 means the option is wrong, 1 means the option is correct. Then, select the t sentences with the largest change as evidence sentences, and its expression is: evi_list=[e1,…,e t ],e trepresents the tth evidence sentence selected.

[0036] Step 6: Retrain model M1.

[0037] In order to indicate the correctness of evidence sentence extraction, the new chapter formed by merging option a with the evidence sentence list is spliced ​​and input into model M1 with the same structure as M0 for training to obtain the accuracy Acc1, which is compared with Acc0 to measure the effectiveness of annotation.

[0038] So far, from step 1 to step 6, the first training is performed first, and the obtained model is used to measure the importance of each sentence to the option after erasure, and then the evidence sentences are selected according to their importance. Finally, the evidence sentences are used for retraining to measure the effectiveness of the evidence sentences.

[0039] Beneficial Effects

[0040] Compared with the prior art, the method of the present invention can save a lot of time and manpower in evidence sentence extraction and annotation, and achieve good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0042] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0043] Example

[0044] Taking the VGaokao dataset as an example, this embodiment will explain in detail the specific operation steps of the method of the present invention with a specific example;

[0045] A method for automatic extraction of evidence sentences based on meta-erasure, such as Figure 1 As shown, the following steps are included:

[0046] Step 1: Segment the passage into sentences and select the 6 sentences that are most relevant to the options.

[0047] Specifically in this embodiment, the original article text is as follows:

[0048] "Black box" is a concept in cybernetics, meaning an object whose internal situation is completely unknown to the subject in terms of cognition. The meaning of "technological black box" is different from this. It is a special facility or process for storing and operating knowledge. Users are like facing a black box. They do not need to open it, understand or master the knowledge in it. They only need to operate according to the rules to get the expected results. For example, computers, mobile phones, cameras, chips, and medicines. It can be said that almost all the intermediate and final results of technology are technological black boxes. In the production process of technological black boxes, scientific knowledge is the foundation, and values ​​and ethics select scientific knowledge. ……

[0049] The option text is as follows:

[0050] Unlike a black box, the operation of a technological black box is controllable. Users do not have to thoroughly understand its working principles. They only need to follow the rules to get the expected results.

[0051] After step 1, we get:

[0052] The following is a list of 6 sentences selected from the original article:

[0053] ["The meaning of "technological black box" is different from this. It is a special facility or process for storing and operating knowledge. Users are like facing a black box. They don't need to open it, understand or master the knowledge in it. They just need to follow the rules to get the expected results.",--------0

[0054] "Just imagine, if all computer users in the world were to thoroughly understand the working principles of computers and the electronic theory on chips, how long would it take?",--------1

[0055] "People can rarely make any improvements to a hammer, but a computer after a month of use has been deeply marked with personal marks, which shows that when cognition becomes simple and easy, practice becomes complex and important.",--------2

[0056] "On the surface, it is the user who controls the black box of technology. However, the black box of technology exerts a subtle influence on the user without the user's "knowledge", ...",--------3

[0057] "It is worth noting that while the black box of science and technology makes scientific and technological knowledge widely shared by users, it also often causes this part of knowledge to depreciate or even be forgotten due to sharing.",--------4

[0058] "In addition, since users do not have to learn from scratch to operate the technology black box, the technology black box may be abused."]. --------5

[0059] Step 2: After the article formed by merging the options and the sentence list is spliced, it is input into the model M0 for training, and the training set accuracy Acc0 = 79.429% and the sample probability value p are obtained.

[0060] p=[-0.7886366844177246,0.33234280347824097], where the two numbers represent the probability values ​​of incorrect and correct options respectively (without softmax).

[0061] Step 3: Erase one sentence at a time, merge the remaining sentences into a new article, concatenate the options with it, and input it into the M0 model to obtain the probability list p of each sentence after erasure x :

[0062] p x =

[0063] [[-0.04151652753353119,-0.28928422927856445],--------0

[0064] [-0.4634270668029785,0.010878026485443115],--------1

[0065] [-0.3860200345516205,0.057259783148765564],--------2

[0066] [-0.9890433549880981,0.459352970123291],--------3

[0067] [-0.6935306787490845,0.1989961564540863],--------4

[0068] [-0.7593105435371399,0.2956990599632263]],--------5

[0069] Because the options are correct, arrange them in descending order according to the size of the change in the second digit, select the first two and then arrange them in the order of the original text to get a list of changes: [[0,0.6216270327568054],[1,0.32146477699279785]], where 0,1 represent the sentence numbers and the decimals represent the size of the change.

[0070] Step 4: Extract evidence sentences based on the change list obtained in step 3 and obtain two lists of evidence sentences:

[0071] ["The meaning of "technological black box" is different from this. It is a special facility or process for storing and operating knowledge. Users are like facing a black box. They don't need to open it, understand or master the knowledge in it. They just need to follow the rules to get the expected results.",--------0

[0072] "Just imagine, if all computer users in the world were to thoroughly understand the working principles of computers and the electronic theory on chips, how long would it take?"]. --------1

[0073] Step 5: Concatenate the option with the "new article formed by merging the list of evidence sentences obtained in step 4", input it into the model M1 with the same structure as M0, and train it again to obtain a new accuracy rate on the training set: Acc1 = 83.003%.

Claims

1. A method for automatically extracting evidence sentences based on meta-erasure, characterized in that: The following steps are involved: Step 1: Preprocess the text sequence d to obtain a list of all sentences in d, sents_list, where sents_list = [s1,…,s x ], where s x represents the xth sentence, x represents the number of sentences in d; The expression of the chapter sequence d is: d={w1,w2,…,w N }, which represents a passage in reading comprehension. There are N words w1, w2, ..., w N , subscript N is the sequence length, and w represents a word; Step 2: Encode all sentences in the options and sentence list sents_list; First, we use the pre-trained word vectors to train option a and sentence s in the sentence list sents_list. x Vector representation, a is the answer sequence; then average pooling is performed on each sequence to obtain a one-dimensional vector representation v of option a a and sents_dense_list = [v s1 ,…,v sx ], where v sx It represents the one-dimensional vector representation of the x-th sentence after average pooling, and sents_dense_list is the one-dimensional vector representation of all sentences; Among them, the answer sequence a is expressed as: represents the candidate answer sequence corresponding to the question, a j represents the jth candidate answer, with a total of L j Words Subscript L j Sequence length, w represents the word; Step 3: Select the most relevant k sentences for each option as the new chapter, where k is a hyperparameter; Represent v as a one-dimensional vector using option a a Perform cosine similarity with the one-dimensional vector representation of each sentence in sents_dense_list to obtain a list of similarity scores for each sentence: sents_sim = [(1,sim1),…,(x,sim x )], where sim x represents the similarity score of the xth sentence; after sorting it in descending order, select the first k sentences in sents_list according to the value of x; and then arrange them in the order of the original list sents_list to obtain new_sents_list = [s1,…,s k ],s k represents the kth sentence selected, and new_sents_list is the new sentence list; Step 4: Train model M0; The new chapter composed of option a and the new sentence list new_sents_list is concatenated and input into the pre-trained model. A linear layer is added as the top classifier. The loss function is the cross entropy loss. The classifier prediction method is as follows: p 2 =softmax(W T ×U0) Among them, p 2 Indicates the probability distribution of whether an option is correct or not, indicating whether the passage supports the description of the option. The superscript 2 indicates binary classification. U indicates the vector representation of the fused context information obtained after the text sequence is input into the model. The subscript 0 indicates the one-dimensional vector representation corresponding to the cls label of the pre-trained model. W indicates the weight matrix for calculating the binary classification probability value. T indicates the matrix transpose. Multiply the vector representation and the corresponding weight matrix and perform a softmax operation to obtain the correct and incorrect probabilities of each option, and finally obtain the accuracy Acc0 of the data set; Step 5: Erase each sentence with replacement, and obtain the degree of change in the predicted probability value after erasure of each sentence according to model M0, and select the first t sentences with the largest change as evidence sentences; Step 6: Retrain model M1; to indicate the correctness of evidence sentence extraction, concatenate option a with the new chapter formed by merging the evidence sentence list, input it into model M1 with the same structure as M0 for training, and obtain the accuracy Acc1, which is compared with Acc0 to measure the effectiveness of annotation.

2. The method for automatically extracting evidence sentences based on meta-erasure as claimed in claim 1, characterized in that: In step 5, for option a, each sentence in new_sents_list is erased once, and the remaining k-1 sentences are merged to obtain k new chapter lists: erased_list = [d1,…,d k ], where d k It represents the new chapter obtained by merging other sentences after erasing the kth sentence; Combine options a and d x Splice and input into M0 for verification to get the probability after erasing the xth sentence To get the importance of the xth sentence to the option, use Where c represents the change, abs represents the absolute value; p 2 Indicates the probability distribution of whether the option is correct or not; the y value is 0 or 1, 0 means the option is wrong, 1 means the option is correct; then, select the t sentences with the largest change as evidence sentences, and its expression is: evi_list=[e1,…,e t ],e t represents the tth evidence sentence selected.

Images