A machine reading comprehension method based on repeated span prediction

Abstract

The present application relates to a kind of machine reading comprehension method based on repeated span prediction, belong to natural language processing machine reading comprehension field.The method includes: for the reading comprehension task of span prediction form, increase the task of predicting repeated span.The task first finds all repeated spans in text paragraph based on greedy algorithm, then the short span contained in long span is filtered, after obtaining repeated span set, for each group of repeated span, randomly select one as answer span, and other spans in group are replaced by mask.Processed text is input into pre-training model, and representation vector is obtained, and based on softmax, which span in paragraph should the mask position point to be predicted.In the task, the obtained model is further fine-tuned on target task.The method considers the problem of lack of span knowledge of pre-training model, and constructs data in unsupervised manner, so that the model can better learn span representation, and improve the performance of model in span prediction.

Description

Technical Field

[0001] This invention relates to a multi-stage machine reading comprehension method, specifically a pre-training method that combines a repetitive span prediction task, belonging to the field of natural language processing machine reading comprehension technology. Background Technology

[0002] In recent years, with the development of deep learning technology, especially the introduction of pre-trained models, machine reading comprehension technology has made significant progress and has been widely used in many aspects such as text question answering, information extraction and search engines.

[0003] Machine reading comprehension can be categorized into four types based on task format: cloze tests, multiple choice, span prediction, and free-response question answering. Among these, span prediction requires the model to extract a passage of text as the answer given context and a question. This demands a deep understanding of the text content and the ability to make inferences. Compared to cloze tests and multiple choice tasks, span prediction places higher demands on the model and provides a more accurate measure of its text comprehension.

[0004] With the development of pre-training techniques and the release of relevant datasets, reading comprehension techniques based on span prediction have rapidly advanced. The pre-training-fine-tuning training model has become a widely used paradigm. During pre-training, the model learns rich semantic knowledge through self-supervised learning. During fine-tuning, the model approaches the data distribution of the target task, thereby achieving better performance on the target task. Currently, on multiple span-based reading comprehension datasets, neural network models using this model have achieved scores exceeding human capabilities.

[0005] While pre-training techniques significantly improve model performance, for specific tasks, a certain amount of labeled dataset is required during the model fine-tuning phase; otherwise, overfitting and other problems can easily occur. Since constructing labeled datasets is very time-consuming and labor-intensive, some works have emerged targeting small datasets, such as few-shot learning and semi-supervised learning. These methods improve the model itself and have achieved good results with limited data.

[0006] Nevertheless, for reading comprehension tasks involving span prediction, these works have not started from the essence of span prediction and further explored the semantic unit of span. As a result, the models cannot learn sufficient span knowledge when faced with small-scale datasets. Therefore, existing techniques are not effective in solving small-scale reading comprehension tasks. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies that fail to learn the span in reading comprehension, resulting in poor performance on small datasets. In order to fully utilize the span knowledge in the text to solve the problem of limited samples in reading comprehension, a machine reading comprehension method based on repeated span prediction is creatively proposed.

[0008] The innovation of this method lies in the addition of a repeated span prediction task between pre-training and fine-tuning of the target task. Data is constructed in an unsupervised manner, allowing the model to learn the semantic relationship between span boundaries and spans, thus achieving good results even in span prediction scenarios with small samples.

[0009] The present invention is achieved using the following technical solution.

[0010] First, let's explain the relevant concepts:

[0011] Definition 1: Chapter sequence p

[0012] Its expression is: {p=w1,w2,...,w N} represents a passage in reading comprehension, containing N words w1, w2, ..., w N The subscript N represents the sequence length, and w represents a word.

[0013] Definition 2: Problem sequence q

[0014] Its expression is: {q=w1,w2,...,w M} represents the question corresponding to the passage, which has M words w1, w2, ..., w M The subscript M represents the sequence length, and w represents a word.

[0015] Definition 3: Answer sequence a

[0016] Its expression is: {a=w1,w2,...,w L} represents the sequence of answers to the question, which originates from the passage sequence p, consisting of L words w1, w2, ..., w L The subscript L represents the sequence length, and w represents a word.

[0017] Definition 4: Span sequence s

[0018] Its expression is: {s=w x ,w x+1 ,...,w x+n The span sequence comes from the passage sequence p, consisting of n+1 words w. x ,w x+1 ,...,w x+n Composition, the subscript x indicates the starting position of the span, and w represents a word.

[0019] Definition 5: Masking Mechanism

[0020] This is one of the self-supervised tasks in the pre-training phase. Partial spans in the passage sequence p are replaced with masks, and the model is then asked to predict the words at the mask positions. This allows the model to learn the relationships within the sequence context.

[0021] Definition 6: Multi-stage training

[0022] Multi-stage training is a common method in transfer learning. First, the model is initialized with parameters using different datasets or tasks to transfer knowledge from other datasets or semantics. Then, in the next stage, the model learns the target task.

[0023] A multi-stage reading comprehension method based on predicting repetition span includes the following steps:

[0024] Step 1: Preprocess the passage sequence p to find all repeated spans in p.

[0025] Specifically, a greedy algorithm is used to find all repeating spans in the passage sequence p and store them in a list span_cluster, whose expression is: span_cluster[i]={(s1,e1),…,(s n ,e n )}, where each item (s) x ,e x ) represents the start and end positions of the i-th repeated span in the text, and n represents the number of repeated spans.

[0026] Step 2: Filter out duplicate spans.

[0027] Sort by span length, filter for overlapping spans, and retain the longest overlapping span.

[0028] Step 3: Shuffle the filtered span set, and for each group of duplicate spans, randomly select a span (s). j ,e j Without making any changes, the other repeated spans in the same group are replaced with masks [QUESTION], and finally each group of repeated spans forms n-1 training data.

[0029] Step 4: Encode the text sequence.

[0030] Specifically, for the masked sequence p, its word vector encoding, position encoding, and segment encoding are summed and input into the pre-trained model BERT. BERT predicts the corresponding span (s) for each [QUESTION] position. j ,e jThe location is predicted as follows:

[0031]

[0032] Among them, P s P represents the probability distribution of the starting position of the span. e The probability distribution represents the end position of the span; U represents the vector representation of the text sequence after it is input into the model, which incorporates contextual information. This represents the weight matrix used to calculate the starting index of the span; This represents the weight matrix used to calculate the end index of the span; T represents the matrix transpose.

[0033] Multiply the vector representation by the corresponding weight matrix and then perform a softmax operation to obtain the start and end probabilities at each position in the sequence.

[0034] Step 5: Initialize the parameters using the obtained model, then preprocess the target task dataset by concatenating the passage sequence p and the question sequence q. To maintain consistency with the previous input format, add a [QUESTION] mask marker at the end of the sequence.

[0035] Step 6: Input the sequence concatenated in Step 5 into the model to obtain the vector representation of the sequence. Using the same method as in Step 4, calculate the start and end positions of the answer span, use cross-entropy as the loss function, and update the model parameters using gradient descent to obtain the answer obtained by the model inference.

[0036] Thus far, from step 1 to step 6, the prediction of repeated spans is first performed, and the corpus data is constructed in an unsupervised manner so that the model learns the meaning of spans and the referential relationship between spans. Then, in the second stage, the data format of the target task is changed to enable the model to further learn the knowledge of the target task.

[0037] Beneficial effects

[0038] Compared with existing technologies, the method of this invention can achieve excellent results in scenarios with small sample span prediction. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0040] The method of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0041] Example

[0042] Taking the reading comprehension data in the CMRC2018 dataset as an example, this example will illustrate the specific operation steps of the method described in the present invention with specific examples;

[0043] A machine reading comprehension method based on repeated span prediction, as Figure 1 shown, includes the following steps:

[0044] Step 1: Calculate the set of repeated spans.

[0045] Specifically in this example, the original paragraph text is as follows:

[0046] The Qin Taishan Stele, also known as the Stele of Sealing Taishan, was erected when Emperor Qin Shi Huang made an eastern inspection tour of Taishan in the 28th year of his reign (219 BC). The stele is four feet five inches high and one foot four inches wide, with neat characters and round and vigorous strokes. It is the small seal script after Emperor Qin Shi Huang unified the characters, and is said to be written by Prime Minister Li Si. On the north, east, and west sides of the stele are the imperial edicts of Emperor Qin Shi Huang, totaling 144 characters. On the south side are the 78 characters of the imperial edict of the Second Emperor inscribed in the first year of the reign of Emperor Qin Er Shi (209 BC). The original stele stood on the top of Mount Tai. By the Ming Dynasty, only 29 characters remained. In the fifth year of the Qianlong reign of the Qing Dynasty (1740), it was damaged by fire and the stele was lost. In the twentieth year of the Jiaqing reign (1815), two pieces of the remaining stele were found in the Jade Maiden Pool on the top of the mountain, with 10 characters remaining. They were moved to the Dai Temple. Now it is located on the west side in front of the terrace of the East Imperial Seat in the Dai Temple of Mount Tai, with only 10 remaining characters of the imperial edict of the Second Emperor: Si, Chen, Qu, Ji, Mei, Si, Chen, Qing, Yi, Chen.

[0047] After word segmentation, find the set of all repeated spans from left to right. In this example, the set is ([(1,1),(14,14),(60,60),(92,92)],[(14,15),(60,61),(92,93)],[(14,16),(60,62),(92,94)],[(2,2),(10,10),(30,30)],[(2 ,3),(10,11),(30,31)],[(3,3),(11,11),(31,31)],[(4,4),(82,82)],[(4,5),(82,83)],[(5,5),(83,83)],[(12,12),(36,36)],[(15,15),(61,61),(93,93)],[(1 5,16),(61,62),(93,94)],[(16,16),(62,62),(94,94)],[(20,20),(26,26)],[(28,28),(86,86)],[(33,33),(79,79)],[(38,38),(46,46)],[(39,39),(45,45)],[ (41,41),(47,47)],[(44,44),(64,64)],[(49,49),(66,66)],[(72,72),(76,76)],[(74,74),(91,91)],[(80,80),(96,96)]), where each element [] is a set of repeated spans, and the index is the index position after paragraph segmentation.

[0048] Step 2: Filter out spans with overlapping indexes.

[0049] Specifically in this embodiment, for example, (14, 14), (14, 15), (14, 16) are cases where the long span contains the short span. Filter the short spans and only retain the set of spans with the longest length. The filtered span set is (([(14, 16), (60, 62), (92, 94)], ('Qin', 'Shi', 'Huang')), ([(2, 3), (10, 11), (30, 31)], ('Tai', 'Shan')), ([(4, 5), (82, 83)], ('Ke', 'Shi')), ([(12, 12), (36, 36)], ('Bei',)), ([(20, 20), (26, 26)], ('Year',)), ([(28, 28), (86, 86)], ('East',)), ([(33, 33), (79, 79)], ('Place',)), ([(38, 38), (46, 46)], ('Four',)), ([(39, 39), (45, 45)], ('Chi',)), ([(41, 41), (47, 47)], ('Cun',)), ([(44, 44), (64, 64)], ('One',)), ([(49, 49), (66, 66)], ('Character',)), ([(72, 72), (76, 76)], ('Xiang',)), ([(74, 74), (91, 91)], ('Wei',)), ([(80, 80), (96, 96)], ('Book',))).

[0050] Step 3: Select spans to apply masks.

[0051] For each group of repeated spans, randomly select one span to remain unchanged and replace the other spans with masks. Taking the set ([(14, 16), (60, 62), (92, 94)], ('Qin', 'Shi', 'Huang')) as an example, using [QUESTION] to replace the spans, a new passage sequence is obtained:

[0052] The Stele Inscription on Mount Tai by Emperor Qin Shi Huang, also known as the Stele of Mount Tai, was erected when Emperor Qin Shi Huang made an inspection tour to Mount Tai in the 28th year of his reign (219 BC). The stele is four chi and five cun high and one chi and four cun wide. The font is regular and the strokes are round and strong. It is the small seal script after [QUESTION] unified the writing system. It is said to be written by Prime Minister Li Si. On the north, east, and west sides of the stele are [QUESTION] imperial edicts, totaling 144 characters. On the south side are 78 characters of the imperial edict of the Second Emperor inscribed in the first year of the reign of Emperor Qin Er Shi (209 BC). The stele was originally erected on the summit of Mount Tai. By the Ming Dynasty, only 29 characters remained. In the fifth year of the Qianlong reign of the Qing Dynasty (1740), it was damaged by fire and the stele was lost. In the twentieth year of the Jiaqing reign (1815), two pieces of the damaged stele were found in the Jade Maiden Pool on the summit, with 10 characters remaining. They were moved to the Dai Temple. Now it is located on the west side in front of the terrace of the East Imperial Seat in the Dai Temple of Mount Tai, with only the 10 remaining characters of the imperial edict of the Second Emperor: Si, Chen, Qu, Ji, Mei, Si, Chen, Qing, Yi, Chen.

[0053] Step 4: Encode the model to predict the repeating span;

[0054] Specifically, in this embodiment, the text sequence obtained in step 3 is combined with the position encoding and segment encoding of the sequence and input into BERT to obtain the vector representation of each word with fused context information. Then, a linear transformation is performed on the vector representations, and the start and end positions corresponding to [QUESTION] are predicted by softmax. The loss function used is cross-entropy loss.

[0055] Step 5: Process the target task data.

[0056] Specifically, in this embodiment, the chapters and questions corresponding to the original target task are concatenated, and the same mask marker as in the previous stage is added at the end to form the following training data: [CLS] Qin Taishan Inscription, also known as the Fengtaishan Stele, was erected in the 28th year of Qin Shi Huang's eastern tour of Taishan in 219 BC. The stele is 4 feet 5 inches high and 1 foot 4 inches wide, with neat characters and rounded strokes. It is the Small Seal script after Qin Shi Huang unified the writing system, and it is said to have been written by Prime Minister Li Si. The north, east, and west sides of the stele contain Qin Shi Huang's edicts, totaling 144 characters, while the south side contains the Second Emperor's edicts, which were carved in the first year of Qin Er Shi's reign (209 BC), totaling 78 characters. The stele was originally located on the summit of Mount Tai, but only 29 characters remained by the Ming Dynasty. The stone inscription was destroyed by fire in the fifth year of the Qianlong reign (1740) and lost. In the twentieth year of the Jiaqing reign (1815), two fragments of the stone were found in the Jade Maiden Pool on the mountaintop, with 10 characters still remaining. They were then moved to the Dai Temple... [SEP] When was the Qin Taishan Stone Inscription erected? [QUESTION][SEP] The answer spans (14,30), and the corresponding text fragment is "When Qin Shi Huang made his eastern tour of Taishan in the twenty-eighth year of his reign (219 BC)".

[0057] Step 6: Model training.

[0058] Specifically, in this embodiment, the sequence from step 5 is input into the model to obtain the contextual representation of the sequence. Similar to step 4, the vector representation is first linearly transformed, and then the sentence representation is input into the Softmax layer to calculate the probabilities of the corresponding start and end positions, thus obtaining the answer obtained by the model's inference.

Claims

1. A multi-stage reading comprehension method based on predicted repetition span, characterized in that, Comprising the following steps: Step 1: Preprocess the passage sequence p, find all the repeated spans in p; Wherein, the chapter sequence p expression is: {p = w1, w2, …, w N}, indicating the chapter in the reading comprehension, the chapter has N words w1, w2, …, w N , the subscript N is the sequence length, and w represents a word; Step 2: Sort by the length of the span, filter the spans with overlapping indexes, and keep the longest span in the overlapping span; Step 3: Shuffle the filtered span set, for each list of repeated spans, randomly select one span (s j ,e j ) without any change, and replace the other repeated spans in the same group with the mask [QUESTION], respectively. Finally, each group of repeated spans forms n-1 pieces of training data, where n represents the number of repeated spans. Step 4: Encode the text sequence; For the masked passage sequence p, add its word vector encoding, position encoding and segment encoding to the pre-trained model bert, and bert predicts the corresponding span (s j ,e j ) position for each [QUESTION] position. The prediction method is as follows: where P s denotes the probability distribution of the span start position, P e denotes the probability distribution of the span end position; U denotes the vector representation of the fusion context information obtained after the text sequence is input into the model; denotes the weight matrix for calculating the span start index; denotes the weight matrix for calculating the span end index; T denotes matrix transposition; Multiply the vector representation and the corresponding weight matrix, and do softmax operation to get the start and end probability of each position in the sequence; Step 5: Initialize the parameters using the obtained model, then preprocess the target task data set, concatenate the passage sequence p and the question sequence q; add the [QUESTION] mask mark at the end of the sequence; wherein the question sequence q expression is: {q = w1, w2,..., w M}, representing a question corresponding to the passage, the question having M words w1, w2,..., w M , the subscript M being the sequence length, and w representing a word; Step 6: Input the sequence concatenated in step 5 into the model to get the vector representation of the sequence, use the same method as step 4 to calculate the start and end positions of the answer span, use cross entropy as the loss function, and update the model parameters using gradient descent to get the answer inferred by the model.

2. The method of claim 1, wherein, In step 1, a greedy algorithm is used to find all the repeated spans in the paragraph sequence p and store them in a list span cluster, which is expressed as: span cluster[i] = {(s1, e1), …, (s n ,e n )}, where each item (s x ,e x ) represents the start position and end position of the i-th group of repeated spans in the paragraph.

Images