An event extraction method based on a machine reading comprehension model

Abstract

The present invention discloses an event extraction method based on a machine reading comprehension model. The method models the two subtasks of event extraction: event detection and event element extraction, into machine reading comprehension tasks, and uses a multi-round question-answering approach to extract complete information about the event. The present invention constructs a machine reading comprehension framework by using a BERT pre-training model. In order for the model to learn the entity information in the sentence, the entities in the sentence are explicitly marked in the input sentence. In addition, a historical conversation information encoding module is constructed, and an attention mechanism is used to capture important information from historical conversations to assist in the current round of question-answering. Compared with existing methods, the method of the present invention can better utilize prior knowledge to assist in event extraction, and has broad application prospects in automatic summarization, automatic question-answering, and the construction of causal graphs.

Description

Technical Field

[0001] The present invention designs an event extraction method based on a machine reading comprehension model, belonging to the technical field of natural language processing. Background Art

[0002] Event extraction is a key research area in information extraction, with significant application value in areas such as automatic summarization, automatic question answering, and information retrieval. It is also a key technology for constructing event graphs. Event extraction is the process of extracting structured descriptions of events from given unstructured text. It is generally divided into two subtasks: event detection and event element extraction. Event detection involves accurately detecting the triggering words of an event and classifying it as the correct event type. Event element extraction, on the other hand, involves accurately extracting the key elements of an event and assigning them appropriate roles.

[0003] In early work, event extraction was often treated as a sequence labeling or classification task, the core of which was to obtain a good semantic feature representation for a sentence. Traditional methods primarily rely on feature engineering to obtain sentence feature representations, relying on researchers to manually design features at different levels based on their expertise. The performance of such methods is significantly affected by the features themselves, and the handcrafted features have poor transferability across different scenarios. In recent years, neural network-based methods have been widely used for event extraction. Compared to feature engineering-based methods, neural network methods use a layer-by-layer abstract representation learning approach to obtain semantic features of sentences. This approach is independent of specific resources and better preserves the semantic information of sentences. Therefore, neural network methods have effectively promoted the progress of event extraction. However, when using classification or sequence labeling for event extraction, large amounts of labeled data are required to train the model, and the trained model can only be used to extract events of known types. In reality, due to the complexity of events, event labeling is very expensive, resulting in a relatively limited number of instances and the types of events covered in existing event extraction datasets. Therefore, how to train a transferable event extraction model in low-resource settings has become a hot topic for researchers.

[0004] In response to the problem of lack of labeled data, researchers have proposed in recent years to use machine reading comprehension models to solve the problem of event extraction ([1]. Du X, Cardie C. Event Extraction by Answering (Almost) Natural Questions [C] / / Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020: 671-683; [2]. Liu J, Chen Y, Liu K, et al. Event extraction as machine reading comprehension [C] / / Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020: 1641-1651; [3]. Li F, Peng W, Chen Y, et al. Event extraction as multi-turn question answering [C] / / Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Processing: Findings. 2020: 829-838.). These methods can bring two advantages by converting event extraction into a reading comprehension task: (1) On the one hand, by encoding prior information in the question, the model can extract some event role information that has not been seen before; (2) On the other hand, the existing large-scale reading comprehension data can be used to assist in training the model, thereby alleviating the problem of lack of labeled data. Although the above-mentioned methods based on machine reading comprehension models have effectively promoted the research progress of event extraction in low-resource scenarios, in the extraction process, the existing methods have not fully utilized the prior information in the sentence, such as the entity information in the sentence. In addition, the extraction of each element of the event is carried out independently, and the correlation between the event element information and the interactivity between multiple rounds of question and answer are not effectively utilized. In event extraction, the entity information in the sentence is helpful for event extraction. On the one hand, event elements are all entities. By explicitly marking the entity in the input sentence, it will help to determine the boundary of the event element. On the other hand, entity information helps to infer the event type. For example, in the sentence "The Iraqi unit(organization)in possession of those guns fired mortars (weapon)in the direction of the 7th Cavalry (organization)", if we know that the entity type of "Iraqi unit" and "7th Cavalry" is organization (Organization), and the entity type of "mortars" is weapon (Weapon), then it will help the model to infer " fired " expresses an attack event (Conflict.Attack) rather than a resignation event (Personnel.End-Position). On the other hand, there is correlation between event elements, and the interaction of the question-answering process is beneficial to event extraction. For example, in the sentence "In the January attack, two Palestinians suicide bombers blew themselves up in central Tel Aviv, killing 23other people." If you already know " suicide " triggered the attack event (Conflict.Attack), which will help to infer the subsequent killing "The death event (Life.Die) triggered by

[0005] In response to the above problems, the present invention proposes an event extraction method based on a multi-round question-answering model, which extracts event information in an iterative question-answering manner. First, at the input data level, by explicitly adding entity marker symbols to the input sentences, the model can effectively learn the entity information in the sentences. Then, at the model level, this chapter adds a historical conversation information encoding module based on the machine reading comprehension model, and uses the attention mechanism to select useful information from historical conversations to assist in the current round of question-answering. Next, at the answer decoding end, two answer decoding algorithms, single-segment and multi-segment, are designed to extract a single answer or multiple answers from a paragraph. Summary of the Invention

[0006] This paper proposes an event extraction method based on a machine reading comprehension model for event extraction tasks. This method models event detection and event element extraction as a unified question-answering task, implemented through multi-round question-answering. This method builds a reading comprehension framework based on the BERT model. By annotating entity information in the input paragraphs and encoding historical conversation information through a historical conversation information encoding module, the model can better utilize relevant prior knowledge to improve the performance of event extraction tasks.

[0007] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0008] 1. The event extraction process based on the machine reading comprehension model described in the present invention is as follows Figure 1 As shown, it realizes event detection and event element extraction through multiple rounds of question and answer. The event extraction method based on the machine reading comprehension model of the present invention includes the following steps:

[0009] Step 1: First, construct a trigger word extraction question based on the question template;

[0010] Step 2: Explicitly mark the entity information in the input sentence;

[0011] Step 3: Concatenate the question generated in step 1 and the sentence obtained in step 2, and feed them into the machine reading comprehension model to extract trigger words;

[0012] Step 4: For the trigger words extracted in step 3, further construct event type extraction questions based on the question template;

[0013] Step 5: All event types are concatenated with the identifier "[EVENT_TYPE]" to form a new input paragraph. This paragraph is then concatenated with the question generated in step 4 and fed into the machine reading comprehension model to extract the corresponding event type.

[0014] Step 6: Based on the trigger words and the role of the event, we further construct an event role element extraction question. Then, we concatenate the sentence obtained in step 2 into the generated question and input it into the reading comprehension model that integrates historical conversation information to extract the corresponding event role elements.

[0015] Step 7: Repeat step 3 to continue extracting new event information until all events in the sentence are extracted.

[0016] The description of the trigger word extraction problem in step 1 is as follows:

[0017] When constructing questions, each question consists of two parts: question type and question content. The question type indicates the type of interrogative pronoun to use. Questions about trigger words use "Which" as the interrogative pronoun. The question content indicates the main point of interest. The fixed template for question content constructed for trigger word extraction is: "Which is the_①_trigger in this text?", where ① indicates the trigger word in the sentence to be extracted. For example, for the trigger word of the first event, the content of space ① is "first", the second is "second", and so on.

[0018] The following is an explanation of the explicit marking of the entity information of the input sentence in step 2:

[0019] When explicitly marking entities in a text paragraph, we add entity type markers before and after the entity word to get a new sentence. For example, for the sentence "The Iraqi unit in possession of those guns fired mortars in the direction of the 7th Cavalry", adding entity markers will give us the following sentence: "The <org> Iraqi unit< / org> in possession of those <wea> guns< / wea> fired <wea> mortars< / wea> in the direction of the <org> 7th Cavalry< / org> .",in" <org> 、 <wea>” represent organizations and weapons respectively.

[0020] The following is an explanation of the trigger words extracted in step three:

[0021] The question generated in step 1 and the sentence generated in step 2 are concatenated with the marker symbol "[SEP]", and then the "[CLS]" and "[SEP]" flags are added at the beginning and end respectively. After that, they are fed into the BERT model for encoding to obtain the hidden layer output corresponding to each word. The calculation form is as follows:

[0022] c cls ,c q1 ,...,c qn , c [sep] ,c d1 ,...,c dm ,c sep =BERT([CLS]+Q' k +[SEP]+P e ' ntity +[SEP])(1)

[0023] where Q' k is the problem generated in step 1, P e ' ntity The sentence generated in step 2 with entity tag information added is shown below. After obtaining the hidden layer features of each word, the contextualized word vector representation of each word is subjected to two different linear transformations and then fed into the Softmax function to calculate the probability that the word is the beginning and end of the answer. The calculation process is as follows:

[0024] P s (i)=Softmax(W s c di +b s ) (2)

[0025] P s (i)=Softmax(W e c di +b e ) (3)

[0026] in are the parameters of the model; is the corresponding bias.

[0027] Since the trigger word is a span in the input sentence, a single-span decoding algorithm is used to obtain the boundary corresponding to the trigger word. The decoding process is as follows: First, the two probabilities corresponding to all words are sorted in descending order to obtain two probability lists; then, the two lists are traversed in sequence. During the traversal process, only the spans with a starting probability and an ending probability greater than the [CLS] position are selected. Spans that do not meet the requirements are further removed according to the filtering conditions (the filtering conditions include two types: (1) the end position of the span is less than the starting position; (2) the length of the span exceeds the maximum length of the answer). The remaining spans are then added to the candidate list in sequence. Finally, the first span in the candidate list is output as the event trigger word.

[0028] The event types extracted in step 5 are described as follows:

[0029] When extracting event types, since trigger words have already been extracted, it is necessary to encode previous historical conversation information into the model. The specific process is as follows: First, the paragraph formed by splicing with the identifier "[EVENT_TYPE]" is concatenated to the question generated in step 4 and then fed into the reading comprehension model for encoding. The hidden layer feature vector output for each word is calculated as follows:

[0030] c cls ,c q1 ,...,c qn , c [sep] ,c d1 ,...,c dm ,c sep =BERT([CLS]+Q' k +[SEP]+P′ entity +[SEP])(4)

[0031] Next, the answer sentences in the historical question-answer pairs are marked with "[CLS]" and "[SEP]" at the beginning and end, respectively, and then fed into another BERT model for encoding to obtain the corresponding sentence representation. The calculation form is as follows:

[0032]

[0033] In the above formula, A′ i represents the sequence of the answer sentences of round i after splitting, and is the sentence feature vector representation obtained after encoding. Since the historical conversation contains multiple question-answer pairs, and each question-answer has a different contribution to the question-answer of the current round. Therefore, the present invention uses an attention mechanism to filter out information that has important relevance to the question-answer of the current round from the historical question-answer pairs. In the attention calculation process, the attention weight coefficient is calculated as follows:

[0034]

[0035] Where, is the mapping matrix, is the attention weight corresponding to the answer sentence in the i-th round of question-answer pair. According to the attention weight, the feature vector representation of the answer sentence in all rounds of answer pairs is weighted and summed to obtain the feature representation c of the entire historical session information. h :

[0036]

[0037] After obtaining the feature representation c of historical session information h Then, it is concatenated with the feature vector of each word output by the reading comprehension model to obtain the fused feature vector:

[0038]

[0039] Finally, the feature vector of each word is passed through two different linear mapping layers and then fed into the Softmax function to calculate the probability of it being the beginning and end of the answer. The calculation process is as follows:

[0040] P s (i)=Softmax(W s c' pi +b s ) (8)

[0041] P s (i)=Softmax(W e c' pi +b e ) (9)

[0042] Where, are the parameters of the model, is the corresponding bias, h is the BERT hidden layer feature dimension; P s (i), P e (i) represents the probability of the i-th word being the start and end of the answer. After obtaining the start and end probabilities corresponding to each word, the same decoding algorithm as in step 3 is used to decode the event type.

[0043] The corresponding event role elements extracted in step 6 are described as follows:

[0044] Different interrogative pronouns are used for different roles. Table 1 shows the correspondence between different interrogative pronouns and event roles. As can be seen from the table, question types are mainly divided into three categories: questions about general semantic roles, questions about people, and questions about places.

[0045] Table 1 Correspondence between event roles and question pronouns

[0046]

[0047]

[0048] When extracting event role elements, the question content is constructed using the following fixed template: "(Where / Which / What)is the_③_in the_④_event triggered by_②_?" In this template, space ③ represents the event role type, space ④ represents the event type, and space ② represents the trigger word. After question construction, the constructed question and the sentence generated in step 2 are concatenated and input into the machine reading comprehension model to obtain the probability of each word representing the beginning and end of the answer. During the event role information process, historical conversation information also needs to be encoded into the model, using the same encoding method as in step 5. Because some event role elements contain multiple answers, a multi-segment decoding algorithm is used to decode the answers. The specific decoding process is as follows: First, the start and end probabilities corresponding to all words are sorted in descending order to obtain two probability lists; then, the two lists are traversed in sequence. During the traversal process, only the segments with the start and end probabilities greater than the [CLS] position are selected, and the spans that do not meet the requirements are further removed according to the filtering conditions (the filtering conditions include three types: (1) the end position of the span is less than the start position; (2) the span length exceeds the maximum length of the answer; (3) the sum of the start position probability score and the end position probability score in the span is less than the threshold), and the remaining spans are added to the candidate list in sequence. Finally, all spans in the candidate list are output as the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is the event extraction process based on multi-round question and answer proposed by the present invention.

[0050] Figure 2 This is the framework structure of the reading comprehension model proposed by the present invention that integrates historical conversation information. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings:

[0052] Figure 1 This is the event extraction process based on the multi-round reading comprehension framework proposed by the present invention. The present invention models both event detection and event element extraction as reading comprehension tasks. Figure 1 The whole process is explained in detail using the example sentence "Andrewluster, though arrived yesterday at Los Angeles international airport on a commercial flight from Mexico."

[0053] The first step is to extract the first trigger word of the sentence. The specific process of this step is as follows: First, after marking the entities in the sentence, we get the following sentence "The <org> Iraqi unit< / org> in possession of those <wea> guns< / wea> fired <wea> mortars< / wea> in the direction of the <org> 7th Cavalry< / org> "; then, based on the question template, a question is generated to extract the first trigger word: "Which is the first trigger in this text?"; the question and sentence are then concatenated with the identifier "[SEP]", and "[CLS]" and "[SEP]" are added at the beginning and end before being fed into the machine reading comprehension model to obtain the answer.

[0054] The second step is to extract the event type corresponding to the trigger word. The specific process of this step is as follows: First, based on the template, a trigger word type extraction question is generated: "Which is the event type of the trigger word arrived?"; then, all event types are concatenated using the identifier "[EVENT_TYPE]" and further concatenated to the entity-labeled sentence obtained in step 1 to obtain a new input paragraph; this paragraph is then concatenated to the event type extraction question and fed into the machine reading comprehension model to extract the event type.

[0055] The third step is to extract the event's role elements. This process is as follows: First, determine the roles involved in the event based on the event type. Then, determine the interrogative pronouns for the corresponding roles based on the correspondence between event roles and interrogative pronouns described in Table 1. Then, generate a question for extracting event role elements based on the template "(Where / Which / What) is the_③_in the_④_event triggerby_②_?" Finally, feed the sentence generated in the question and trigger word extraction steps into the machine reading comprehension model to extract the trigger words. After extracting the first event, continue constructing trigger word extraction questions to extract new event information from the sentence.

[0056] Figure 2 This is a reading comprehension model that incorporates historical conversation information. It consists of a machine reading comprehension module and a historical conversation information encoding module. The machine reading comprehension module extracts event information, while the historical conversation information module encodes the answer sentences from historical question-and-answer conversations. Both models use the BERT model to encode input sentences. Since only some of the question-and-answer sentences in a historical question-and-answer pair are relevant to the historical conversation information of the current round, an attention mechanism is used to filter out the most relevant information from the historical conversation information. The attention calculation process is as follows: First, the feature vector output by the "[CLS]" label of the question-and-answer comprehension module is dot-producted with the feature vector of the answer sentence in each historical conversation pair output by the historical conversation encoding module to obtain a similarity score. This attention score is then used as a weight to sum the feature vectors of the answer sentences in all historical question-and-answer pairs to obtain a feature vector representation of the historical conversation information. Next, this feature vector of the historical conversation information is concatenated with the feature vector of each word obtained by the reading comprehension model to obtain the corresponding feature output. This feature output is then linearly transformed and subjected to a softmax function to obtain the probability of each word being the start or end of the answer. Finally, one or more spans are selected from the sentence as the answer using the answer decoding algorithm.

[0057] Table 2 shows the changes in the event extraction performance of the model before and after the introduction of the historical session information encoding module on the public dataset ACE 2005. From the experimental results, it can be seen that the performance of the model is effectively improved after the introduction of the historical session information encoding module.

[0058] Table 2 Performance changes of the model before and after introducing historical session information on the ACE 2005 dataset

[0059]

[0060] Table 3 is a comparison of the model proposed in this invention with existing methods on the public dataset ACE 2005. Experiments show that the model proposed in this invention achieves the best result in terms of the comprehensive evaluation index F1 value compared with existing models.

[0061] Table 3 Experimental comparison results of the network model of the present invention and other existing models on the ACE 2005 dataset

[0062]

[0063] The above embodiments are only preferred embodiments of the present invention and are not limitations on the technical solutions of the present invention. Any technical solution that can be implemented on the basis of the above embodiments without creative work should be deemed to fall within the scope of protection of the patent of the present invention.< / wea> < / org>

Claims

1. An event extraction method based on a machine reading comprehension model, characterized in that The following steps are involved: Step 1: First, construct a trigger word extraction question based on the question template; Step 2: Explicitly mark the entity information in the input sentence; Step 3: Concatenate the question generated in step 1 and the sentence obtained in step 2, and feed them into the machine reading comprehension model to extract trigger words; Step 4: For the trigger words extracted in step 3, further construct event type extraction questions based on the question template; Step 5: All event types are concatenated with the identifier "[EVENT_TYPE]" to form a new input paragraph. This paragraph is then concatenated with the question generated in step 4 and fed into the machine reading comprehension model to extract the corresponding event types. Step 6: Based on the trigger words and the role of the event, we further construct an event role element extraction question. Then, we concatenate the sentence obtained in step 2 into the generated question and input it into the reading comprehension model that integrates historical conversation information to extract the corresponding event role elements. Step 7: Repeat step 3 to continue extracting new event information until all events in the sentence are extracted; The machine reading comprehension module is used to extract event information, while the historical conversation information module is used to encode the answer sentences in historical question-and-answer conversations. When performing feature fusion, the following steps are followed: a) First, add the "[CLS]" and "[SEP]" flags at the beginning and end of the answer sentences in the historical question-answer pairs, and then feed them into the BERT model for encoding to obtain the corresponding sentence representation. The calculation form is as follows: In the above formula, A i ' represents the sequence of the answer sentences after the i-th round is split, and and is the sentence feature vector representation obtained after encoding; b) Then, the relevance score between the answer sentence in each historical question-answer pair and the current round question-answer is calculated by dot-product attention. The attention weight coefficient is calculated as follows: Where, is the mapping matrix, is the attention weight corresponding to the answer sentence in the i-th round of question-answer pair; c) Then, based on the attention weights, the feature vector representations of the answer sentences in all rounds are weighted and summed to obtain the feature representation c of the entire historical conversation information. h : d) Finally, we obtain the feature representation of historical session information c h Then, it is concatenated with the feature vector of each word output by the reading comprehension model to obtain the fused feature vector: In the above formula, c cls ,c q1 ,...,c qn ,c [sep] ,c d1 ,...,c dm ,c sep That is, the word feature vector representation that integrates historical conversation information.

2. The event extraction method based on a machine reading comprehension model according to claim 1, characterized in that: The annotation process for explicitly annotating entity information in an input sentence is as follows: a new sentence is obtained by adding entity type symbols before and after the entity word.

3. The event extraction method based on a machine reading comprehension model according to claim 1, characterized in that: When constructing questions for event role extraction, a question consists of two parts: question type and question content. The question type indicates what type of interrogative pronoun should be used; the event role corresponds to the corresponding question code; and the question content is constructed according to a template. The question generation template is as follows: "(Where / Which / What) is the_③_in the_④_eventtrigger by_②_?", where the space ③ is the event role type, the space ④ is the event type, and the space ② is the trigger word.

Images