A document-level event argument extraction method based on sequence labeling

Abstract

The present invention claims a method for extracting document-level event arguments based on sequence annotation, comprising the following steps: obtaining Wikipedia prior knowledge related to corpus entities, generating word span entity semantics enhanced embedding representation; concatenating the word span entity semantics enhanced embedding representation with the context representation obtained by a pre-trained language model to obtain a word vector input for the embedding layer; inputting the word representation into a multi-span bidirectional recurrent neural network to obtain a multi-span context feature representation of the word; inputting the multi-span context feature representation into a context attention mechanism and a gated attention mechanism module to obtain a context semantics fusion feature representation of the word; finally, using sequence annotation to extract event arguments from the output feature representation, and using the optimal model obtained by training to extract event arguments from unknown documents; the present invention effectively improves the effect of document-level event argument extraction by integrating prior knowledge and multi-span contextual semantic feature representations.

Description

Technical Field

[0001] The present invention belongs to the field of information extraction in natural language processing and proposes a document-level event argument extraction model based on sequence annotation. The model can provide basic services for tasks such as knowledge graph construction, relationship extraction, information retrieval, and automatic question answering. Background Art

[0002] The components of an event include trigger words, event types, arguments, and argument roles. The goal of event extraction is to automatically obtain the above information from unstructured information and present it in a structured manner. However, most traditional event extraction tasks are based on the sentence level, which has obvious defects: an event involves trigger words and multiple arguments. Usually, it is rare for an event trigger word and its related arguments to appear in the same sentence. Therefore, studying document-level event extraction to obtain long-sequence complete event information is more conducive to the practical application of event extraction.

[0003] The task of document-level event argument extraction is to identify event arguments and corresponding event role types in the document based on pre-defined event types and their corresponding role types. In documents, event argument information mostly appears in the form of entities. Therefore, accurate understanding of entity semantics is conducive to the model to better complete the task of event argument extraction. In the past, event extraction mostly used entity type features and their subtype features, ignoring the semantic deviation between the entity as a semantic whole and the vocabulary that constitutes the entity.

[0004] The complete argument information of document-level events is scattered in multiple sentences of the document, and it is difficult to extract event information through sentence-level event extraction methods. Therefore, the document-level event argument extraction task requires a good understanding of long-sequence semantics to make decisions. Recently, in the work related to document-level event argument extraction tasks, the pipeline method is used to train classifiers for event argument roles and related context detection separately. However, the pipeline method is prone to cascading errors, and the downstream tasks cannot feed back information to the upstream tasks to assist the upstream tasks in making decisions. The neural network sequence labeling method has been proven to be effective in extracting event information. The sequence labeling method is for multi-classification tasks at the word level, which can directly extract event arguments and match event argument roles at the same time, thereby avoiding the error transmission of the pipeline method and the one-way flow of task information. In addition, existing studies have shown that capturing the semantics of long-sequence text requires understanding both sentence semantics and inter-sentence semantics, but previous studies have not considered the impact of semantic information between words on the document-level event argument extraction task.

[0005] After searching, the patent application number is 202011506990.8, and its name is Document-level event argument extraction method based on sequence annotation. The method disclosed in the present invention uses unstructured document text as input data to construct a document-level event argument extraction model based on sequence annotation, which includes two stages. The first stage is to optimize the entity semantic embedding of word spans through prior knowledge as the word representation of the embedding layer, and the second stage is to obtain the contextual semantic features of sentence spans and paragraph spans in the document through feature extractors and double-layer attention mechanisms, and use them to annotate event arguments in the document. In both stages, the contextual semantic information of multiple spans is simultaneously integrated to generate more accurate event argument extraction results. For the commonly used document-level event argument extraction dataset, the method of the present invention is significantly better than the currently available document-level event argument extraction method, proving that the contextual semantic information of multiple spans integrated into the document is helpful for extracting event arguments from the document. Summary of the invention

[0006] The present invention aims to solve the above problems of the prior art. A method for extracting event arguments at the document level based on sequence annotation is proposed. The technical solution of the present invention is as follows:

[0007] A method for extracting document-level event arguments based on sequence labeling comprises the following steps:

[0008] Step 1: Perform entity recognition on the document corpus and obtain Wikipedia prior knowledge for the entities identified in the document;

[0009] Step 2: Extract the semantic attribute words of each entity through the prior knowledge obtained in step 1, and use the entity semantic attribute words to obtain the word span semantic enhanced embedding representation;

[0010] Step 3: Concatenate the word span entity semantics enhanced embedding representation obtained in step 2 with the contextual semantic representation obtained by the pre-trained language model bert_base to obtain the embedding layer word representation of each word;

[0011] Step 4: Input the embedding layer word representation obtained in step 3 into the recurrent neural network, and pass it through two 3-layer BiLSTM feature extractors for sentence span and paragraph span to obtain the context word representation of each word in sentence span and paragraph span;

[0012] Step 5: Input the context word representations of the two spans obtained in step 4 into the context attention mechanism modules of sentence span and paragraph span respectively to obtain the unequal feature representations of each word in different context spans;

[0013] Step 6: Input the sentence span and paragraph span feature representations obtained in step 5 into the gated attention mechanism module for feature fusion to obtain the final document multi-span context semantic fusion feature representation;

[0014] Step 7: Use the multi-span contextual semantic fusion feature representation obtained in step 6 as the input of the CRF sequence annotation layer, use the BIO annotation format to annotate the event arguments and their role types, train to obtain the optimal model, and finally use the trained extraction model to extract the event arguments from the test set documents of the corpus.

[0015] Furthermore, the step 1 specifically includes the following steps: first, the spaCy tool is used to perform entity recognition on the document, and then Wikipedia and entity-related prior knowledge are crawled through the recognized entities.

[0016] Furthermore, crawling Wikipedia's prior knowledge related to the entity through the identified entity specifically includes: crawling the explanatory articles on the Wikipedia website corresponding to the entity in the document using crawler technology, taking the first paragraph thereof and mapping it with the entity as the prior knowledge corpus.

[0017] Furthermore, in step 2, the semantic attribute words of each entity are extracted through prior knowledge, and the entity semantic attribute words are used to obtain the word span semantic enhanced embedding representation, which specifically includes:

[0018] Through the prior knowledge corpus obtained in step 1, extract and filter the entity semantic word set for each entity. Assuming that each entity has N semantic type words, the semantic type word set of entity e is represented by D e ,s i ∈D e is the i-th semantic attribute word of entity e, is the entity semantic attribute word s i GloVe embedding of entity e, and then generate the entity semantic embedding e of entity e d :

[0019]

[0020] The parameter α is used to control the weight of the entity semantic embedding. For each word w that constitutes the entity j , is the word w that constitutes entity e j GloVe embedding, whose word span semantics enhanced embedding representation s as follows:

[0021]

[0022] Furthermore, the step 3: concatenates the word span entity semantics enhanced embedding representation obtained in step 2 with the contextual semantic representation obtained by the pre-trained language model bert_base to obtain the embedding layer word representation of each word, specifically including:

[0023] For each word x i , and obtained its word span entity semantics enhanced embedding The contextual representations of sentence spans and paragraph spans generated using BERT-base are used for each word x i , has its context representation The final embedding layer word representation is concatenated by the entity semantics enhanced word embedding and the context word representation.

[0024] Furthermore, in step 4, the specific method of representing the sentence span and paragraph span context of each word is:

[0025] First, two 3-layer Bi-LSTM encoders are established, namely BiLSTM sent. and BiLSTM para. , and then divide all documents in the MUC-4 dataset into individual sentences s1,s2,...,s n In order to construct the paragraph span training dataset, the average number of paragraphs of all documents in the dataset is calculated, denoted as m. Starting from each sentence i, m consecutive sentences s are i to i +m-1 are connected as a paragraph to form overlapping candidate sequences of length m. Sequence 1 consists of {s1,s2,s3}, sequence 2 consists of {s2,s3,s4}, and so on. In order to construct the paragraph span development dataset and test dataset, we only need to group the consecutive m sentences in order to generate n / p sequences.

[0026] Sentence span context semantic representation extraction, the sentence span context features of each word in the paragraph It is expressed as follows:

[0027]

[0028] Paragraph span context semantic representation extraction, which extracts the paragraph context features of each word in the paragraph It is expressed as follows:

[0029]

[0030] Furthermore, in step 5, the specific method for representing the unequal features of each word in different context spans is:

[0031] The word context hidden layer representation of its sentence span is obtained through step 4 The single word representation vector of the sentence span is recorded as A bilinear sentence context attention mechanism is introduced into the word representation vector of each word to obtain context-related semantic information that is more important to the sentence, as follows:

[0032]

[0033]

[0034]

[0035] Among them, u si It means that a single word in a sentence represents Represented by the context words in the sentence The correlation of W s represents the learnable weight matrix, b s represents the bias, a si It represents the importance of each word in the sentence context. sent. It represents the weight and contribution of each word in the sentence to the context of the sentence span;

[0036] Similar to the sentence context attention mechanism, the word context hidden layer representation of the paragraph span obtained from step 4 The single word representation vector in the paragraph span is recorded as Word representation vector for each word The bilinear paragraph context attention mechanism is introduced to obtain context-related semantic information that is more important to the paragraph, as follows:

[0037]

[0038]

[0039]

[0040] Among them, u pi It means a single word in a paragraph. Represented with context words in the paragraph The correlation of W p represents the learnable weight matrix, b p represents the bias, a pi It represents the importance of each word in the paragraph in the context of the paragraph. para. It represents the weight and contribution of each word in the paragraph based on the semantic context of the paragraph span.

[0041] Furthermore, the specific steps of feature fusion in step 6 are:

[0042] For each word x i , using gated attention information fusion: gated aggregation calculates the gated vector g through the sentence span context word representation and the paragraph span context word representation i ,In order to control the information contribution from these two representations;

[0043] g i =σ(W c1 R sent. +W c2 R para. +b c )

[0044] Where W c1 , W c2 represents the learnable weight matrix, b c represents the bias; σ represents the sigmoid function;

[0045] Using g i and 1-g i As the sentence span representation R sent. and paragraph span indicates R para. The final gated information fusion representation R i For R sent. and R para. The weight and:

[0046]

[0047] in Indicates the corresponding multiplication operation of elements in the same position.

[0048] Furthermore, in step 7, the specific method of sequence labeling is:

[0049] In the document-level event argument extraction task, the CRF conditional random field is used to annotate the event arguments in the document in the BIO annotation format. CRF considers the dependency between adjacent labels. CRF uses the output of the linear layer as the emission probability matrix P of CRF. i，j represents the probability that the word at the i-th position is label j, A is the transfer matrix of CRF, It represents the transfer score from the i-th label to the i+1-th label. For a label sequence y={y1,y2,...,y k}, the score of the current sequence can be defined as:

[0050]

[0051] As can be seen from the above formula, the score of each word mapped to the label is determined by two parts, including the upper layer output and the transfer matrix of CRF. Softmax is used to calculate the normalized probability, and the formula is

[0052]

[0053] The objective function is optimized by maximizing the log-likelihood function. The log-likelihood of the training sample (x|y) is

[0054]

[0055] In the decoding stage, the dynamic programming Viterbi algorithm is used to find the rightmost path and obtain the probability of the corresponding label in the sequence. The label corresponding to the maximum probability is considered to be the correct label. The probability formula is:

[0056]

[0057] Furthermore, in step 7, the specific method of extracting event arguments of the test set documents is:

[0058] First, the test set documents are converted into symbol sequences through a word segmenter, and then the trained event argument extraction model is used to obtain the corresponding label sequence pairs in BIO format. The corresponding words in the test set documents are restored through the obtained label sequence pairs as the event argument extraction results of the test set documents.

[0059] The advantages and beneficial effects of the present invention are as follows:

[0060] The present invention innovatively proposes a document-level event argument extraction model based on multi-span contextual semantic fusion, which vertically integrates the entity enhancement semantics and contextual semantics of word spans, and horizontally extracts and dynamically integrates the contextual semantics of sentence spans and paragraph spans. Specifically, the present invention first explores the impact of word span entity semantic enhancement on the long-sequence event argument extraction model to optimize the contextual semantics of word spans, and uses the contextual attention mechanism and gated attention mechanism to dynamically integrate the contextual semantics of sentence spans and paragraph spans, while adopting a sequence labeling strategy to solve the problems of long-sequence semantic capture and document-level event argument dispersion. By improving and integrating multi-span contextual semantics, the effect of extracting event arguments from documents is improved.

[0061] The innovations of the present invention are mainly: proposing entity semantic enhancement of word span to optimize the contextual semantics of word span; proposing to use contextual attention mechanism and gated attention mechanism to extract and fuse important information in the context of sentence span and paragraph span to optimize the contextual semantics of sentence span and paragraph span. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The flowchart of the document-level event argument extraction method based on sequence labeling.

[0063] Figure 2 This is the macro-average result graph of document-level event argument extraction based on the MUC-4 dataset.

[0064] Figure 3 This is a diagram showing the extraction results of argument roles for various types of events based on the head noun phrase matching evaluation criteria.

[0065] Figure 4 This is a trend chart showing the impact of word span entity semantic enhancement by incorporating weight α on the extraction of event argument model. DETAILED DESCRIPTION

[0066] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0067] The technical solution of the present invention to solve the above technical problems is:

[0068] The document-level event argument extraction task is to identify event arguments and corresponding event roles in the document based on pre-defined event types and their corresponding role types.

[0069] The document-level event argument extraction method based on sequence annotation proposed in the present invention adopts the commonly used document-level event argument extraction dataset MUC-4 in the implementation stage. The present invention adopts the same data division standard, that is, using 1300 documents as the training set, 200 documents (TST1+TST2) for the development set and 200 documents (TST3+TST4) as the test set.

[0070] The present invention adopts two evaluation criteria: head noun phrase matching and exact matching. The criterion for head noun phrase matching is that if the first noun matches, the tag matching is successful, and the exact matching is that the predicted word and the template annotation word completely match, which is conducive to evaluating the ability of the model to match event element boundaries. The experimental results of the present invention give precision (P), recall (R) and F-1 score, and also give the macro average of all event argument experimental results.

[0071] Figure 1 It is the technical roadmap of the present invention, such as Figure 1 As shown, the method includes:

[0072] Step 1: Obtain Wikipedia prior knowledge of document corpus entities;

[0073] The goal of this stage is to obtain prior knowledge of the document corpus so as to optimize the representation of the contextual semantics of word spans. First, spaCy is used to perform entity recognition on the entire MUC-4 dataset, and an entity set is obtained through manual screening. Then, the corresponding Wikipedia entity explanatory articles are crawled through the obtained entity set.

[0074] Step 2: Generate word span entity semantics enhanced embedding representation;

[0075] In step 1, we obtain the Wikipedia-related explanatory articles of the entity. Next, we use this Wikipedia prior knowledge to generate a semantically enhanced embedding representation of the word span entity.

[0076] Through observation, it is found that there are semantic type words associated with entity semantics in the first paragraph of Wikipedia's entity-related explanation articles. First, all entity-related Wikipedia explanation articles are crawled for the entities extracted in the previous article, and the frequency of nouns appearing in the first paragraph of Wikipedia articles is calculated. Here, the present invention manually selects a few seed words, and uses spacy to calculate word similarity to expand these seed words, and finally obtains a dictionary containing 1762 entity semantic attribute words.

[0077] Generating word span entity semantic enhanced embedding aggregation representation first uses the obtained semantic attribute words of each entity to generate entity semantic embeddings, and then combines them with existing word embeddings to generate entity semantic enhanced embeddings. Each semantic attribute word of an entity can be regarded as a semantic component of the entity. Based on a set of semantic type words for each entity, an entity semantic embedding can be generated. The semantic type word of each entity is represented by a 100-dimensional GloVe pre-trained word embedding trained from 6B web crawlers. Assuming that each entity has N semantic type words, the set of semantic type words of entity e is represented as D e Then generate the entity semantic embedding e of entity e d as follows:

[0078]

[0079] where s i ∈D e is the i-th semantic attribute word of entity e, is the entity semantic attribute word s i GloVe embeddings of .

[0080] An entity semantic embedding is created for each entity by linearly aggregating entity semantic attribute word embeddings. For the word span entity semantics enhanced embedding, it is reasonable to assume that each word that constitutes the entity contains the semantics of the entity as a whole, so the word span entity semantics enhanced embedding is represented as a weighted sum of the entity semantic embedding and the GloVe word embedding of each word that constitutes the entity. The parameter α is used to control the weight of the entity semantic embedding. For each word w that constitutes the entity j , whose word span semantically enhanced embedding representation e s as follows:

[0081]

[0082] in is the word w that constitutes entity e j GloVe embeddings.

[0083] Step 3: Generate embedding layer word representation;

[0084] Contextual embeddings generated by pre-trained language models have been shown to be able to model context beyond sentence boundaries and improve performance on various tasks.

[0085] In the embedding layer of our invention, for each word x in the sentence i ,The concatenation matrix of word span entity semantics enhanced embedding and context word representation of pre-trained language model is adopted. However, the pre-trained language model will slice words in the preprocessing stage, which will increase the generation of OOV words. It is assumed that the slices of words that constitute the entity also have the complete semantics of the whole word. Based on this, an embedding dictionary containing 9326 word span entity semantics enhanced embeddings is generated, which contains the representation of word span entity semantics enhanced embedding of word slices.

[0086] Henceforth, for each word x i , the word span entity semantics enhanced embedding is obtained from step 2 Here, the present invention uses the contextual representation of sentence span and paragraph span generated by BERT-base. For each word x i , has its context representation The final embedding layer word representation is concatenated by the entity semantics enhanced word embedding and the context word representation.

[0087] Step 4: Generate context word representations of document sentence spans and paragraph spans;

[0088] Although the embeddings of words from different spans are the same, the context encodings of words in sentence span and paragraph span are different. In order to help the model better capture the context features between words in different span sequences in the document, the present invention establishes two 3-layer Bi-LSTM encoders, namely BiLSTM sent. and BiLSTM para. The present invention divides all documents in the MUC-4 dataset into single sentences s1, s2, ..., s n In order to construct the paragraph span training dataset, the average number of paragraphs of all documents in the dataset is calculated, denoted as m. Starting from each sentence i, m consecutive sentences s i to i +m-1 are connected as a paragraph to form overlapping candidate sequences of length m, where sequence 1 consists of {s1, s2, s3}, sequence 2 consists of {s2, s3, s4}, and so on. In order to keep the training set balanced, the present invention samples the same number of positive and negative sequences from the candidate sequences, where a positive sequence refers to a sequence containing at least one event argument, and a negative sequence refers to a sequence that does not contain an event argument. In order to construct the paragraph span development dataset and test dataset, it is only necessary to group the consecutive m sentences in order to generate n / p sequences.

[0089] Sentence span context semantic feature representation extraction, the present invention extracts the sentence span context feature of each word in the paragraph It is expressed as follows:

[0090]

[0091] The paragraph span context semantic feature representation extraction, the present invention extracts the paragraph context feature of each word in the paragraph It is expressed as follows:

[0092]

[0093] Step 5: Generate unequal feature representations of sentence span and paragraph span;

[0094] Through step 4, we get the hidden layer representation of the contextual semantic features of the sentence span. The single word representation vector in the sentence span is recorded as Because the importance of the context word representation of each word to the sentence is unequal, a bilinear sentence context attention mechanism is introduced for the word representation vector of each word to obtain context-related semantic information that is more important to the sentence, so as to make accurate decisions on the type of word. The details are as follows:

[0095]

[0096]

[0097]

[0098] where u si It means that a single word in a sentence represents Represented by the context words in the sentence The correlation of W s represents the learnable weight matrix, b s It represents the bias. si It represents the importance of each word in the sentence context. sent. It represents the weight and contribution of each word in the sentence to the context of the sentence span.

[0099] Similar to the sentence context attention mechanism, the hidden layer representation of the paragraph span context semantic features obtained from step 4 is The single word representation vector in the paragraph span is recorded as Word representation vector for each word The bilinear paragraph context attention mechanism is introduced to obtain context-related semantic information that is more important to the paragraph, so as to make accurate decisions on the type of words. The details are as follows:

[0100]

[0101]

[0102]

[0103] where u pi It means a single word in a paragraph. Represented with context words in the paragraph The correlation of W p represents the learnable weight matrix, b p It represents the bias. pi It represents the importance of each word in the paragraph in the context of the paragraph. para. It represents the weight and contribution of each word in the paragraph based on the semantic context of the paragraph span.

[0104] Step 6: Generate multi-span contextual semantic fusion feature representation;

[0105] For each word x i In order to aggregate the contextual semantic inequality representations learned by the sentence span and paragraph span obtained in step 5, gated attention information fusion is adopted: gated aggregation calculates the gated vector g through the sentence span context word representation and the paragraph span context word representation i , in order to control the information contribution from these two representations.

[0106] g i =σ(W c1 R sent. +W c2 R para. +b c )

[0107] Where W c1 , W c2 represents the learnable weight matrix, b c represents the bias. σ represents the sigmoid function.

[0108] The present invention adopts g i and 1-g i As the sentence span representation R sent. and paragraph span indicates R para. The final gated information fusion representation R i For R sent. and R para. The weight and

[0109]

[0110] in Indicates the corresponding multiplication operation of elements in the same position.

[0111] Step 7: Output the predicted label based on the fused feature representation and train the optimal model to extract the event arguments of the test set documents;

[0112] In the present invention, the CRF conditional random field is used to predict the labels of event arguments in the document in the BIO annotation format. CRF takes into account the dependencies between adjacent labels. CRF uses the output of the linear layer as the emission probability matrix P of CRF, P i，j represents the probability that the word at position i is label j. A is the transfer matrix of CRF, It represents the transfer score from the i-th label to the i+1-th label. For the label sequence y = {y1, y2, ..., y k}, the score of the current sequence can be defined as:

[0113]

[0114] From the above formula, we can see that the score of each word mapped to the label is determined by two parts, including the multi-span contextual semantic fusion feature representation obtained in step 6 and the transfer matrix of CRF. Softmax is used to calculate the normalized probability, and the formula is

[0115]

[0116] The objective function is optimized by maximizing the log-likelihood function. The log-likelihood of the training sample (x|y) is

[0117]

[0118] In the decoding stage, the dynamic programming Viterbi algorithm is used to find the rightmost path and obtain the probability of the corresponding label in the sequence. The label corresponding to the maximum probability is considered to be the correct label. The probability formula is:

[0119]

[0120] After training and learning the parameters of the overall extraction model using the known annotated data in the training set, the unknown documents to be extracted are used as the test set data to extract relevant event arguments.

[0121] For the documents in the test set, the same processing method as the training set data is used. The documents are converted into symbol sequences through the word segmenter, and then input into the previously trained extraction model to output the BIO label sequence pairs with the maximum likelihood probability. After obtaining the BIO label sequence, the words or phrases in the text are restored according to the BIO labels to obtain the event argument extraction results in the test set documents.

[0122] exist Figure 2 The macro-average experimental results using the head noun matching standard and the exact matching standard are shown in . Figure 3 More detailed experimental results for each event argument role are shown in .

[0123] In order to explore the importance of word-span entity semantic reinforcement and contextual attention semantic reinforcement in the document-level event argument extraction task, an entity semantic reinforcement model and a contextual attention mechanism semantic reinforcement model were set up respectively. In order to explore the impact of the proportion of entity semantic embeddings on the performance of the document-level event argument extraction model, the present invention has set up a variable experiment SADEAE-Word-span Entity Semantic reinforced Extract for the entity semantic proportion weight a, and took α as 0.1, 0.3, 0.5, 0.7, and 0.9 for experiments. The variable experimental results show that at the same time, word-level entity semantic reinforcement embedding effectively improves the effect of the neural network event argument extraction model. Figure 2 and Figure 3 Based on the experimental data, a macro average F-1 value trend chart based on head noun matching and a F-1 value trend chart for each event role type are plotted. Figure 4 ,With the increase of ,α, the overall effect of the model shows an ,experimental effect that first increases and then decreases, and the F-1 macro-average ,achieves the best effect when ,α = 0.7.

[0124] In order to explore the influence of semantic enhancement of contextual attention mechanism on sentence span and paragraph span of document-level event argument extraction model, the present invention sets up a model SADEAE-Sentence-Span Contextual Semantic that only incorporates sentence span contextual attention mechanism and a model SADEAE-Paragraph-Span Contextual Semantic that only incorporates paragraph-level contextual attention mechanism, and Figure 3 and Figure 4 The results of all experiments are shown in . The experimental results show that the introduction of the contextual attention mechanism has a significant effect on the contextual semantic extraction of the sentence span and paragraph span of the document.

[0125] exist Figure 2 and Figure 3 The last row shows the experimental results of the SADEAE, a document-level event extraction model based on sequence annotation proposed in the present invention, which integrates entity semantic enhancement and contextual attention semantic enhancement, and obtains multiple levels of semantics at the same time. It is found from the experimental results that the document-level event argument extraction method based on sequence annotation of the present invention can achieve better results than the currently best performing multi-granularity reader. By integrating word-level entity semantic enhancement and contextual semantics under a dual-layer attention mechanism, the F-1 value of the document-level event argument extraction model (SADEAE) based on sequence annotation has increased by more than 3 points on the basis of the previous optimal model, reaching a score of >63, with an accuracy increase of 3.85 and a recall increase of 4.71. This effectively proves that the model of the present invention has greatly improved its ability to match document-level event arguments and their boundaries.

[0126] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0127] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0128] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A document-level event argument extraction method based on sequence labeling, It is characterized in that The following steps are involved: Step 1: Perform entity recognition on the document corpus and obtain Wikipedia prior knowledge for the entities identified in the document; Step 2: Extract the semantic attribute words of each entity through the prior knowledge obtained in step 1, and use the entity semantic attribute words to obtain the word span semantic enhanced embedding representation; Step 3: Concatenate the word span entity semantics enhanced embedding representation obtained in step 2 with the contextual semantic representation obtained by the pre-trained language model bert_base to obtain the embedding layer word representation of each word; Step 4: Input the embedding layer word representation obtained in step 3 into the recurrent neural network, and pass it through two 3-layer BiLSTM feature extractors for sentence span and paragraph span to obtain the context word representation of each word in sentence span and paragraph span; Step 5: Input the context word representations of the two spans obtained in step 4 into the context attention mechanism modules of sentence span and paragraph span respectively to obtain the unequal feature representations of each word in different context spans; Step 6: Input the sentence span and paragraph span feature representations obtained in step 5 into the gated attention mechanism module for feature fusion to obtain the final document multi-span context semantic fusion feature representation; Step 7: Use the multi-span contextual semantic fusion feature representation obtained in step 6 as the input of the CRF sequence annotation layer, use the BIO annotation format to annotate the event arguments and their role types, train to obtain the optimal model, and finally use the trained extraction model to extract the event arguments from the test set documents of the corpus; In step 2, the semantic attribute words of each entity are extracted through prior knowledge, and the entity semantic attribute words are used to obtain the word span semantic enhanced embedding representation, which specifically includes: Through the prior knowledge corpus obtained in step 1, extract and filter the entity semantic word set for each entity. Assuming that each entity has N semantic type words, the semantic type word set of entity e is represented by D e ,s i ∈D e is the i-th semantic attribute word of entity e, is the entity semantic attribute word s i GloVe embedding of entity e, and then generate the entity semantic embedding e of entity e d : The parameter α is used to control the weight of the entity semantic embedding. For each word w that constitutes the entity j , is the word w that constitutes entity e j GloVe embedding, whose word span semantics enhanced embedding representation s as follows: In step 4, the specific method of representing the sentence span and paragraph span context of each word is: First, two 3-layer Bi-LSTM encoders are established, namely BiLSTM sent. and BiLSTM para. , and then divide all documents in the MUC-4 dataset into single sentences s 1 ,s 2 ,...,s n In order to construct the paragraph span training dataset, the average number of paragraphs of all documents in the dataset is calculated, denoted as m. Starting from each sentence i, m consecutive sentences s are i to i +m-1 are connected as a paragraph to form an overlapping candidate sequence of length m. Sequence 1 consists of {s 1 ,s 2 ,s 3 }, sequence 2 consists of {s 2 ,s 3 ,s 4 }, and so on; To construct the paragraph span development dataset and test dataset, we only need to group the consecutive m sentences in order to generate n / p sequences; Sentence span context semantic representation extraction, the sentence span context features of each word in the paragraph It is expressed as follows: Paragraph span context semantic representation extraction, which extracts the paragraph context features of each word in the paragraph It is expressed as follows: In step 5, the specific method of representing the unequal features of each word in different context spans is: The word context hidden layer representation of its sentence span is obtained through step 4 The single word representation vector of the sentence span is recorded as A bilinear sentence context attention mechanism is introduced into the word representation vector of each word to obtain context-related semantic information that is more important to the sentence, as follows: Among them, u si It means that a single word in a sentence represents Represented by the context words in the sentence The correlation, W s represents the learnable weight matrix, b s represents the bias, a si It represents the importance of each word in the sentence context. sent. It represents the weight and contribution of each word in the sentence to the context of the sentence span; Similar to the sentence context attention mechanism, the word context hidden layer representation of the paragraph span obtained from step 4 The single word representation vector in the paragraph span is recorded as Word representation vector for each word The bilinear paragraph context attention mechanism is introduced to obtain context-related semantic information that is more important to the paragraph, as follows: Among them, u pi It means a single word in a paragraph. Represented with context words in the paragraph The correlation, W p represents the learnable weight matrix, b p represents the bias, a pi It represents the importance of each word in the paragraph in the context of the paragraph. para. It represents the weight and contribution of each word in the paragraph based on the semantics of the paragraph span context.

2. According to the method for extracting document-level event arguments based on sequence labeling according to claim 1, It is characterized in that The step 1 specifically includes the following steps: firstly, the spaCy tool is used to perform entity recognition on the document, and then the prior knowledge related to the entity is crawled from Wikipedia through the recognized entity.

3. According to the method for extracting document-level event arguments based on sequence labeling according to claim 2, It is characterized in that The method of crawling Wikipedia's prior knowledge related to the entity through the identified entity specifically includes: crawling the explanatory articles on the Wikipedia website corresponding to the entity in the document using crawler technology, taking the first paragraph thereof and mapping it with the entity as the prior knowledge corpus.

4. According to the method for extracting document-level event arguments based on sequence labeling according to claim 1, It is characterized in that The step 3: concatenate the word span entity semantics enhanced embedding representation obtained in step 2 with the contextual semantic representation obtained by the pre-trained language model bert_base to obtain the embedding layer word representation of each word, specifically including: For each word x i , and obtained its word span entity semantics enhanced embedding The contextual representations of sentence spans and paragraph spans generated using BERT-base are used for each word x i , has its context representation The final embedding layer word representation is concatenated by the entity semantics enhanced word embedding and the context word representation.

5. According to the method for extracting document-level event arguments based on sequence labeling according to claim 1, It is characterized in that The specific steps of feature fusion in step 6 are: For each word x i , using gated attention information fusion: gated aggregation calculates the gated vector g through the sentence span context word representation and the paragraph span context word representation i ,In order to control the information contribution from these two representations; g i =σ(W c1 R sent. +W c2 R para. +b c ) Where W c1 , W c2 represents the learnable weight matrix, b c represents the bias; σ represents the sigmoid function; Using g i and 1-g i As the sentence span representation R sent. and paragraph span indicates R para. The allocation weight of The final gated information fusion representation R i For R sent. and R para. The weight and: in Indicates the corresponding multiplication operation of elements in the same position.

6. According to the method for extracting document-level event arguments based on sequence labeling according to claim 5, It is characterized in that In step 7, the specific method of sequence labeling is: In the document-level event argument extraction task, the CRF conditional random field is used to annotate the event arguments in the document in the BIO annotation format. CRF considers the dependency between adjacent labels. CRF uses the output of the linear layer as the emission probability matrix P of CRF. i，j represents the probability that the word at the i-th position is label j, A is the transfer matrix of CRF, It represents the transfer score from the i-th label to the i+1-th label. For a label sequence y={y 1 ,y 2 ,...,y k }, the score of the current sequence can be defined as: As can be seen from the above formula, the score of each word mapped to the label is determined by two parts, including the upper layer output and the transfer matrix of CRF. Softmax is used to calculate the normalized probability, and the formula is The objective function is optimized by maximizing the log-likelihood function. The log-likelihood of the training sample (x|y) is In the decoding stage, the dynamic programming Viterbi algorithm is used to find the rightmost path and obtain the probability of the corresponding label in the sequence. The label corresponding to the maximum probability is considered to be the correct label. The probability formula is:

7. The method for extracting document-level event arguments based on sequence labeling according to claim 6, It is characterized in that In step 7, the specific method of extracting event arguments of the test set documents is: First, the test set documents are converted into symbol sequences through a word segmenter, and then the trained event argument extraction model is used to obtain the corresponding label sequence pairs in BIO format. The corresponding words in the test set documents are restored through the obtained label sequence pairs as the event argument extraction results of the test set documents.

Images