An event argument extraction method based on event element interaction and label semantic enhancement

Abstract

The present invention discloses an event argument extraction method based on event element interaction and label semantic enhancement. First, the data set text is preprocessed to output the preprocessed text sentence and argument role set; secondly, the text is vectorized in combination with the pre-trained language model BERT, the context vector representation of the word is output, and the argument role label is vectorized using the word embedding technology at the same time; thirdly, the candidate argument span is identified using a bidirectional long short-term memory network and a feedforward neural network; then, an event element interaction module is constructed based on the attention mechanism to extract the semantic features of the correlation between event elements; finally, a label semantic enhancement module is constructed based on the attention mechanism to realize the argument role classification of the candidate argument. The present invention fully models the information interaction between event elements based on deep learning technology, uses label semantics to help argument role classification, and has good event argument extraction performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of information extraction, and in particular relates to an event argument extraction method based on event element interaction and label semantic enhancement. Background Art

[0002] In recent years, the Internet has developed rapidly, the size of my country's Internet users has also grown rapidly, and the Internet's influence on people's lives and society has also deepened. Compared with traditional information dissemination media, online media information has significant characteristics such as large scale, fast dissemination, rich information, strong real-time, and high participation. Online media generates a large amount of text data every day. Faced with the growing amount of Internet information, it is crucial to quickly locate specific events discussed by the public. This can not only help public opinion regulators quickly locate specific events and understand the specific elements of the events, but also provide event extraction results to other natural language processing tasks for more in-depth analysis and application.

[0003] The event argument extraction task is a key subtask of the event extraction task. Since the task itself involves the interaction of more elements and features, it is relatively complex, and event detection research has developed rapidly. Therefore, in recent years, event argument extraction has gradually become a bottleneck in the performance of event extraction systems. After extracting the trigger words in the text sentences and determining the classification of event types, event argument extraction aims to identify entities that serve as event arguments, that is, to identify the entities in the text that are actually involved in the event, and classify the roles they play in the event.

[0004] In recent years, event argument extraction has attracted the attention of many researchers and scholars and has made some progress. Early researchers usually used feature engineering-based methods to extract event arguments by manually designing complex features and heuristic rules. With the development of neural networks, more and more researchers have begun to use convolutional neural networks and recurrent neural networks for event argument extraction, in order to use the ability of neural networks to automatically extract local features or global sequence features of sentences. Since 2018, the pre-trained language model BERT has gradually entered the field of vision of researchers due to its outstanding performance in multiple natural language processing tasks.

[0005] Although the existing methods have made some progress, they still have some shortcomings, which hinder the further improvement of event argument extraction performance; for example, the existing event argument extraction methods usually consider each candidate argument separately, ignoring the relationship between arguments; the existing event argument extraction methods ignore the use of label semantic information of argument roles. Therefore, there is still much room for research in event argument extraction technology, so the present invention designs an event argument extraction method based on event element interaction and semantic label enhancement. Summary of the invention

[0006] Purpose of the invention: The present invention provides an event argument extraction method based on event element interaction and label semantic enhancement. On the basis that the event detection task has been completed, the pre-trained language model BERT, word embedding technology and bidirectional long short-term neural network, Transformer model, and attention mechanism are used to extract event arguments, and the method has good event argument extraction performance.

[0007] Technical solution: The present invention provides an event argument extraction method based on event element interaction and label semantic enhancement, comprising the following steps:

[0008] (1) Preprocess the pre-acquired data set text and output the preprocessed text sentences and argument role sets;

[0009] (2) Use the pre-trained language model BERT to vectorize the text, output the contextual representation of the word, and use word embedding technology to vectorize the argument role label;

[0010] (3) Using bidirectional long short-term memory networks and feedforward neural networks to identify candidate argument spans;

[0011] (4) Construct a span representation sequence of event elements spans = {trigger word, prior event type, candidate argument 1, ..., candidate argument m}, jointly model the trigger word related information and all candidate arguments, and draw on the multi-head self-attention mechanism of the Transformer model to simultaneously extract the correlation between the trigger word related information and the arguments and the correlation between the arguments;

[0012] (5) An attention network layer ATT is set up to measure the correlation between candidate arguments and argument role labels, so that the label semantic features of argument roles can be explicitly used to guide argument role classification.

[0013] Furthermore, the step (1) comprises the following steps:

[0014] (11) extracting all argument role types according to the predefined event ontology and organizing them into a preprocessed argument role set;

[0015] (12) Split the XML document in the dataset text into individual text sentences;

[0016] (13) According to the tag correspondence in the XML document, each preprocessed text sentence is organized.

[0017] Furthermore, the step (2) comprises the following steps:

[0018] (21) Organize the preprocessed text sentences into the form of <event type, text sentence> sentence pairs, specifically {[CLS], Event_Type, [SEP], S, [SEP]}, where [CLS] and [SEP] are special symbols, Event_Type represents the event type of the text sentence, and S represents the text sentence itself; input the pre-trained language model BERT to vectorize the text;

[0019] (22) The preprocessed argument role set is used to vectorize the argument role labels using the word2vec technology and the embedding layer.

[0020] Furthermore, the step (3) comprises the following steps:

[0021] (31) Input the vectorized text sentence into the bidirectional long short-term memory network BiLSTM, extract the sequence features of the text sentence, and output the word vector sequence after the sequence feature extraction;

[0022] (32) Using two independent feed-forward neural networks FFNN with Sigmoid activation function, each word represents x i , that is, the i-th sequence context feature vector output by BiLSTM, passes through two feedforward neural networks, and then uses the Sigmoid activation function, which is expressed as:

[0023] p s (x i )=Sigmoid(FFNN s (x i ))

[0024] p e (x i )=Sigmoid(FFNN e (x i ))

[0025] According to p s (x i ) and p e (x i ) exceeds a preset probability threshold, thereby determining whether the word is the start position or the end position of the candidate argument span, and completing binary classification;

[0026] Adopt the nearest candidate argument start-candidate argument end pair matching principle to form all candidate argument spans; adopt binary cross entropy loss Loss start and Loss end As the objective function of argument span binary classification, it is expressed as:

[0027] Loss start=BinaryCrossEntropy(Y s , P s )

[0028] Loss end =BinaryCrossEntropy(Y e , P e )

[0029] Among them, Y s represents the actual starting position of the argument span in the text sentence S, Y s 1 in the word represents the start position, and 0 represents not the start position; e represents the actual end position of the argument span in the text sentence S, where 1 represents the end position of the word and 0 represents it is not the end position; P s and P e They represent the probability of each word in the text sentence S being predicted as the start position and the end position respectively.

[0030] Furthermore, the step (4) comprises the following steps:

[0031] (41) Based on the interval range provided by the candidate argument start-candidate argument end pair, the trigger words, prior event types and all candidate arguments are represented in span using the text vectors vectorized by the pre-trained language model BERT; the word vectors of each span interval are summed and averaged;

[0032] (42) All span representations are connected to obtain the input sequence spans, which is input into the Transformer model;

[0033] (43) The semantic information of the input sequence spans is feature encoded with the help of the self-attention mechanism of the Transformer model;

[0034] (44) The Transformer model outputs the feature representation H of argument i i .

[0035] Furthermore, the step (5) comprises the following steps:

[0036] (51) taking the argument vector representation and the label embedding vector representation as input;

[0037] (52) After calculating the attention scores of the argument vector representation and the label embedding vector table, the attention score of the argument specific to each argument role label is obtained; specifically, for an argument span representation H output by the Transformer model i , and argument role labels are embedded in Table E l={l1,l2,…,l |R|}, where |R| represents the number of argument roles; the attention network ATT is based on H i and E l As input, calculate H i For each label l j The attention weight coefficient α j , expressed as:

[0038] s(l j , H i )=v T tanh(Wl j +UH i +b j )

[0039]

[0040] Among them, W and U are learnable parameter matrices, b j is a bias term; the model selects the normalized attention score α j The highest category is used as the argument role of the argument; cross entropy loss Loss is used role As the objective function of argument role classification, it is expressed as:

[0041] Loss role =CrossEntropy(Y r , P r )

[0042] Among them, Y r represents the actual argument role in the text sentence S, Y r 1 in P means that the argument plays a role, and 0 means that it does not. r Represents the probability that each word in the text sentence S is predicted to be a variety of argument roles; the training goal is to minimize the cross entropy loss Loss of argument span recognition start 、Loss end And the cross entropy loss Loss for argument role classification role The AdamW optimizer is used to update the model parameters during training, and the Dropout mechanism is used to prevent overfitting.

[0043] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention: Compared with traditional methods, the present invention utilizes deep learning technologies such as pre-trained language models, neural networks, etc. to fully model the information interaction between event elements, and utilizes label semantics to assist in argument role classification. It aims at the problem that the existing event argument extraction methods based on deep learning do not make sufficient use of event element interactions and do not mine label features, resulting in low extraction performance. The present invention can effectively extract the semantic features of the correlation between event elements, and effectively utilize the semantic features of argument role labels to guide the argument role classification process. The present invention is an effective method for event argument extraction, and has good event argument extraction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the present invention;

[0045] Figure 2 Schematic diagram of the framework of the event argument extraction model established in the present invention;

[0046] Figure 3 This is a schematic diagram of the attention network ATT calculation of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0048] This paper proposes an event argument extraction method based on event element interaction and label semantic enhancement, and constructs Figure 2 The event argument extraction model shown in the figure includes an encoding module, an argument span recognition module, an event element interaction module, and a label semantic enhancement module; the encoding module combines the pre-trained language model BERT to vectorize the text, output the contextual representation of the word, and use the word embedding technology to vectorize the argument role label; the argument span recognition module uses a bidirectional long short-term memory network and a feedforward neural network to identify the candidate argument span; the event element interaction module constructs the span representation sequence of event elements spans = {trigger word, prior event type, candidate argument 1, ..., candidate argument m}, jointly models the trigger word related information and all candidate arguments, and draws on the multi-head self-attention mechanism of the Transformer model to extract the correlation between the trigger word related information and the argument and the correlation between the argument; the label semantic enhancement module sets an attention network layer ATT to measure the correlation between the candidate argument and the argument role label, so as to explicitly guide the argument role classification with the label semantic features of the argument role. Figure 1 As shown, the specific steps include:

[0049] S1: Preprocess the data set text and output the preprocessed text sentences and argument role sets.

[0050] According to the predefined event ontology, all argument role types are extracted and organized into a preprocessed argument role set. Taking the ACE2005 English corpus as an example, the event ontology defines 35 event argument roles. The argument role set obtained by 1-1 preprocessing is a list of length 35: [Attacker, Time-At-Beginning, Seller, Victim, …, Sentence].

[0051] The XML documents in the dataset text are split into individual text sentences. According to the label correspondence in the XML document, each pre-processed text sentence is organized. For example, "British troops control the outskirts of the town, and today destroyed another building." According to the label correspondence, the pre-processed text sentence identifies "destroyed" as the trigger word, the event type is "Conflict: Attack", and the standard arguments include: British troops, another building, and today, and the argument roles are Attacker, Target, and Time-Within.

[0052] S2: Combine the pre-trained language model BERT to vectorize the text, output the contextual representation of the word, and use word embedding technology to vectorize the argument role label, such as Figure 2 The coding module is shown below.

[0053] The pre-processed text sentences are organized into the form of <event type, text sentence> sentence pairs, specifically {[CLS], Event_Type, [SEP], S, [SEP]}, where [CLS] and [SEP] are special symbols, Event_Type represents the event type of the text sentence, and S represents the text sentence itself. The pre-trained language model BERT is input to vectorize the text. Taking the text sentence "British troops control the outskirts of the town, and today destroyed another building." as an example, the text input to BERT will be {[CLS], Attack, SEP], British, troops, control, the, outskirts, of, the, town, and, today, destroyed, another, building, [SEP]}.

[0054] The preprocessed argument role set is vectorized using word2vec technology and the embedding layer Embedding. For example, the object to be vectorized by the embedding layer Embedding is the 35 event argument roles defined in the event ontology [Attacker, Time-At-Beginning, Seller, Victim, …, Sentence].

[0055] S3: Use the bidirectional long short-term memory network and feedforward neural network to identify candidate argument spans, such as Figure 2 The argument span recognition module is shown in Figure 2.

[0056] The vectorized text sentence is input into the bidirectional long short-term memory network BiLSTM, the sequence features of the text sentence are extracted, and the word vector sequence after sequence feature extraction is output. Taking the text sentence "British troops control the outskirts of the town, and today destroyed another building." as an example, the word vector sequence obtained after sequence feature extraction in this step is Where n represents the length of the text sentence, h BiLSTM Represents the word vector sequence corresponding to the text sentence output by BiLSTM, h t Represents the vector representation of the trigger word in the text sentence, and the rest of h i (i∈[1,t)∪(t,n]) represents the word vector representation of the corresponding position of the text sentence except the trigger word.

[0057] Using two independent feed-forward neural networks FFNN with Sigmoid activation function, each word is represented by x i (i.e., the i-th sequence context feature vector output by BiLSTM) passes through two feedforward neural networks respectively, and then uses the Sigmoid activation function respectively, which is expressed as:

[0058] p s (x i )=Sigmoid(FFNN s (x i ))

[0059] p e (x i )=Sigmoid(FFNN e (x i ))

[0060] Finally, according to p s (x i ) and p e(x i ) exceeds the pre-set probability threshold, thereby determining whether the word is the start or end position of the candidate argument span to complete binary classification. The nearest candidate argument start-candidate argument end pair matching principle is adopted to form all candidate argument spans. Binary cross entropy loss Loss is used start and Loss end As the objective function of argument span binary classification, it is expressed as:

[0061] Loss starf =BinaryCrossEntropy(Y s , P s )

[0062] Loss end =BinaryCrossEntropy(Y e , P e )

[0063] Among them, Y s represents the actual starting position of the argument span in the text sentence S, Y s 1 in the code represents the start position of the word, and 0 represents it is not the start position. e Represents the actual end position of the argument span in the text sentence S, where 1 represents the end position of the word and 0 represents it is not the end position. s and P e Respectively represent the probability of each word in the text sentence S being predicted as the start position and the end position. For example, the probability threshold is selected as the default value of the Sigmoid function 0.5, the candidate argument start position list is (0,10,12), the candidate argument end position list is (1,10,13), and the nearest candidate argument start-candidate argument end pair matching principle is adopted, and finally three candidate argument spans (0,1), (10,10) and (12,13) ​​are obtained, corresponding to (British, troops), (today), (another building), respectively.

[0064] S4: Construct a span representation sequence of event elements spans = {trigger word, prior event type, candidate argument 1, ..., candidate argument m}, jointly model the trigger word related information and all candidate arguments, and draw on the multi-head self-attention mechanism of the Transformer model to extract the correlation between the trigger word related information and the arguments and the correlation between the arguments, such as Figure 2 The event element interaction module is shown.

[0065] According to the interval range provided by the candidate argument start-candidate argument end pair, the trigger words, prior event types, and all candidate arguments are represented by spans using the text vectors vectorized by the pre-trained language model BERT. The word vectors within each span interval (interval left closed and right closed) are summed and averaged.

[0066] To find the "candidate argument" here, we first find all the legal start (that is, the starting position) and end (end position), and then the start and end need to be paired. The pairing principle adopted is "nearest pairing", that is, the nearest start and the nearest end will form a pair, and other pairs are also formed according to this principle.

[0067] All span representations are connected to obtain the input sequence spans, which are input into the Transformer model. Taking the text sentence "British troops control the outskirts of the town, and today destroyed another building." as an example, the spans obtained are {B(destroyed), B(Attack), AVG(B(British)+B(troops)), B(today), AVG(B(another)+B(building))}, where B(·) represents the word feature representation output by the pre-trained language model BERT, and AVG(·) represents the sum and average operation.

[0068] The Transformer model’s self-attention mechanism is used to encode the semantic information of the input sequence spans. The Transformer model outputs the feature representation H of argument i. i Taking the sentence “British troops control the outskirts of the town, and today destroyed another building.” as an example, the feature representations of the three arguments are output, namely H Britishtroops ,H today ,H anotherbuilding .

[0069] S5: Set up an attention network layer ATT to measure the correlation between candidate arguments and argument role labels, so that the label semantic features of argument roles can be used to explicitly guide argument role classification, such as Figure 2 The label semantic enhancement module is shown below.

[0070] The argument vector representation and label embedding vector representation are used as input; taking the text sentence "British troops control the outskirts of the town, and today destroyed another building." as an example, the feature representations of the three arguments are H British troops ,H today ,H another building , with H British troops For example, and the argument role label embedding table E l together as input.

[0071] By calculating the attention scores of the argument vector representation and the label embedding vector table, the attention score of the argument specific to each argument role label is obtained. Specifically, for an argument span representation H output by the Transformer model i , and argument role labels are embedded in Table E l ={l1,l2,…,l |R|}, where |R| represents the number of argument roles. Figure 3 As shown, the attention network ATT is based on H i and E l As input, calculate H i For each label l j The attention weight coefficient α j , expressed as:

[0072] s(l j , H i )=v T tanh(Wl j +UH i +b j )

[0073]

[0074] Among them, W and U are learnable parameter matrices, b j is a bias term. The model selects the normalized attention score α j The highest (i.e., the most probable) category is used as the argument role of the argument. Cross entropy loss is used. role As the objective function of argument role classification, it is expressed as:

[0075] Loss role =CrossEntropy(Y r , P r )

[0076] Among them, Y r represents the actual argument role in the text sentence S, Yr 1 in P means that the argument plays a role, and 0 means that it does not. r Represents the probability that each word in the text sentence S is predicted to be a variety of argument roles.

[0077] The goal of training is to minimize the cross entropy loss Loss of argument span recognition start 、Loss end And the cross entropy loss Loss for argument role classification ro1e The AdamW optimizer is used to update the model parameters during training, and the Dropout mechanism is used to prevent overfitting.

[0078] Take the sentence “British troops control the outskirts of the town, and today destroyed another building.” as an example, output the feature representation of the three arguments, which are: H British troops ,H today ,H another bui1ding , with H British troops For example, the attention score is calculated by embedding the argument role label in Table E1, and the highest attention score α is obtained. j When j is 0, it corresponds to the argument role label Attacker. Therefore, the argument role label assigned to the argument span (0,1) is Attacker, that is, the role played by "British troops" in this "Conflict:Attack" event instance is "Attacker". In addition, during training, the learning rate of the AdamW optimizer is 0.00003 and the Dropout rate is 0.2.

[0079] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and variations can be made without departing from the technical principles of the present invention, and these improvements and variations should also be considered as the protection scope of the present invention. The method of the present invention can be compiled into a program code, the code is stored in a computer storage medium, the program code is transmitted to a processor, and the method of the present invention is executed by the processor.

Claims

1. An event argument extraction method based on event element interaction and label semantic enhancement, characterized in that: The following steps are involved: (1) Preprocess the pre-acquired data set text and output the preprocessed text sentences and argument role sets; (2) Use the pre-trained language model BERT to vectorize the text, output the contextual representation of the word, and use word embedding technology to vectorize the argument role label; (3) Using bidirectional long short-term memory networks and feedforward neural networks to identify candidate argument spans; (4) Construct a span representation sequence of event elements spans = {trigger word, prior event type, candidate argument 1, ..., candidate argument m}, jointly model the trigger word related information and all candidate arguments, and draw on the multi-head self-attention mechanism of the Transformer model to simultaneously extract the correlation between the trigger word related information and the arguments and the correlation between the arguments; (5) Setting an attention network layer ATT to measure the correlation between candidate arguments and argument role labels, so as to explicitly guide argument role classification with the label semantic features of argument roles; The step (3) comprises the following steps: (31) Input the vectorized text sentence into the bidirectional long short-term memory network BiLSTM, extract the sequence features of the text sentence, and output the word vector sequence after the sequence feature extraction; (32) Using two independent feed-forward neural networks FFNN with Sigmoid activation function, each word represents x i , that is, the i-th sequence context feature vector output by BiLSTM, passes through two feedforward neural networks, and then uses the Sigmoid activation function, which is expressed as: p s (x i )=Sigmoid(FFNN s (x i )) p e (x i )=Sigmoid(FFNN e (x i )) According to p s (x i ) and p e (x i ) exceeds a preset probability threshold, thereby determining whether the word is the start position or the end position of the candidate argument span, and completing binary classification; Adopt the nearest candidate argument start-candidate argument end pair matching principle to form all candidate argument spans; adopt binary cross entropy loss Loss start and Loss end As the objective function of argument span binary classification, it is expressed as: Loss start =BinaryCrossEntropy(Y s ,P s ) Loss end =BinaryCrossEntropy(Y e ,P e ) Among them, Y s represents the actual starting position of the argument span in the text sentence S, Y s 1 in the word represents the start position, and 0 represents not the start position; e represents the actual end position of the argument span in the text sentence S, where 1 represents the end position of the word and 0 represents it is not the end position; P s and P e Respectively represent the probability of each word in the text sentence S being predicted as the start position and the end position; The step (4) comprises the following steps: (41) Based on the interval range provided by the candidate argument start-candidate argument end pair, the trigger words, prior event types and all candidate arguments are represented in span using the text vectors vectorized by the pre-trained language model BERT; the word vectors of each span interval are summed and averaged; (42) All span representations are connected to obtain the input sequence spans, which is input into the Transformer model; (43) The semantic information of the input sequence spans is feature encoded with the help of the self-attention mechanism of the Transformer model; (44) The Transformer model outputs the feature representation H of argument i i ; The step (5) comprises the following steps: (51) taking the argument vector representation and the label embedding vector representation as input; (52) After calculating the attention scores of the argument vector representation and the label embedding vector table, the attention score of the argument specific to each argument role label is obtained; specifically, for an argument span representation H output by the Transformer model i , and argument role labels are embedded in Table E l ={l1,l2,...,l |R| }, where |R| represents the number of argument roles; the attention network ATT is based on H i and E l As input, calculate H i For each label l j The attention weight coefficient α j , expressed as: s(l j ,H i )=v T tanh(Wl j +UH i +b j ) Among them, W and U are learnable parameter matrices, b j is a bias term; the model selects the normalized attention score α j The highest category is used as the argument role of the argument; cross entropy loss Loss is used role As the objective function of argument role classification, it is expressed as: Loss role =CrossEntropy(Y r ,P r ) Among them, Y r represents the actual argument role in the text sentence S, Y r 1 in P means that the argument plays a role, and 0 means that it does not. r Represents the probability that each word in the text sentence S is predicted to be a variety of argument roles; the training goal is to minimize the cross entropy loss Loss of argument span recognition start 、Loss end And the cross entropy loss Loss for argument role classification role The AdamW optimizer is used to update the model parameters during training, and the Dropout mechanism is used to prevent overfitting.

2. The event argument extraction method based on event element interaction and label semantic enhancement according to claim 1 is characterized in that: The step (1) comprises the following steps: (11) extracting all argument role types according to the predefined event ontology and organizing them into a preprocessed argument role set; (12) Split the XML document in the dataset text into individual text sentences; (13) According to the tag correspondence in the XML document, each preprocessed text sentence is organized.

3. The event argument extraction method based on event element interaction and label semantic enhancement according to claim 1 is characterized in that: The step (2) comprises the following steps: (21) Organize the preprocessed text sentences into the form of <event type, text sentence> sentence pairs, specifically in the form of {[CLS], Event_Type, [SEP], S, [SEP]}, where [CLS] and [SEP] are special symbols, Event_Type represents the event type of the text sentence, and S represents the text sentence itself; input the pre-trained language model BERT to vectorize the text; (22) The preprocessed argument role set is used to vectorize the argument role labels using the word2vec technology and the embedding layer.

Images