Event argument extraction method based on event co-occurrence prefix

Abstract

The invention belongs to the field of event argument extraction, and provides an event co-occurrence prefix-based event argument extraction method, which comprises the steps of BART-Marge model construction, coding intermediate key and coding intermediate vector extraction, template coding unit coding, prefix splicing, text coding, network output calculation, vector Infoext acquisition, bias addition and event argument extraction. According to the method, semantic information of all events existing in input is extracted, and prefixes are extracted and added, so that the capability of processing a complex event relation by a model is enhanced; through the arrangement for introducing two Info-Qs, the constraint of the model on the output format is enhanced, and the disorder of the output format of the model is prevented; by introducing an adjustable copying mechanism, the problem of potential noise caused by introduction of extra information in attention calculation of each layer is solved.

Description

Technical Field

[0001] The present invention relates to the field of event argument extraction, and in particular to an event argument extraction method based on event co-occurrence prefixes. Background Art

[0002] Event extraction is abbreviated as EE. Event argument extraction, abbreviated as EAE, is a key subtask in the EE task. The goal of EAE is to extract the arguments mentioned in the event and associate them with predefined roles in the event type to form structured event knowledge. This structured event knowledge can be further used to improve downstream tasks such as dialogue systems, recommendation systems, question answering, information retrieval, knowledge graph construction, and public affairs management.

[0003] However, existing hint-based methods usually only consider the current event type to be extracted, ignoring the possibility of multiple events in the input and their complex relations. This problem has not been effectively addressed. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide an event argument extraction method based on event co-occurrence prefixes to achieve the capture and analysis of event boundaries.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for extracting event arguments based on event co-occurrence prefixes, comprising:

[0007] Construct a BART-Large model; the BART-Large model includes: 12 encoders and 12 decoders; the encoder includes: a co-occurrence-self-attention layer; the decoder includes: a co-occurrence-cross-attention layer; the replication mechanism of the BART-Large model includes:

[0008] and

[0009]

[0010] Among them, p gen (y i =t|y<i,x i, ...x m ) is the prediction of the current result when the traditional generative model generates the result; To control the probability of the generated model generating results, the hidden state of the last decoder in the BART-Large model is calculated to obtain; [x1,...,x m ] is the input token vector; The probability of directly copying the input is obtained by calculating the cross-attention weight in the last decoder in the BART-Large model; λ is a hyperparameter;

[0011] Inputting text, event mentions, and prompts into the encoder, and extracting intermediate keys and intermediate vectors of the event mentions and the prompts to obtain encoded intermediate keys and encoded intermediate vectors;

[0012] The templates corresponding to all event types in the text are spliced, and the splicing result is encoded using the template encoding unit to obtain a T-occur vector representation; the adjacent templates are separated by special characters <point>To split;

[0013] Introduce the hyperparameter adjustable Info-Q, input the T-occur vector representation and the adjustable Info-Q into the feedforward network for calculation, and obtain INF ooccur , and the INF ooccur concatenating the encoded intermediate key and the encoded intermediate vector in a prefix manner;

[0014] Encoding the text using the template encoding unit to obtain a second vector representation;

[0015] Introducing static Info-Q, inputting the second vector representation and the static Info-Q into a feedforward network for calculation, and outputting the network;

[0016] Calculate the product of the network output, the concatenated encoding intermediate key and the encoding intermediate vector, and average the product results to obtain the vector Info ext ;

[0017] Extracting the product result of the decoding intermediate key and the decoding intermediate vector in the decoder to obtain an attention weight;

[0018] The vector Info ext Add the attention weight as a bias to obtain attention information;

[0019] The attention information is input into the event argument extraction module of the BART-Large model to obtain the target argument extraction result.

[0020] Preferably, the prompt includes: a basic template, an event trigger and an event type description.

[0021] Preferably, the static Info-Q is 64.

[0022] The present invention discloses the following technical effects:

[0023] The present invention provides an event argument extraction method based on event co-occurrence prefixes. By extracting the semantic information of all events in the input and extracting and adding prefixes, the defect of the conventional model in processing complex event relations is poor, and the extraction and analysis of time relations is realized; by setting up for introducing two Info-Qs, the problem of chaotic output format of the conventional model is solved, and the constraint on the output format is realized; by introducing an adjustable replication mechanism, the potential noise problem caused by introducing additional information in each layer of attention calculation is solved, and stable interaction between attention layers is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A schematic diagram of an event argument extraction process provided by an embodiment of the present invention;

[0026] Figure 2 A structural diagram of the BART-Large model provided in an embodiment of the present invention;

[0027] Figure 3 This is a structural diagram of the Information-Prefix module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The purpose of the present invention is to provide an event argument extraction method based on event co-occurrence prefixes to achieve the capture and analysis of event boundaries.

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Figure 1 A schematic diagram of the event argument extraction process provided by an embodiment of the present invention, Figure 2 The BART-Large model structure diagram provided by the embodiment of the present invention is as follows: Figure 3 The structure diagram of the Information-Prefix module provided in the embodiment of the present invention is as follows: Figure 1 , Figure 2 as well as Figure 3 As shown, the present invention provides an event argument extraction method based on event co-occurrence prefixes, comprising:

[0032] Step 100: Construct a BART-Large model; the BART-Large model includes: 12 encoders and 12 decoders; the encoder includes: a co-occurrence-self-attention layer; the decoder includes: a co-occurrence-cross-attention layer; the replication mechanism of the BART-Large model includes:

[0033] and

[0034]

[0035] Among them, p gen (y i =t|y<i,x i, ...x m ) is the prediction of the current result when the traditional generative model generates the result; To control the probability of the generated model, the hidden state of the last decoder in the BART-Large model is calculated; [x1,...,x m ] is the input token vector; The probability of directly copying the input is obtained by calculating the cross-attention weights in the last decoder in the BART-Large model; λ is a hyperparameter;

[0036] Step 200: Input text, event mentions, and prompts into the encoder, and extract intermediate keys and intermediate vectors of event mentions and prompts to obtain encoded intermediate keys and encoded intermediate vectors;

[0037] Step 300: splice the templates corresponding to all event types in the text, encode the spliced ​​results using the template encoding unit, and obtain the T-occur vector representation; special characters are used between adjacent templates. <point>To split;

[0038] Step 400: Introduce the hyperparameter adjustable Info-Q, input the T-occur vector representation and the adjustable Info-Q into the feedforward network for calculation, and obtain INF ooccur , and INF ooccur Concatenate with the encoding intermediate key and encoding intermediate vector respectively in prefix manner;

[0039] Step 500: Encode the text using a template encoding unit to obtain a second vector representation;

[0040] Step 600: introducing static Info-Q, inputting the second vector representation and the static Info-Q into the feedforward network for calculation, and outputting the network;

[0041] Step 700: Calculate the product of the network output and the concatenated encoded intermediate key and encoded intermediate vector, and average the product results to obtain the vector Info ext ;

[0042] Step 800: extract the product result of the decoded intermediate key and the decoded intermediate vector in the decoder to obtain the attention weight;

[0043] Step 900: Vector Info ext Add it to the attention weight as a bias to get the attention information;

[0044] Step 1000: Input the attention information into the event argument extraction module of the BART-Large model to obtain the target argument extraction result.

[0045] Specifically, the prompts include: basic template, event trigger and event type description.

[0046] Optionally, the static Info-Q is 64.

[0047] Specifically, Figure 2 and Figure 3 The English parameters are explained as follows. Prefix: prefix; Tanh: an activation function; Dropout: early stopping, a training method; FFN: feedforward neural network; Concatnate: connection; Multi-headAttention: multi-head attention; INF ooccur : Event occurs; Template_Encoder: Template encoder; INFO-Q: Event query; Input-1: Input; Template_occur: Figure 3 Templates corresponding to all co-occurrence event types mentioned in the examples in ; Event Mention: event mention; Prompt: instruction; Label: label; Bias: bias; FeedForward: forward propagation; Add&Norm: weighted normalization; Aggregation: aggregation; Mean: average; co-occurrence-self-attention: co-occurrence-self-attention layer; co-occurrence-cross-attention: co-occurrence-cross attention layer.

[0048] Further, the remaining English parameters are explained as follows: T-occur: word-occurrence; Info-Q: information quality; INF ooccur : Occurrence information; Info ext : extract information; Information-Prefix: prefix information; co-occurrence-self-attention: co-occurrence-self-attention layer; co-occurrence-cross-attention: co-occurrence-cross-attention layer; Encoder-layer: encoding layer; Decoder-layer: decoding layer; T_occur: information_occurrence; prefix: prefix; self-attention: self-attention; cross_attention: cross-attention; Bias: bias; Encoder: encoder; Decoder: decoder; T-ext: word-extraction; information: information; input_1: input_1; input_2: input_2; input_3: input_3; Info_occur: information-occurrence; K: key; V: value; OutputText: output text; Multi-headAttention: multi-head attention; Information-Bias: information bias; arguments: parameters; token span: token span; token: word; Movement: movement; Transport: transportation; injury: injury; Life: life; spansomeone: span someone.

[0049] Preferably, the ECPEAE (Event Co-occurrence prefix Event Argument Extraction Model) proposed in this embodiment inherits the simple and easily extensible DEGREE (EAE) as the basic model. The backbone model of DEGREE (EAE) uses the BART-Large model with an Encoder-Decoder Transformer structure. The BART-Large model has 12 layers of Encoder-layer (encoding layer) and 12 layers of Decoder-layer (decoding layer). In its core components, this embodiment proposes co-occurrence-self-attention and co-occurrence-cross-attention to replace the self-attention in its traditional Encoder-layer and the cross-attention in the Decoder-layer, so as to introduce event co-occurrence information into EAE.

[0050] Specifically, the backbone of ECPEAE is BART-Large, which consists of 12 layers of Encoders and 12 layers of Decoders. The dense information matrix of T_occur is extracted using the Information-Prefix module, and the dense information matrix is ​​added to the self-attention in each layer of the Encoder and the cross_attention in each layer of the Decoder in the form of a prefix, so as to incorporate the information of co-occurrence events. The dense information matrix of T-ext is also extracted using the Bias module, and added to the self-attention calculation in each layer of the Encoder in the form of a Bias. The left part of the figure shows the overall structure of the information module, where ①, ②, and ③ represent input_1, input_2, and input_3 in the information module, respectively.

[0051] Preferably, the dense matrix Info-Q controls the size of the dense information matrix carrying the amount of information T-occur. At the end of the module, the dense information matrix Info_occur is obtained, and the obtained Info_occur is concatenated with K and V in BART-Large in the form of prefix, and participates in the attention operation of each layer of co-occurrence-self-attention and co-occurrence-cross-attention, so that each layer can control the attention size of different positions of these template information.

[0052] Specifically, the template corresponding to the current event type to be extracted is called: T-ext. This template will not only be spliced ​​with the event mention and input into the model to guide the model to generate, but also is the format of the sentence generated by the model. Specifically, ECPEAE will replace the arguments extracted from the event mention with the placeholders of the corresponding roles in the template, and generate the result OutputText (output text) containing the correct arguments and successfully associated with the target role. Then, regular expressions can be used to extract all arguments from the Output Text text.

[0053] Preferably, ECPEAE incorporates the information of all co-occurring event templates in event mentions: T-occur into the model by adding prefixes. The Information-Prefix module extracts information from T-occur to generate the corresponding dense information matrix INF ooccur , the dense information matrix integrates the dense information of T-occur. Specifically, first, the templates corresponding to all event types appearing in the event mention are spliced, and special characters are used between templates. <point>Then, it will be sent to the template encoding unit for encoding to obtain the vector representation of T-occur. ECPEAE does not restrict the use of encoders. Considering the number of parameters and performance, BART_Encoder is used as the template encoding unit. It should be noted that the BART_Encoder used in the template encoding unit and the BART-Large of the backbone model are not the same BART. The two have different parameters.

[0054] Furthermore, after obtaining the vector representation of T-occur, we introduce the vector Info-Q, which plays the role of Query in the Multi-headAttention in Information-Prefix. The length of Info-Q determines the final length of the prefix, which is a hyperparameter and can be adjusted manually. Figure 2 After Multi-headAttention and a series of feedforward networks, we get the final INF ooccur , the INF ooccur It will be concatenated with the intermediate keys and intermediate vectors in the self-attention of the Encoder part and the cross_attention of the Decoder part in BART-Large in the form of prefixes to introduce the information of T-occur into the model.

[0055] Furthermore, ECPEAE integrates the information of T-ext into the model by introducing Bias. In order to introduce the information of T-ext, this embodiment designs an Information-Bias module, which requires three inputs, namely Input_1, Input_2, and Input_3, where Input_1 corresponds to T-ext, Input_2 and Input_3 correspond to the intermediate keys and values ​​participating in Multi-headAttention in the Encoder-layer of BART-Large, and the intermediate keys and values ​​have been concatenated with prefix.

[0056] Specifically, in the first step, T-ext is encoded by the template encoding unit to obtain its vector representation. The template encoding unit here uses the same BART-Encoder as the template encoding unit in Information-Prefix and shares the same parameters. In the second step, after obtaining the vector representation of T-ext, Info-Q is introduced. Unlike the Info-Q in Information-Prefix, the Info-Q of Information-Bias is not a hyperparameter, but a fixed value. ECPEAE sets it to 64. This is to ensure that the shape of the value obtained after a series of feedforward networks can be multiplied with Input_2 and Input_3. In the third step, the result of the multiplication is averaged to obtain a vector Info of length 64. ext ,Info ext The attention weights are added as the result of multiplying the Bias with the Query and the intermediate key in the Encoder-layer.

[0057] Preferably, because of the particularity of the Encoder-DecoderTransformer structure of BART-Large, each layer of the complete structure contains two self-attentions and one cross-attention. Experiments have shown that adding prefixes to the self-attention in the Encoder-layer and the cross-attention in the Decoder-layer can achieve the best results. ECPEAE introduces INF in the cross_attention of each layer of the Decoder in BART-Large ooccur , INF generated by T-occur ooccur It is concatenated with the middle key and value of Multi-headAttention in cross_attention in the form of a prefix to introduce the information of the corresponding templates of all event types in the event mention. ooccur The cross_attention of the Decoder-layer is called co-occurrence-cross-attention.

[0058] Furthermore, similar to co-occurrence-cross-attention, ECPEAE also introduces the dense information matrix INF of T-occur in each layer of the encoder. ooccur However, the BART-Large model has 12 layers of encoders, and ECPEAE introduces T-occur information in each encoder layer, but only introduces T-ext when event mentions and prompts are input into the model. In order to increase the amount of T-ext information and strengthen the constraints of the model output results, not only does it add T-occur information to each encoder layer, but it also hopes to introduce additional T-ext information.

[0059] Preferably, ECPEAE adds the Bias that integrates T-ext information to the attention weight, introduces T-ext information into each layer of the Encoder-layer, and strengthens the constraint of t_ext on the model output results. The self_attention that introduces prefix and Bias is called co-occurrence-self-attention.

[0060] Furthermore, the purpose of training the model is to enable the model to correctly extract arguments from event mentions and replace them with placeholders of corresponding roles in the template. During training, although all templates corresponding to co-occurring event types in event mentions are input into the model, the model will only extract arguments of the current event to be extracted, and the format of the final generated Output text is the same as T-ext. Because some structures in BART-Large have been modified, there is a gap with the way BART-Large is trained. In order to strengthen the restrictions on the output format, a common copy mechanism is introduced to constrain its generation results:

[0061]

[0062] Where P gen (y i =t|·) is the prediction of the current result by the traditional generative model when generating results. Controls the probability of the generative model generating results. This value is calculated by the lastdecoderhidden state in BART-Large. is the probability of copying x_j directly from the input, which is calculated by the cross-attentionweights in the last decoderlayer.

[0063] This embodiment also uses a regularization method to introduce the traditional replication mechanism Regularization on t encourages the model to make more copies of the input:

[0064]

[0065] where λ is a hyperparameter.

[0066] Specifically, in order to analyze the effectiveness of adding all event information existing in the current input to the model for processing input containing complex events, this embodiment will study the ability of the model to process complex events from two aspects, namely, multiple event input and overlapping events.

[0067] Furthermore, multiple event inputs. As shown in Table 1, the number of event mentions with multiple events in the test sets of the ACE and ERE datasets is listed. Among them, 1 represents that there is only one event in the event mention. 2 represents that there are two events in the event mention, and so on. Event mentions with only one event in the test set are summarized as ACE-1 and ERE-1, and event mentions with two or more events in the test set are summarized as ACE-2+ and ERE-2+. The trained DEGREE and ECPEAE models are tested in the above summarized datasets. The test results are shown in Table 2. It can be seen that after ECPEAE adds the semantic information of all events included in the event mention, when processing event mentions with co-occurrence of events in the ACE2005-EN dataset (including more than 2 events), the Arg-C index is improved by 1.24% compared with DEGREE. Similarly, when processing event mentions with co-occurrence of events in the ERE dataset, the Arg-C index is also improved by 3.89%. At the same time, it is found that even in event mentions without event co-occurrence (the number of events included is only 1), ECPEAE still performs better than DEGREE. This proves that ECPEAE improves the EAE task by incorporating event co-occurrence information.

[0068] Table 1

[0069] Dataset 1 2 3 4 >＝5 ACE2005-EN 185 82 15 1 1 ERE 115 43 12 0 1

[0070] Table 2

[0071]

[0072]

[0073] Table 2 shows the different performances of DEGREE(EAE) and ECPEAE in processing event mentions containing only one event and event mentions containing multiple events. In the above table, the F1 scores of the models on the ACE 2005-EN and ERE datasets Arg-I and Arg-C are given. Based on this table, it can be observed that the model always outperforms DEGREE(EAE) in both single event mention processing and multi-event event mention processing, as shown by the higher F1 scores.

[0074] Specifically, in order to study the model's ability to handle complex event relationships, the test sets of the two datasets are also classified to see whether they contain overlapping events. Overlapping events are multiple events that use the same token span as arguments. The Movement:Transport event triggered by the token and the Life:Injury event triggered by the injury share the same text span someone, so these two events are defined as overlapping events. As can be seen from Table 3, ECPEAE performs better than DEGREE (EAE) in handling overlapping events on both datasets. The Arg-C index of ECPEAE in handling overlapping events on the ACE dataset is improved by 2.03% compared with DEGREE (EAE) and by 11.42% on the ERE dataset. Not only that, ECPEAE also improves compared with DEGREE (EAE) in the no case. This shows the superiority of ECPEAE in handling overlapping events. The above table shows the ability of different models to handle overlapping events on the two datasets, where Overlap represents overlapping events and N_o represents non-overlapping events.

[0075] Table 3

[0076]

[0077] The above table in Table 3 illustrates the ability of different models to handle overlapping events in two datasets. In the table, "Overlap" indicates overlapping events, while "N_o" indicates non-overlapping events.

[0078] The beneficial effects of the present invention are as follows:

[0079] The present invention enhances the model's ability to handle complex event relationships by extracting the semantic information of all events present in the input and extracting and adding prefixes; by setting up a method for introducing two Info-Qs, the model's constraints on the output format are enhanced to prevent confusion in the model's output format; and by introducing an adjustable replication mechanism, the potential noise problem caused by introducing additional information in each layer of attention calculation is overcome.

[0080] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0081] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.< / point> < / point> < / point>

Claims

1. A method for extracting event arguments based on event co-occurrence prefixes, characterized in that: include: Construct the BART-Large model; The BART-Large model includes: 12 encoders and 12 decoders; the encoder includes: a co-occurrence-self-attention layer; the decoder includes: a co-occurrence-cross-attention layer; the replication mechanism of the BART-Large model includes: Among them, p gen (y i =t|y<i,x i, ...x m ) is the prediction of the current result when the traditional generative model generates the result; To control the probability of the generated model generating results, the hidden state of the last decoder in the BART-Large model is calculated to obtain; [x1,...,x m ] is the input token vector; The probability of directly copying the input is obtained by calculating the cross-attention weight in the last decoder in the BART-Large model; λ is a hyperparameter; Inputting text, event mentions, and prompts into the encoder, and extracting intermediate keys and intermediate vectors of the event mentions and the prompts to obtain encoded intermediate keys and encoded intermediate vectors; The templates corresponding to all event types in the text are spliced, and the splicing result is encoded using the template encoding unit to obtain a T-occur vector representation; the adjacent templates are separated by special characters <point> To split;< / point> Introduce the hyperparameter adjustable Info-Q, input the T-occur vector representation and the adjustable Info-Q into the feedforward network for calculation, and obtain INF ooccur , and the INF ooccur concatenating the encoded intermediate key and the encoded intermediate vector in a prefix manner; Encoding the text using the template encoding unit to obtain a second vector representation; Introducing static Info-Q, inputting the second vector representation and the static Info-Q into a feedforward network for calculation, and outputting the network; Calculate the product of the network output, the concatenated encoding intermediate key and the encoding intermediate vector, and average the product results to obtain the vector Info ext ; Extracting the product result of the decoding intermediate key and the decoding intermediate vector in the decoder to obtain an attention weight; The vector Info ext Add the attention weight as a bias to obtain attention information; The attention information is input into the event argument extraction module of the BART-Large model to obtain the target argument extraction result.

2. The event argument extraction method based on event co-occurrence prefixes according to claim 1, characterized in that: The prompt includes: a basic template, an event trigger, and an event type description.

3. The event argument extraction method based on event co-occurrence prefixes according to claim 1, characterized in that: The static Info-Q is 64.

Images