A method for event argument extraction based on event co-occurrence prefixes

Abstract

The present invention belongs to the field of event argument extraction and provides an event argument extraction method based on event co-occurrence prefixes, including: BART-Large model construction, encoding intermediate key and encoding intermediate vector extraction, template encoding unit encoding, prefix splicing, text encoding, network output calculation, vector Info ext Acquisition, bias addition, and event argument extraction. This invention enhances the model's ability to handle complex event relationships by extracting semantic information of all events present in the input and extracting and adding prefixes. By setting up a method for introducing two Info-Qs, the model's constraints on the output format are strengthened, preventing confusion in the model's output format. By introducing an adjustable replication mechanism, the potential noise problem caused by introducing additional information in each layer of attention calculation is overcome.

Description

Technical Field

[0001] The present invention relates to the field of event argument extraction, and in particular to an event argument extraction method based on event co-occurrence prefixes. Background Art

[0002] Event extraction (EE) is abbreviated as EE. Event argument extraction (EAE) is a key subtask within EE. The goal of EAE is to extract arguments from event mentions and associate them with predefined roles within event types to form structured event knowledge. This structured event knowledge can be further leveraged to improve downstream tasks such as dialogue systems, recommender systems, question answering, information retrieval, knowledge graph construction, and public affairs management.

[0003] However, existing hint-based methods usually only consider the current event type to be extracted, ignoring the possibility of multiple events in the input and their complex relations. This problem has not been effectively addressed. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, the present invention aims to provide an event argument extraction method based on event co-occurrence prefixes to achieve the capture and analysis of event boundaries.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] An event argument extraction method based on event co-occurrence prefixes, comprising:

[0007] Construct a BART-Large model; the BART-Large model includes: 12 encoders and 12 decoders; the encoder includes: a co-occurrence-self-attention layer; the decoder includes: a co-occurrence-cross-attention layer; the replication mechanism of the BART-Large model includes:

[0008] and

[0009]

[0010] Among them, p gen (y i =t|y<i,x i, ...x m ) is the prediction of the current result when the traditional generative model generates the result; To control the probability of the generated model, the hidden state of the last decoder in the BART-Large model is calculated; [x1,...,x m ] is the input token vector; The probability of directly copying the input is obtained by calculating the cross-attention weight in the last decoder in the BART-Large model; λ is a hyperparameter;

[0011] Inputting text, event mentions, and prompts into the encoder, and extracting intermediate keys and intermediate vectors of the event mentions and the prompts to obtain encoded intermediate keys and encoded intermediate vectors;

[0012] The templates corresponding to all event types in the text are spliced ​​together, and the spliced ​​results are encoded using the template encoding unit to obtain a T-occur vector representation; adjacent templates are separated by special characters <point>Perform segmentation;

[0013] Introduce the hyperparameter adjustable Info-Q, input the T-occur vector representation and the adjustable Info-Q into the feedforward network for calculation, and obtain INF ooccur and place the INF ooccur concatenating the encoding intermediate key and the encoding intermediate vector in a prefix manner;

[0014] Encoding the text using the template encoding unit to obtain a second vector representation;

[0015] Introducing static Info-Q, inputting the second vector representation and the static Info-Q into a feedforward network for calculation, and outputting the network;

[0016] Calculate the product of the network output, the concatenated encoding intermediate key and the encoding intermediate vector, and average the product results to obtain the vector Info ext ;

[0017] Extracting the product of the decoding intermediate key and the decoding intermediate vector in the decoder to obtain an attention weight;

[0018] The vector Info ext Add the attention weight as a bias to obtain attention information;

[0019] The attention information is input into the event argument extraction module of the BART-Large model to obtain the target argument extraction result.

[0020] Preferably, the prompt includes: a basic template, an event trigger, and an event type description.

[0021] Preferably, the static Info-Q is 64.

[0022] The present invention discloses the following technical effects:

[0023] The present invention provides an event argument extraction method based on event co-occurrence prefixes. By extracting the semantic information of all events in the input and extracting and adding prefixes, the defect of conventional models in handling complex event relationships is solved, and the extraction and analysis of time relationships is realized. By setting up a method for introducing two Info-Qs, the problem of chaotic output format of conventional models is solved, and constraints on the output format are realized. By introducing an adjustable replication mechanism, the potential noise problem caused by introducing additional information in each layer of attention calculation is solved, and stable interaction between attention layers is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of the event argument extraction process provided by an embodiment of the present invention;

[0026] Figure 2 A structural diagram of the BART-Large model provided in an embodiment of the present invention;

[0027] Figure 3 This is a structural diagram of the Information-Prefix module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0029] The purpose of the present invention is to provide an event argument extraction method based on event co-occurrence prefixes to achieve the capture and analysis of event boundaries.

[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Figure 1 A schematic diagram of the event argument extraction process provided by an embodiment of the present invention, Figure 2 The BART-Large model structure diagram provided by the embodiment of the present invention is as follows: Figure 3 The structure diagram of the Information-Prefix module provided in the embodiment of the present invention is as follows: Figure 1 、 Figure 2 as well as Figure 3 As shown, the present invention provides an event argument extraction method based on event co-occurrence prefixes, comprising:

[0032] Step 100: Construct a BART-Large model. The BART-Large model includes 12 encoders and 12 decoders. The encoder includes a co-occurrence-self-attention layer. The decoder includes a co-occurrence-cross-attention layer. The replication mechanism of the BART-Large model includes:

[0033] and

[0034]

[0035] Among them, p gen (y i =t|y<i,x i, ...x m ) is the prediction of the current result when the traditional generative model generates the result; To control the probability of the generated model, the hidden state of the last decoder in the BART-Large model is calculated; [x1,...,x m ] is the input token vector; The probability of directly copying the input is obtained by calculating the cross-attention weight in the last decoder of the BART-Large model; λ is a hyperparameter;

[0036] Step 200: Input text, event mentions, and prompts into the encoder, and extract intermediate keys and intermediate vectors of the event mentions and prompts to obtain encoded intermediate keys and encoded intermediate vectors;

[0037] Step 300: Splice the templates corresponding to all event types in the text, and use the template encoding unit to encode the spliced ​​results to obtain T-occur vector representation; special characters are used to separate adjacent templates. <point>Perform segmentation;

[0038] Step 400: Introduce the hyperparameter adjustable Info-Q, input the T-occur vector representation and the adjustable Info-Q into the feedforward network for calculation, and obtain INF ooccur , and INF ooccur Concatenate with the encoding intermediate key and encoding intermediate vector respectively in a prefix manner;

[0039] Step 500: Encode the text using a template encoding unit to obtain a second vector representation;

[0040] Step 600: Introduce static Info-Q, input the second vector representation and the static Info-Q into the feedforward network for calculation, and output the network;

[0041] Step 700: Calculate the product of the network output and the concatenated encoded intermediate key and encoded intermediate vector, and perform an average operation on the product result to obtain the vector Info ext ;

[0042] Step 800: Extract the product of the decoding intermediate key and the decoding intermediate vector in the decoder to obtain the attention weight;

[0043] Step 900: Vector Info ext Add it to the attention weight as a bias to get the attention information;

[0044] Step 1000: Input the attention information into the event argument extraction module of the BART-Large model to obtain the target argument extraction result.

[0045] Specifically, the prompts include: basic templates, event triggers, and event type descriptions.

[0046] Optionally, the static Info-Q is 64.

[0047] Specifically, Figure 2 and Figure 3 The English parameters are explained as follows. Prefix: prefix; Tanh: an activation function; Dropout: early stopping, a training method; FFN: feedforward neural network; Concatnate: connection; Multi-headAttention: multi-head attention; INF ooccur : Event occurs; Template_Encoder: Template encoder; INFO-Q: Event query; Input-1: Input; Template_occur: Figure 3 Templates corresponding to all co-occurrence event types mentioned in the examples; Event Mention: event mention; Prompt: instruction; Label: label; Bias: bias; FeedForward: forward propagation; Add&Norm: weighted normalization; Aggregation: aggregation; Mean: average; co-occurrence-self-attention: co-occurrence-self-attention layer; co-occurrence-cross-attention: co-occurrence-cross attention layer.

[0048] Further, the remaining English parameters are explained as follows: T-occur: word-occurrence; Info-Q: information quality; INF ooccur : Occurrence information; Info ext : Extract information; Information-Prefix: Prefix information; co-occurrence-self-attention: Co-occurrence-self-attention layer; co-occurrence-cross-attention: Co-occurrence-cross-attention layer; Encoder-layer: Encoding layer; Decoder-layer: Decoding layer; T_occur: Information_occurrence; prefix: Prefix; self-attention: Self-attention; cross_attention: Cross-attention; Bias: Bias; Encoder: Encoder; Decoder: Decoder; T-ext: Word-extraction; information: Information; input_1: Input_1; input_2: Input_2; input_3: Input_3; Info_occur: Information-occurrence; K: Key; V: Value; OutputText: Output text; Multi-headAttention: Multi-head Attention; Information-Bias: Information bias; arguments: Parameters; token span: token span; token: word; Movement: movement; Transport: transportation; injury: injury; Life: life; spansomeone: span someone.

[0049] Preferably, the ECPEAE (Event Co-occurrence prefix Event Argument Extraction Model) proposed in this embodiment inherits the simple and easily scalable DEGREE (EAE) as the basic model. The backbone model of DEGREE (EAE) uses the BART-Large model with an Encoder-Decoder Transformer structure. The BART-Large model has 12 layers of Encoder-layer (encoding layer) and 12 layers of Decoder-layer (decoding layer). In its core components, this embodiment proposes co-occurrence-self-attention and co-occurrence-cross-attention to replace the self-attention (self-attention) in its traditional Encoder-layer and the cross-attention (cross attention) in the Decoder-layer, so as to introduce event co-occurrence information into EAE.

[0050] Specifically, the backbone of ECPEAE is BART-Large, which consists of a 12-layer encoder and a 12-layer decoder. The Information-Prefix module is used to extract the dense information matrix of T_occur, and this dense information matrix is ​​added as a prefix to the self-attention in each layer of the encoder and the cross_attention in each layer of the decoder to incorporate information about co-occurring events. The Bias module is also used to extract the dense information matrix of T-ext and add it as a bias to the self-attention calculation in each layer of the encoder. The left part of the figure shows the overall structure of the information module, where ①, ②, and ③ represent input_1, input_2, and input_3 in the information module, respectively.

[0051] Preferably, the dense matrix Info-Q controls the size of the dense information matrix carrying the amount of information T-occur. At the end of the module, the dense information matrix Info_occur is obtained. This Info_occur is concatenated with K and V in BART-Large as a prefix and then used in the attention operations of each layer of co-occurrence-self-attention and co-occurrence-cross-attention, allowing each layer to control the amount of attention paid to different locations of the template information.

[0052] Specifically, the template corresponding to the event type to be extracted is called T-ext. This template is not only concatenated with the event mention and input into the model to guide the model's generation, but also forms the format of the sentence generated by the model. Specifically, ECPEAE replaces the arguments extracted from the event mention with the placeholders of the corresponding roles in the template, generating the resulting OutputText (output text) containing the correct arguments and successfully associated with the target role. Regular expressions can then be used to extract all arguments from the OutputText.

[0053] Preferably, ECPEAE incorporates the information of all co-occurring events in the event mention template: T-occur into the model by adding prefixes. The Information-Prefix module extracts information from T-occur to generate the corresponding dense information matrix INF ooccur , the dense information matrix integrates the dense information of T-occur. Specifically, first, the templates corresponding to all event types appearing in the event mention are spliced ​​together, and special characters are used between templates. <point>The segmentation is then fed into the template encoding unit for encoding, resulting in a vector representation of T-occur. ECPEAE does not restrict the use of encoders. Considering both parameter count and performance, BART_Encoder is used as the template encoding unit. It should be noted that the BART_Encoder used in the template encoding unit is not the same BART as the BART-Large of the backbone model; they have different parameters.

[0054] Furthermore, after obtaining the vector representation of T-occur, we introduce the vector Info-Q. This vector acts as a query in the Multi-headAttention of Information-Prefix. The length of Info-Q determines the final length of the prefix. This length is a hyperparameter that can be adjusted manually. Figure 2 After the Multi-headAttention and a series of feedforward networks, the final INF ooccur , the INF ooccur It will be concatenated with the intermediate keys and intermediate vectors in the self-attention of the encoder part and the cross_attention of the decoder part in BART-Large in a prefix manner to introduce the T-occur information into the model.

[0055] Furthermore, ECPEAE incorporates T-ext information into the model by introducing Bias. In order to introduce T-ext information, this embodiment designs an Information-Bias module, which requires three inputs, namely Input_1, Input_2, and Input_3. Input_1 corresponds to T-ext, Input_2 and Input_3 correspond to the intermediate keys and values ​​participating in Multi-headAttention in the Encoder-layer in BART-Large, and the intermediate keys and values ​​have been spliced ​​with prefix.

[0056] Specifically, in the first step, T-ext is encoded through the template encoding unit to obtain its vector representation. The template encoding unit here uses the same BART-Encoder as the template encoding unit in Information-Prefix and shares the same parameters. In the second step, after obtaining the vector representation of T-ext, Info-Q is introduced. Unlike the Info-Q in Information-Prefix, the Info-Q of Information-Bias is not a hyperparameter, but a fixed value. ECPEAE sets it to 64. This is to ensure that the shape of the value obtained after a series of feedforward networks can be multiplied with Input_2 and Input_3. In the third step, the result of the multiplication is averaged to obtain a vector Info of length 64. ext , Info ext The attention weights are added as the result of multiplying the Bias with the Query and the intermediate key in the Encoder-layer.

[0057] Preferably, due to the special nature of the Encoder-DecoderTransformer structure of BART-Large, each layer of the complete structure contains two self-attentions and one cross-attention. Experiments have shown that adding prefixes to the self-attention in the Encoder-layer and the cross-attention in the Decoder-layer can achieve the best results. ECPEAE introduces INF in the cross_attention of each Decoder layer of BART-Large. ooccur , INF generated by T-occur ooccur In the form of a prefix, it is concatenated with the middle key and value of Multi-headAttention in cross_attention to introduce the information of the corresponding templates of all event types in the event mention. ooccur The cross_attention of the Decoder-layer is called co-occurrence-cross-attention.

[0058] Furthermore, similar to co-occurrence-cross-attention, ECPEAE also introduces the dense information matrix INF of T-occur in the encoder of each layer. ooccur However, the BART-Large model has 12 encoder layers, and ECPEAE incorporates T-occur information at every encoder layer, but only introduces T-ext when event mentions and prompts are fed into the model. To increase the amount of T-ext information and tighten the constraints on the model's output, we not only incorporate T-occur information at every encoder layer, but also incorporate additional T-ext information.

[0059] Preferably, ECPEAE adds the bias that integrates T-ext information to the attention weight, introducing T-ext information into each encoder layer to strengthen the constraints of t_ext on the model output. The self-attention that introduces prefix and bias is called co-occurrence-self-attention.

[0060] Furthermore, the purpose of training the model is to enable it to correctly extract arguments from event mentions and replace them with placeholders for corresponding roles in the template. During training, although the model is fed with templates corresponding to all co-occurring event types in the event mentions, it only extracts arguments for the current event to be extracted, and the resulting output text has the same format as T-ext. Because some structural modifications to BART-Large differ from the training method for BART-Large, a common replication mechanism is introduced to constrain the generated results in order to strengthen the constraints on the output format:

[0061]

[0062] Among them, P gen (y i =t|·) is the prediction of the current result by the traditional generative model when generating results. Controls the probability of the generative model generating results. This value is calculated by the lastdecoderhidden state in BART-Large. is the probability of copying x_j directly from the input, which is calculated by the cross-attentionweights in the last decoder layer.

[0063] This embodiment also uses a regularization method to introduce the traditional replication mechanism Regularization on t encourages the model to make more copies of the input:

[0064]

[0065] where λ is a hyperparameter.

[0066] Specifically, in order to analyze the effectiveness of adding all event information present in the current input to the model for processing input containing complex events, this embodiment will study the model's ability to process complex events from two aspects: multiple event inputs and overlapping events.

[0067] Furthermore, multiple event inputs. As shown in Table 1, the number of event mentions with multiple events in the test sets of the ACE and ERE datasets is listed. Among them, 1 represents that there is only one event in the event mention. 2 represents that there are two events in the event mention, and so on. Event mentions with only one event in the test set are summarized as ACE-1 and ERE-1, and event mentions with two or more events in the test set are summarized as ACE-2+ and ERE-2+. The trained DEGREE and ECPEAE models are tested on the above summarized datasets. The test results are shown in Table 2. It can be seen that after ECPEAE adds the semantic information of all events included in the event mention, when processing event mentions with co-occurrence of events in the ACE2005-EN dataset (including more than 2 events), the Arg-C index is improved by 1.24% compared to DEGREE. Similarly, when processing event mentions with co-occurrence of events in the ERE dataset, the Arg-C index is also improved by 3.89%. We also found that even in the case of event mentions without event co-occurrence (including only 1 event), ECPEAE still outperformed DEGREE, demonstrating that the integration of event co-occurrence information into ECPEAE improves the EAE task.

[0068] Table 1

[0069] Dataset 1 2 3 4 >＝5 ACE2005-EN 185 82 15 1 1 ERE 115 43 12 0 1

[0070] Table 2

[0071]

[0072]

[0073] Table 2 shows the different performance of DEGREE(EAE) and ECPEAE when processing event mentions containing only one event and event mentions containing multiple events. The above table shows the F1 scores of the models on the ACE 2005-EN and ERE datasets Arg-I and Arg-C. Based on this table, it can be observed that the model consistently outperforms DEGREE(EAE) in processing single-event event mentions and multi-event event mentions, as shown by the higher F1 scores.

[0074] Specifically, to investigate the model's ability to handle complex event relationships, the test sets of the two datasets were also classified based on whether they contained overlapping events. Overlapping events are defined as multiple events that share the same token span as an argument. The Movement:Transport event triggered by the token and the Life:Injury event triggered by the injury share the same text span "someone," so these two events are defined as overlapping events. Table 3 shows that ECPEAE outperforms DEGREE(EAE) in handling overlapping events on both datasets. ECPEAE's Arg-C metric for handling overlapping events improved by 2.03% compared to DEGREE(EAE) on the ACE dataset and by 11.42% on the ERE dataset. Furthermore, ECPEAE also achieved an improvement over DEGREE(EAE) in the "no" case, demonstrating ECPEAE's superiority in handling overlapping events. The table above shows the ability of different models to handle overlapping events on the two datasets, where "Overlap" represents overlapping events and "N_o" represents non-overlapping events.

[0075] Table 3

[0076]

[0077] Table 3 shows the ability of different models to handle overlapping events in two datasets. In the table, "Overlap" represents overlapping events, while "N_o" represents non-overlapping events.

[0078] The beneficial effects of the present invention are as follows:

[0079] The present invention enhances the model's ability to handle complex event relationships by extracting the semantic information of all events present in the input and extracting and adding prefixes; by setting up a method for introducing two Info-Qs, the model's constraints on the output format are strengthened, preventing confusion in the model's output format; and by introducing an adjustable replication mechanism, the potential noise problem caused by introducing additional information in each layer of attention calculation is overcome.

[0080] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0081] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.< / point> < / point> < / point>

Claims

1. A method for extracting event arguments based on event co-occurrence prefixes, characterized in that: include: Construct the BART-Large model; The BART-Large model includes: 12 encoders and 12 decoders; the encoder includes: a co-occurrence-self-attention layer; the decoder includes: a co-occurrence-cross-attention layer; the replication mechanism of the BART-Large model includes: Among them, p gen (y i =t|y<i,x i, ...x m ) is the prediction of the current result when the traditional generative model generates the result; To control the probability of the generated model, the hidden state of the last decoder in the BART-Large model is calculated; [x1,...,x m ] is the input token vector; The probability of directly copying the input is obtained by calculating the cross-attention weight in the last decoder in the BART-Large model; λ is a hyperparameter; Inputting text, event mentions, and prompts into the encoder, and extracting intermediate keys and intermediate vectors of the event mentions and the prompts to obtain encoded intermediate keys and encoded intermediate vectors; The templates corresponding to all event types in the text are spliced ​​together, and the spliced ​​results are encoded using the template encoding unit to obtain a T-occur vector representation; adjacent templates are separated by special characters <point> Perform segmentation;< / point> Introduce the hyperparameter adjustable Info-Q, input the T-occur vector representation and the adjustable Info-Q into the feedforward network for calculation, and obtain INF ooccur and place the INF ooccur concatenating the encoding intermediate key and the encoding intermediate vector in a prefix manner; Encoding the text using the template encoding unit to obtain a second vector representation; Introducing static Info-Q, inputting the second vector representation and the static Info-Q into a feedforward network for calculation, and outputting the network; Calculate the product of the network output, the concatenated encoding intermediate key and the encoding intermediate vector, and average the product results to obtain the vector Info ext ; Extracting the product of the decoding intermediate key and the decoding intermediate vector in the decoder to obtain an attention weight; The vector Info ext Add the attention weight as a bias to obtain attention information; The attention information is input into the event argument extraction module of the BART-Large model to obtain the target argument extraction result.

2. The event argument extraction method based on event co-occurrence prefixes according to claim 1, characterized in that: The prompt includes: a basic template, an event trigger, and an event type description.

3. The event argument extraction method based on event co-occurrence prefixes according to claim 1, characterized in that: The static Info-Q is 64.

Images