Event argument extraction method for enhancing double-prompt interaction through entity representation

Abstract

The invention discloses an event argument extraction method for enhancing double-prompt interaction through entity representation, which comprises the following steps of: (1) creating a double-prompt template, namely creating an input template and an event prompt template which are respectively used for prompting an entity and an argument role in input, and introducing a position mark into the event prompt template to distinguish that the same argument role appears for multiple times; (2) entity representation enhancement: for each entity, firstly masking the entity, obtaining a corresponding representation by pre-training a mask language model, and then aggregating different mask representations as enhanced representations of the entity according to the position of the entity appearing in the corpus; (3) double-prompt interaction: respectively sending the input and the event prompt template into an encoder and a decoder of the BART to realize interaction between the input and the event prompt template, and in addition, adaptively fusing the enhanced entity representation into the BART to enhance the entity representation; the start and end positions of its corresponding argument in the input are predicted for each argument role in the event template using argument range prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of information extraction in natural language processing, and mainly to event extraction, and specifically to an event argument extraction method for entity representation enhanced dual prompt interaction. Background Art

[0002] In recent years, with the continuous development of information technology, the amount of information on the Internet has been expanding. Automatically and accurately extracting information of interest to users from massive amounts of information has become an urgent problem to be solved. In this context, information extraction technology for big data has become a particularly important research direction. Event extraction, as an important subtask of information extraction, aims to extract events of interest to users from a large amount of unstructured text, including words that trigger events (called trigger words), arguments contained in events (who, where, when, what), and the roles played by these arguments (participants, places, time, etc.). Event extraction has a great role in promoting other research. For example, in the field of information retrieval, event extraction technology can help users quickly obtain the required event information, thereby improving the efficiency of users receiving information; in the field of automatic summarization, combined with event extraction technology, it can more comprehensively cover the event information that occurred in the article to obtain a relatively complete summary content. In addition, event extraction technology also has very important research significance in practical application scenarios. For example, extracting the basic situation of earthquake events from news reports: time, location, magnitude, casualties, etc., can help people quickly and easily understand the important information of the events they are concerned about.

[0003] Event extraction can be further divided into two subtasks, event detection and event argument extraction. The former aims to identify the keywords (usually verbs and nouns) that trigger the event from the candidate text and determine its event type; the latter is to mark the event arguments and the roles they play (including the time, place, and people of the event) from the identified events. Since event detection has made great progress in recent years, the present invention mainly focuses on event argument extraction. Depending on the input, event argument extraction can be divided into sentence-level and document-level event argument extraction. For example, in the sentence "Theessence of the American case for war against Saddam Hussein", given the trigger word "war" and the event type "Conflict.Attack" it triggers, sentence-level event argument extraction needs to extract the arguments "American" and "Saddam Hussein" contained therein, as well as the roles "Attacker" and "Target" they play. For document-level event argument extraction, the arguments are not only contained in one sentence, but may span multiple sentences. The present invention simultaneously models sentence-level and document-level event argument extraction. For document-level event argument extraction, the present invention sequentially concatenates sentences in a document paragraph into a long sentence.

[0004] At present, the methods for event argument extraction can be roughly divided into three categories: 1) Relation extraction-based: This type of method directly models the relationship between trigger words and event arguments, and then converts event argument extraction into a relation extraction task to extract trigger words and event argument pairs. This type of method generally includes two steps, first identifying candidate arguments and then classifying them. However, a significant problem is the existence of cascading errors. 2) Question-answering-based: Some studies convert event argument extraction into a question-answering task to identify the start and end boundaries of arguments. However, this method can only complete the extraction of one argument at a time, which is inefficient. In addition, this method requires the design of specific questions for events and argument roles. 3) Generation-based: There are also some works that treat event paper extraction as a text generation task. In addition, with the popularity of prompt learning and its excellent performance, some methods convert event argument extraction into a generation task based on event prompt templates. However, these methods ignore the explicit modeling of entities in the input. According to the definition of event arguments in the Automatic Context Extraction (ACE) evaluation: the arguments of an event are the entities contained in the event. Only designing prompt templates for events without explicitly modeling the entities in the input cannot well implement the interaction between argument roles and entities. Therefore, explicitly modeling the entities in the input is also very important. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an event argument extraction method for enhancing dual prompt interaction through entity representation.

[0006] This method explicitly models entities in the input and proposes to design appropriate input and event prompt templates to prompt entities and argument roles respectively, so that the pre-trained model can better realize the interaction between the two. In addition, in addition to the input prompt template that explicitly prompts the entity, this method proposes an entity representation enhancement method with an adaptive selection mechanism to enhance the representation of entities in the input. Finally, based on the event prompt template, the argument range prediction is used to detect the start and end positions of the arguments corresponding to the argument role in the input. The invention also designs position markers to solve the problem of multiple appearances of the same argument role, which has a certain improvement in the F1 value evaluation indicator compared to the current optimal model.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] An event argument extraction method for entity representation enhanced dual prompt interaction includes the following steps:

[0009] (1) Dual prompt template creation, including input and event prompt template creation, which are used to prompt entity and argument roles respectively;

[0010] (2) Entity representation enhancement: For each entity in the input, it is first masked, and then the representation of the masked word is obtained through the pre-trained masked language model. Then, different masked representations are aggregated according to the locations where the entity appears multiple times in the corpus as the enhanced representation of the entity.

[0011] (3) Dual prompt interaction: the input and event prompt templates are used as the input of the pre-trained language model BART encoder and decoder respectively to realize the interaction between entity and argument roles. In addition, the enhanced entity representation is adaptively integrated into BART to enhance the entity representation. Finally, argument range prediction is used to detect the start and end positions of the corresponding argument in the input for each argument role.

[0012] Furthermore, step (1) specifically includes:

[0013] (101) Input prompt template creation

[0014] The present invention adopts input prompt template to explicitly model the entities in the input, and its implementation strategy is to insert special words before and after the entities; specifically, given input X = [x1, ..., x i ,…,x n ], where X represents a sentence or document. If it is a document, the sentences in the document are concatenated into a long sentence in order, n is the number of input words, and x i is the i-th word; assuming xi If it is an entity, the input prompt template is as follows:

[0015] X′=[x1,…, <e> ,x i ,< / e> ,…,x n ] (1)

[0016] Where X′ represents the input prompt template, <e>and< / e> It is a special word used to prompt the entity. In order to model more fine-grained and distinguish different types of entities, the present invention introduces the entity label as a special word. Specifically, assuming that the original input is "The essence of the American case for war against Saddam Hussein", it contains two entities "American" and "Saddam Hussein", and their entity types are "GPE" and "PER" respectively, then the final input prompt template is:

[0017] “The essence of the <gpe> American< / gpe> case for <t> war< / t> against <per>Saddam

[0018] Hussein< / per> ”

[0019] in" <gpe> "and"< / gpe> " is a special word used to indicate that the entity type of American is "GPE", <per> "and"< / per> ” has a similar effect to “ <t> "and"< / t> ” is a special word used to indicate a trigger word;

[0020] (102) Event prompt template creation

[0021] The present invention adopts event prompt template to model argument roles. Each event has an event prompt template, which contains all the predefined argument roles of the event. At the same time, additional position markers are introduced in the event prompt template to solve the problem that the same argument role appears multiple times. Specifically, given the event type is "Conflict.Attack", the argument roles include "{Attacker,Target,Victim,Instrument,Place}", then the final event prompt template P e The form is as follows:

[0022] < arg01> Attacker attacked< arg02> Target hurting <arg03>< / arg03> Victim

[0023] using <arg04>< / arg04> Instrument at <arg05>< / arg05> Place

[0024] The underlined special words are positional markers, and it can be seen that each argument role has a pair of unique special markers before and after; even if an argument role appears multiple times, it can still be distinguished by the positional markers; the predefined event pattern only defines the argument roles contained in each event, and does not define the number of times each argument role may appear. The present invention calculates the number of occurrences of the argument role of each event by counting the maximum number of occurrences of the argument role of each event in the training corpus; in addition, the special markers in the input and event prompt templates are added to the vocabulary of the pre-trained model as updateable parameters, and are continuously updated as the model is trained.

[0025] Furthermore, step (2) specifically includes:

[0026] Given a single sentence S in the corpus, S = [x1,…,e,…,x l ], where l is the number of words in sentence S; assuming e is an entity, first replace it with [MASK] to obtain the masked sentence S′=[x1,…,[MASK],…,x l ], and then send S′ to the pre-trained masked language model, and regard the representation of the position [MASK] at the last layer of the pre-trained masked language model as the representation of entity e, as shown in formula (2):

[0027] r e =PMLM(S′) (2)

[0028] where r e Refers to the representation of entity e, PMLM represents the pre-trained mask language model. In the implementation process, it can be selected according to the situation, such as RoBERTa; for other locations where entity e appears, its corresponding representation can also be obtained. Finally, by aggregating all r e To obtain the enhanced representation of entity e, as shown in formula (3):

[0029]

[0030] Where E(e) refers to the enhanced representation of entity e, S e Represents all sentences containing entity e; in this way, the present invention creates an enhanced entity representation for each entity offline; for corpus without entity annotations, the present invention performs data preprocessing through automated tools such as spaCy to obtain entity annotation results.

[0031] Furthermore, step (3) specifically includes:

[0032] Given an input, the input prompt template X′ and event prompt template P are obtained through step (1) e , the interaction between the two is achieved through the BART pre-trained language model; specifically, the input prompt template X′ is first encoded through the BART encoder:

[0033] H enc =BART-Encoder(X′) (4)

[0034] Among them, H enc Refers to the encoder representation of the input prompt template X′, where each word in the template is encoded as a corresponding word embedding vector; the enhanced entity representation is then incorporated into BART through an adaptive selection mechanism to enhance each word in the entity:

[0035]

[0036]

[0037] Where e represents the entity, and E(e) refers to the enhanced representation of entity e, obtained from step (2). s Indicates the position interval of entity e in the input prompt template, refers to the encoder representation of the i-th word of entity e, refers to the encoder representation of the jth word of entity e, W enc represents the weight parameter, exp represents the exponential function, α i represents the score of the i-th word of entity e; Refers to the encoder representation of the i-th word of entity e after enhancement. For words that are not entities, no enhancement is performed; then the decoder representation of the input and event prompt template is obtained through the BART decoder:

[0038]

[0039]

[0040] Among them, H dec refers to the decoder representation of the input prompt template X′, H pe Refers to the event prompt template P e The decoder indicates that Refers to the encoder representation of the enhanced input prompt template X′; for the event prompt template P e The argument roles surrounded by position markers in are represented by their start and end positions to obtain position-aware argument role representations in the following form:

[0041]

[0042] in Refers to the representation of the kth argument role in the event prompt template, start k and end k represents the start and end positions of the k-th argument role in the event prompt template, including the position mark, and d represents the hidden layer output dimension of the BART decoder;

[0043] After obtaining the representation of all argument roles, we then use argument range prediction to detect the start and end positions of the arguments corresponding to each argument role in the input; given the argument role representation First, get the feature representation of its start and end positions:

[0044]

[0045]

[0046] where h start and h end refers to the feature representation of the start and end positions of the k-th argument role, W start , b start , W end , b end is the weight parameter, ReLU represents the activation function, and d represents the hidden layer output dimension of the BART decoder; before obtaining the start and end position feature representations, first The shape of changes from 3×d to 1×3d;

[0047] Argument range prediction aims to obtain dec Get the argument role representation in The range of corresponding arguments (s k ,e k ), where s k and e k represent The corresponding arguments are the start and end positions in the input; the argument roles of an event do not all appear in the event. If an argument role has no corresponding argument in the input, its range is (0,0); before predicting the range, the enhanced entity representation is also adaptively integrated into H dec :

[0048]

[0049]

[0050] Where e represents the entity in the input prompt template, and E(e) refers to the enhanced representation of entity e, obtained in step (2). s Represents the position interval of entity e in the input prompt template, refers to the decoder representation of the i-th word of entity e, refers to the decoder representation of the jth word of entity e, W dec represents the weight parameter, exp represents the exponential function, β i represents the score of the i-th word of entity e; Refers to the decoder representation of the enhanced i-th word of entity e. For words that are not entities, no enhancement is performed. Next, the start and end positions are obtained in the following way:

[0051]

[0052]

[0053] where h start and h end Refers to the feature representation corresponding to the start and end positions of the kth argument role, s start and end represents the probability distribution of the start and end positions, Refers to the decoder representation of the enhanced input prompt template X′, n represents the number of input words, and Softmax represents the normalization function;

[0054] In order to optimize the model, the following loss function is used:

[0055] L k (X)=-(s k log s start +e k log s end ) (16)

[0056]

[0057] Where D represents all inputs in the training corpus, s k and e k Respectively represent s k and e k The probability distribution of y (a one-hot vector of length n).

[0058] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0059] (1) To explicitly model entities in the input, we propose an event argument extraction method that enhances double-prompt interaction with entity representation, thus achieving better interaction between argument roles and entities.

[0060] (2) We introduce an entity representation enhancement method with adaptive selection to enhance each word in the entity;

[0061] (3) Design position markers to distinguish multiple-occurring argument roles;

[0062] (4) The present invention was verified on multiple sentence-level and document-level event argument extraction datasets, and the experimental results demonstrated the effectiveness of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is an architecture diagram of an event argument extraction method for entity representation enhanced dual prompt interaction provided by the present invention. It includes three submodules, namely: 1) dual prompt template creation, 2) entity representation enhancement, and 3) dual prompt interaction. DETAILED DESCRIPTION

[0064] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0065] The present invention provides an event argument extraction method for entity representation enhanced dual prompt interaction. The overall framework of the method is as follows: Figure 1 The whole system includes the steps of data preprocessing, dual prompt template creation, entity representation enhancement and dual prompt interaction.

[0066] The specific steps are as follows:

[0067] (1) In order to verify the effectiveness of the present invention, experiments were conducted on multiple datasets, including ACE2005 [1] 、RAMS [2] and WIKIEVENT [3] Dataset.

[0068] ACE2005 is a sentence-level event argument extraction dataset, which contains 599 annotated English documents, 33 event types, 22 argument roles, and 7 entity types.

[0069] RAMS is a document-level event argument extraction dataset, which has 3993 annotated documents, 9124 samples, and each sample contains 5 sentences. RAMS defines 139 event types and 65 argument roles. However, RAMS does not provide entity annotation, so the invention first uses spaCy for preprocessing and obtains entity annotation results. After data preprocessing, 18 entity types are defined.

[0070] WIKIEVENT is another document-level event argument extraction dataset, which annotates 246 English documents, 59 argument roles and 17 entity types.

[0071] In addition, for each data set, the present invention adopts the same data division method as the predecessors.

[0072] (2) Double prompt template creation

[0073] (201) Input prompt template creation

[0074] The present invention adopts input prompt template to explicitly model the entities in the input, and its implementation strategy is to insert special words before and after the entities; specifically, given input X = [x1, ..., x i ,…,x n ], where X represents a sentence or document. If it is a document, the sentences in the document are concatenated into a long sentence in order, n is the number of input words, and x i is the i-th word; assuming x i If it is an entity, the input prompt template is as follows:

[0075] X′=[x1,…, <e> ,x i ,< / e> ,…,x n ] (1)

[0076] Where X′ represents the input prompt template, <e> and< / e> It is a special word used to prompt the entity. In order to model more fine-grained and distinguish different types of entities, the present invention introduces the entity label as a special word. Specifically, assuming that the original input is "The essence of the American case for war against Saddam Hussein", it contains two entities "American" and "Saddam Hussein", and their entity types are "GPE" and "PER" respectively, then the final input prompt template is:

[0077] “The essence of the <gpe> American< / gpe> case for〈t〉war〈 / t〉against〈PER〉Saddam

[0078] Hussein

[0079] in" <gpe> "and"< / gpe> " is a special word used to indicate that the entity type of American is "GPE", <per> "and"< / per> ” has a similar effect to “ <t> "and"< / t> ” is a special word used to indicate a trigger word;

[0080] (202) Event prompt template creation

[0081] The present invention adopts event prompt template to model argument roles. Each event has an event prompt template, which contains all the predefined argument roles of the event. At the same time, additional position markers are introduced in the event prompt template to solve the problem that the same argument role appears multiple times. Specifically, given the event type is "Conflict.Attack", the argument roles include "{Attacker, Target, Victim, Instrument, Place}", then the final event prompt template P e The form is as follows:

[0082] <arg01>< / arg01> Attacker attacked <arg02>< / arg02> Target hurting <arg03>< / arg03> Victim

[0083] using< arg04> Instrument at <arg05>< / arg05> Place

[0084] The underlined special words are positional markers, and it can be seen that each argument role has a pair of unique special markers before and after; even if an argument role appears multiple times, it can still be distinguished by the positional markers; the predefined event pattern only defines the argument roles contained in each event, and does not define the number of times each argument role may appear. The present invention calculates the number of occurrences of the argument role of each event by counting the maximum number of occurrences of the argument role of each event in the training corpus; in addition, the special markers in the input and event prompt templates are added to the vocabulary of the pre-trained model as updateable parameters, and are continuously updated as the model is trained.

[0085] (3) Enhanced entity representation

[0086] Given a single sentence S in the corpus, S = [x1,…,e,…,x l ], where l is the number of words in sentence S; assuming e is an entity, first replace it with [MASK] to obtain the masked sentence S′=[x1,…,[MASK],…,x l ], and then send S′ to the pre-trained masked language model, and regard the representation of the position [MASK] at the last layer of the pre-trained masked language model as the representation of entity e, as shown in formula (2):

[0087] r e =PMLM(S′) (2)

[0088] where r eRefers to the representation of entity e, PMLM represents the pre-trained mask language model. In the implementation process, it can be selected according to the situation, such as RoBERTa; for other locations where entity e appears, its corresponding representation can also be obtained. Finally, by aggregating all r e To obtain the enhanced representation of entity e, as shown in formula (3):

[0089]

[0090] Where E(e) refers to the enhanced representation of entity e, S e Represents all sentences containing entity e; in this way, the present invention creates an enhanced entity representation for each entity offline; for corpus without entity annotations, the present invention performs data preprocessing through automated tools such as spaCy to obtain entity annotation results.

[0091] (4) Dual prompt interaction

[0092] Given an input, the input prompt template X′ and event prompt template P are obtained through step (1) e , the interaction between the two is achieved through the BART pre-trained language model; specifically, the input prompt template X′ is first encoded through the BART encoder:

[0093] H enc =BART-Encoder(X′) (4)

[0094] Among them, H enc Refers to the encoder representation of the input prompt template X′, where each word in the template is encoded as a corresponding word embedding vector; the enhanced entity representation is then incorporated into BART through an adaptive selection mechanism to enhance each word in the entity:

[0095]

[0096]

[0097] Where e represents the entity, and E(e) refers to the enhanced representation of entity e, obtained from step (2). s Indicates the position interval of entity e in the input prompt template, refers to the encoder representation of the i-th word of entity e, refers to the encoder representation of the jth word of entity e, W enc represents the weight parameter, exp represents the exponential function, α i represents the score of the i-th word of entity e; Refers to the encoder representation of the i-th word of entity e after enhancement. For words that are not entities, no enhancement is performed; then the decoder representation of the input and event prompt template is obtained through the BART decoder:

[0098]

[0099]

[0100] Among them, H dec refers to the decoder representation of the input prompt template X′, H pe Refers to the event prompt template P e The decoder indicates that Refers to the encoder representation of the enhanced input prompt template X′; for the event prompt template P e The argument roles surrounded by position markers in are represented by their start and end positions to obtain position-aware argument role representations in the following form:

[0101]

[0102] in Refers to the representation of the kth argument role in the event prompt template, start k and end k represents the start and end positions of the k-th argument role in the event prompt template, including the position mark, and d represents the hidden layer output dimension of the BART decoder;

[0103] After obtaining the representation of all argument roles, we then use argument range prediction to detect the start and end positions of the arguments corresponding to each argument role in the input; given the argument role representation First, get the feature representation of its start and end positions:

[0104]

[0105]

[0106] where h start and h end refers to the feature representation of the start and end positions of the k-th argument role, W start , b start , W end , b end is the weight parameter, ReLU represents the activation function, and d represents the hidden layer output dimension of the BART decoder; before obtaining the start and end position feature representations, first The shape of changes from 3×d to 1×3d;

[0107] Argument range prediction aims to obtain dec Get the argument role representation in The range of corresponding arguments (s k ,e k ), where s k and e k represent The corresponding arguments are the start and end positions in the input; the argument roles of an event do not all appear in the event. If an argument role has no corresponding argument in the input, its range is (0,0); before predicting the range, the enhanced entity representation is also adaptively integrated into H dec :

[0108]

[0109]

[0110] Where e represents the entity in the input prompt template, and E(e) refers to the enhanced representation of entity e, obtained in step (2). s Represents the position interval of entity e in the input prompt template, refers to the decoder representation of the i-th word of entity e, refers to the decoder representation of the jth word of entity e, W dec represents the weight parameter, exp represents the exponential function, β i represents the score of the i-th word of entity e; Refers to the decoder representation of the enhanced i-th word of entity e. For words that are not entities, no enhancement is performed. Next, the start and end positions are obtained in the following way:

[0111]

[0112]

[0113] where h start and h end Refers to the feature representation corresponding to the start and end positions of the kth argument role, s start and end represents the probability distribution of the start and end positions, Refers to the decoder representation of the enhanced input prompt template X′, n represents the number of input words, and Softmax represents the normalization function;

[0114] In order to optimize the model, the following loss function is used:

[0115] L k (X)=-(s k log s start +e k log s end ) (16)

[0116]

[0117] Where D represents all inputs in the training corpus, s k and e k Respectively represent s k and e k The probability distribution of y (a one-hot vector of length n).

[0118] In the specific implementation process, the present invention uses BART Base and Large models for experiments. By setting various hyperparameters in advance, for ACE2005, the batch size of the Base and Large models is set to 32 and 16 respectively. For RAMS and WIKIEVENT, the batch size of the Base and Large models is set to 8 and 4 respectively. The learning rate is set to 2e-5, the optimizer uses AdamW, and the Warmup is set to 0.1. The epoch is set to 50, and the early stop is set to 8. For ACE2005, RAMS, and WIKIEVENT, the length of the input prompt template is set to 240, 550, and 550 respectively, and the length of the event prompt template is set to 50, 50, and 80 respectively.

[0119] In order to verify the effectiveness of the method of the present invention, the method of the present invention (DPIERE) is compared with the current advanced and representative models (EEQA, RCEE_ER, FEAE, DocMRC, BART-Gen, PAIE), which can be roughly divided into two categories:

[0120] Question-Answer Based Approach:

[0121] 1) EEQA: Du et al. [4] A new event extraction paradigm is introduced, which is transformed into a question answering (QA) task to solve the sentence-level event argument extraction task in an end-to-end manner.

[0122] 2) RCEEE_ER: Liu et al. [5] The sentence-level event extraction is explicitly cast as a machine reading comprehension (MRC) problem, which includes an unsupervised question generation process followed by BERT-based question answering to retrieve the answers as event extraction results.

[0123] 3) FEAE: Wei et al. [6] The document-level event argument extraction is converted into a question-answering problem, and a model based on machine reading comprehension is used to extract the argument range. Curriculum learning knowledge distillation is introduced to drive the final model.

[0124] 4) DocMRC: Liu et al. [7] A new perspective is adopted to solve the data scarcity problem faced by document-level implicit event argument extraction by combining event argument extraction with machine reading comprehension and designing two data augmentation mechanisms for event argument extraction through machine reading comprehension.

[0125] Generation-based methods:

[0126] 5) BART-Gen: Li et al. [3] The event argument extraction task is transformed into a conditional generation task based on event prompt templates, and an end-to-end document-level event argument extraction model is created.

[0127] 6) PAIE: Ma et al. [8] By designing a suitable event prompt template, all arguments and their argument roles can be extracted at once, taking into account both efficiency and performance, and can be used for sentence-level and document-level event argument extraction.

[0128] The specific evaluation indicators used are precision (Precision, P), recall (Recall, R), and F1 value to evaluate the experimental performance.

[0129] Table 1 shows the experimental results of the model and all the comparison methods on the sentence-level event argument dataset ACE2005, and Table 2 shows the experimental results of model 1 and all the comparison methods on the document-level event argument extraction datasets RAMS and WIKIEVENT. The higher the F1 score value, the better the performance of the model. The "-b" in the PLM column represents the Base model, the "-l" represents the Large model, and the "-" represents that the relevant results are not given in the original paper. In addition, for DPIERE, the present invention takes the average of 5 experimental results as the final experimental result.

[0130] Table 1 Performance of the proposed method and other methods on the sentence-level event argument extraction dataset ACE2005

[0131]

[0132] Table 2 Performance of the proposed method and other methods on document-level event argument extraction datasets RAMS and WIKIEVENT

[0133]

[0134] From the experimental results in Table 1 and Table 2, we can see that:

[0135] 1) Through the dual-prompt interaction enhanced by entity representation, DPIERE can better capture the interaction between argument roles and entities and outperforms most baseline models. The method of the present invention outperforms most models at both sentence-level and document-level corpora by designing appropriate input and event prompt templates to prompt entities and argument roles respectively. In addition, the present invention introduces position markers in the event prompt template to distinguish argument roles that appear multiple times.

[0136] 2) By explicitly modeling entities in the input, DPIERE surpasses other sequence generation methods with hint templates. Compared with PAIE, DPIERE achieves better performance through additional input hint templates and entity representation enhancement. Specifically, for sentence-level event argument extraction, the F1 of DPIERE's Base and Large models improve by 0.7% and 1.5%, respectively. The improvements of DPIERE's Base and Large models on the RAMS dataset are 0.7% and 0.4%, respectively, and on the WIKIEVENT dataset are 0.9% and 1.1%, respectively. It is worth noting that the improvement effect of DPIERE on RAMS is not as good as that on ACE2005 and WIKIEVENT. One possible reason is that the entity annotation results of the RAMS dataset are not completely accurate.

[0137] 3) Compared with question-answering based methods, generation-based methods have better performance with the same scale of parameters by designing appropriate templates, indicating that they can better utilize the knowledge of pre-trained language models.

[0138] In addition, in order to verify the effect of each module, the present invention implements a series of comparative experiments. Table 3 shows the degradation experimental results of the model on the sentence-level event argument extraction dataset ACE2005, and Table 4 shows the degradation experimental results of the model on the document-level event argument extraction datasets RAMS and WIKIEVENT. Among them, "w / o PM" means removing the position mark, "w / oIPT" removes the input prompt template and directly uses the original input, "w / o ERE" means removing the entity representation enhancement, and "w / oALL" means removing "PM", "IPT", and "ERE".

[0139] Table 3 Degradation experimental results of the proposed method on the ACE2005 dataset

[0140] Model PLM P(％) R(％) <![CDATA[F1(%)]]> DPIERE BART-b 68.6 72.4 70.6 w / o PM BART-b 70.0 69.8 69.9 w / o IPT BART-b 66.8 70.2 68.5 w / o ERE BART-b 69.2 68.1 68.7 w / o ALL BART-b 66.5 66.5 66.6

[0141] Table 4 Degradation experimental results of the proposed method on RAMS and WIKIEVENT data

[0142]

[0143] Table 5 F1 values ​​of different entity modeling methods on various data sets

[0144]

[0145]

[0146] Table 6 F1 values ​​of the models on standard and non-standard entity annotation datasets

[0147] Model PLM ACE2005 WIKIEVENT DPIERE-G BART-b 70.6 62.6 DPIERE-N BART-b 70.2 62.3

[0148] From the experimental results in Tables 3 and 4, we can see that position marking, input prompt template, and entity representation enhancement are all indispensable, which verifies the effect of each module. In order to observe the impact of different entity modeling methods on the model results, Table 5 shows the impact of different entity modeling methods on the model results. Among them, "DPIERE-I" means that the input prompt template is uniformly used with " <e> "and"< / e> " to prompt the entity, "DPIERE-E" means that the entity representation enhancement is performed only in the BART encoder part, "DPIERE-D" means that the entity representation enhancement is performed only in the BART decoder part, and "DPIERE-S" means that the entity representation enhancement is performed in the BART encoder and decoder parts, but the adaptive selection mechanism is removed. In order to further observe the impact of entity annotation quality on the model results, the present invention conducts experiments on standard entity annotation and automatic entity annotation datasets, and the results are shown in Table 6. Among them, DPIERE-G means that the model uses standard entity annotation, and DPIERE-N means that the model uses automatic entity annotation, and the automatic entity annotation is completed by spaCy. It can be seen from the experimental results that the effectiveness of the method proposed in the present invention.

[0149] The above contents are intended to illustrate the technical solution of the present invention schematically, and the present invention is not limited to the embodiments described above. Without departing from the scope of the present invention and the scope protected by the claims, a person skilled in the art may make many specific changes under the guidance of the present invention, which are all within the protection scope of the present invention.

[0150] References:

[0151] [1]Doddington GR, Mitchell A, Przybocki MA, et al.The automatic content extraction(ace)program-tasks,data,and evaluation[C] / / Lrec.2004,2(1):837-840.

[0152] [2]Ebner S,Xia P,Culkin R,et al.Multi-Sentence Argument Linking[C] / / Proceedings of the 58th Annual Meeting of the Association for ComputationalLinguistics.2020:8057-8077.

[0153] [3]Li S,Ji H,Han J.Document-Level Event Argument Extraction byConditional Generation[C] / / Proceedings of the 2021Conference of the NorthAmerican Chapter of the Association for Computational Linguistics:HumanLanguage Technologies.2021:894-908.

[0154] [4]Du X,Cardie C.Event Extraction by Answering(Almost)NaturalQuestions[C] / / Proceedings of the 2020 Conference on Empirical Methods inNatural Language Processing(EMNLP).2020:671-683.

[0155] [5]Liu J,Chen Y,Liu K,et al.Event extraction as machine readingcomprehension[C] / / Proceedings of the 2020 Conference on Empirical Methods inNatural Language Processing(EMNLP).2020:1641-1651.

[0156] [6]Wei K,Sun X,Zhang Z,et al.Trigger is not sufficient:Exploitingframe-aware knowledge for implicit event argument extraction[C] / / Proceedingsof the 59th Annual Meeting of the Association for Computational Linguisticsand the 11th International Joint Conference on Natural Language Processing(Volume 1:Long Papers).2021:4672-4682.

[0157] [7]Liu J,Chen Y,Xu J.Machine reading comprehension as dataaugmentation:A case study on implicit event argument extraction[C] / / Proceedings of the 2021 Conference on Empirical Methods in Natural LanguageProcessing.2021:2716-2725.

[0158] [8]Ma Y,Wang Z,Cao Y,et al.Prompt for Extraction?PAIE:PromptingArgument Interaction for Event Argument Extraction[C] / / Proceedings of the60th Annual Meeting of the Association for Computational Linguistics(Volume1:Long Papers).2022:6759-6774.

Claims

1. An event argument extraction method for entity representation enhanced dual prompt interaction, characterized in that: The following steps are involved: (1) Dual prompt template creation, including input and event prompt template creation, which are used to prompt entity and argument roles respectively; (2) Entity representation enhancement: For each entity in the input, it is first masked, and then the representation of the masked word is obtained through the pre-trained masked language model. Then, different masked representations are aggregated according to the locations where the entity appears multiple times in the corpus as the enhanced representation of the entity. (3) Dual prompt interaction: the input and event prompt templates are used as the input of the pre-trained language model BART encoder and decoder respectively to realize the interaction between entity and argument roles. In addition, the enhanced entity representation is adaptively integrated into BART to enhance the entity representation. Finally, argument range prediction is used to detect the start and end positions of the corresponding argument in the input for each argument role.

2. According to claim 1, the event argument extraction method for entity representation enhanced dual prompt interaction is characterized in that: Step (1) specifically includes: (101) Input prompt template creation The present invention adopts input prompt template to explicitly model the entities in the input, and its implementation strategy is to insert special words before and after the entities; specifically, given input X = [x1, ..., x i , …, x n ], where X represents a sentence or document. If it is a document, the sentences in the document are concatenated into a long sentence in order, n is the number of input words, and x i is the i-th word; assuming x i If it is an entity, the input prompt template is as follows: X′=[x1,…, <e> ，x i ，< / e> ,…,x n ] (1) Where X′ represents the input prompt template, <e> and< / e> It is a special word used to prompt the entity. In order to model more fine-grained and distinguish different types of entities, the present invention introduces the entity label as a special word. Specifically, assuming that the original input is "The essence of the American case for war against Saddam Hussein", it contains two entities "American" and "Saddam Hussein", whose entity types are "GPE" and "PER" respectively, then the final input prompt template is: "The essence of the <gpe> American< / gpe> case for <t> war< / t> against <per> Saddam Hussein< / per> ” in" <gpe> "and"< / gpe> " is a special word used to indicate that the entity type of American is "GPE", <per> "and"< / per> " is similar to the function of <t> "and"< / t> ” is a special word used to indicate a trigger word; (102) Event prompt template creation The present invention adopts event prompt template to model argument roles. Each event has an event prompt template, which contains all the predefined argument roles of the event. At the same time, additional position markers are introduced in the event prompt template to solve the problem that the same argument role appears multiple times. Specifically, given the event type is "Conflict.Attack", the argument roles include "{Attacker, Target, Victim, Instrument, Place}", then the final event prompt template P e The form is as follows: <arg01>< / arg01> Attacker attacked <arg02>< / arg02> Target hurting <arg03>< / arg03> Victim using <arg04>< / arg04> Instrument at <arg05>< / arg05> Place The underlined special words are positional markers, and it can be seen that each argument role has a pair of unique special markers before and after; even if an argument role appears multiple times, it can still be distinguished by the positional markers; the predefined event pattern only defines the argument roles contained in each event, and does not define the number of times each argument role may appear. The present invention calculates the number of occurrences of the argument role of each event by counting the maximum number of occurrences of the argument role of each event in the training corpus; in addition, the special markers in the input and event prompt templates are added to the vocabulary of the pre-trained model as updateable parameters, and are continuously updated as the model is trained.

3. According to claim 1, the event argument extraction method for entity representation enhanced dual prompt interaction is characterized in that: Step (2) specifically includes: Given a single sentence S in the corpus S = [x1, ..., e, ..., x l ], where l is the number of words in sentence S; assuming e is an entity, first replace it with [MASK] to obtain the masked sentence S′=[x1,…,[MASK],…,x l ], and then send S′ to the pre-trained masked language model, and regard the representation of the position [MASK] at the last layer of the pre-trained masked language model as the representation of entity e, as shown in formula (2): r e =PMLM(S′) (2) where r e Refers to the representation of entity e, PMLM represents the pre-trained mask language model. In the implementation process, it can be selected according to the situation, such as RoBERTa; for other locations where entity e appears, its corresponding representation can also be obtained. Finally, by aggregating all r e To obtain the enhanced representation of entity e, as shown in formula (3): Where E(e) refers to the enhanced representation of entity e, S e Represents all sentences containing entity e; in this way, the present invention creates an enhanced entity representation for each entity offline; for corpus without entity annotations, the present invention performs data preprocessing through automated tools such as spaCy to obtain entity annotation results.

4. The event argument extraction method for entity representation enhanced dual prompt interaction according to claim 1, characterized in that: Step (3) specifically includes: Given an input, the input prompt template X′ and event prompt template P are obtained through step (1) e , the interaction between the two is achieved through the BART pre-trained language model; specifically, the input prompt template X′ is first encoded through the BART encoder: H enc =BART-Encoder(X′) (4) Among them, H enc Refers to the encoder representation of the input prompt template X′, where each word in the template is encoded as a corresponding word embedding vector; the enhanced entity representation is then incorporated into BART through an adaptive selection mechanism to enhance each word in the entity: Where e represents the entity, and E(e) refers to the enhanced representation of entity e, obtained from step (2). s Indicates the position interval of entity e in the input prompt template, refers to the encoder representation of the i-th word of entity e, refers to the encoder representation of the jth word of entity e, W enc represents the weight parameter, exp represents the exponential function, α i represents the score of the i-th word of entity e; Refers to the encoder representation of the i-th word of entity e after enhancement. For words that are not entities, no enhancement is performed; then the decoder representation of the input and event prompt template is obtained through the BART decoder: Among them, H dec refers to the decoder representation of the input prompt template X′, H pe Refers to the event prompt template P e The decoder indicates that Refers to the encoder representation of the enhanced input prompt template X′; for the event prompt template P e The argument roles surrounded by position markers in are represented by their start and end positions to obtain position-aware argument role representations in the following form: in Refers to the representation of the kth argument role in the event prompt template, start k and end k represents the start and end positions of the k-th argument role in the event prompt template, including the position mark, and d represents the hidden layer output dimension of the BART decoder; After obtaining the representation of all argument roles, we then use argument range prediction to detect the start and end positions of the arguments corresponding to each argument role in the input; given the argument role representation First, get the feature representation of its start and end positions: where h start and h end refers to the feature representation of the start and end positions of the k-th argument role, W start , b start , W end , b end is the weight parameter, ReLU represents the activation function, and d represents the hidden layer output dimension of the BART decoder; before obtaining the start and end position feature representations, first The shape of changes from 3×d to 1×3d; Argument range prediction aims to obtain dec Get the argument role representation in The range of corresponding arguments (s k , e k ), where s k and e k represent The corresponding arguments are the start and end positions in the input; the argument roles of an event do not all appear in the event. If an argument role has no corresponding argument in the input, its range is (0, 0); before predicting the range, the enhanced entity representation is also adaptively integrated into H dec : Where e represents the entity in the input prompt template, and E(e) refers to the enhanced representation of entity e, obtained in step (2). s Represents the position interval of entity e in the input prompt template, refers to the decoder representation of the i-th word of entity e, refers to the decoder representation of the jth word of entity e, W dec represents the weight parameter, exp represents the exponential function, β i represents the score of the i-th word of entity e; Refers to the decoder representation of the enhanced i-th word of entity e. For words that are not entities, no enhancement is performed. Next, the start and end positions are obtained in the following way: where h start and h end Refers to the feature representation corresponding to the start and end positions of the kth argument role, s start and end represents the probability distribution of the start and end positions, Refers to the decoder representation of the enhanced input prompt template X′, n represents the number of input words, and Softmax represents the normalization function; In order to optimize the model, the following loss function is used: L k (X)=-(s k logs start +e k logs end ) (16) Where D represents all inputs in the training corpus, s k and e k Respectively represent s k and e k The probability distribution of y (a one-hot vector of length n).

Images