An event argument extraction method for entity representation enhanced dual prompt interaction

Abstract

The application discloses an event argument extraction method for entity representation enhanced double prompt interaction, comprising the following steps: (1) double prompt template creation, including input and event prompt template creation, which are used to prompt entities and argument roles in the input, in addition, the event prompt template introduces a position marker to distinguish multiple occurrences of the same argument role; (2) entity representation enhancement, for each entity, first mask it, obtain the corresponding representation through a pre-trained mask language model, and then aggregate different mask representations according to the position of the entity in the corpus as the enhanced representation of the entity; (3) double prompt interaction, input and event prompt templates are respectively sent into the encoder and decoder of BART to realize the interaction of the two, in addition, the enhanced entity representation is adaptively integrated into BART for entity representation enhancement; the argument range prediction is used to predict the start and end positions of the corresponding argument in the input for each argument role in the event template.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information extraction in natural language processing, and mainly relates to event extraction, in particular to an entity representation enhanced event argument extraction method based on double prompt interaction. BACKGROUND

[0002] In recent years, with the continuous development of information technology, the amount of information on the Internet is increasing. Accurately extracting information that users are interested in from massive information automatically has become a problem to be solved. Under this background, information extraction technology for big data has become an important research direction. Event extraction, as an important subtask of information extraction, aims to extract events that users are interested in from a large amount of unstructured text, including trigger words (called trigger words) of the event, arguments (who, where, when, what) contained in the event, and the roles (participants, locations, times, etc.) played by these arguments. Event extraction has a great promoting effect on other research. For example, in the field of information retrieval, event extraction technology can help users quickly obtain the required event information, thereby improving the efficiency of users receiving information; in the field of automatic abstracting, combined with event extraction technology, the event information in the article can be more comprehensively covered to obtain a relatively complete abstract content. In addition, event extraction technology also has great research significance in practical application scenarios, such as extracting the basic situation of an earthquake event from a news report: time, location, magnitude, casualty situation, etc., which can help people quickly and easily understand the important information of the event of interest.

[0003] Event extraction can be further divided into two subtasks, event detection and event argument extraction. The former aims to identify the keywords (usually verbs and nouns) that trigger an event from the candidate text and determine its event type; the latter is to annotate the arguments of the event and their roles (including information such as the time, place and characters of the event occurrence) from the identified event. Since event detection has made great progress in recent years, the present invention mainly focuses on event argument extraction. According to the input, event argument extraction can be divided into sentence-level and document-level event argument extraction. For example, in the sentence "The essence of the American case for war against Saddam Hussein" in the data set, given the trigger word "war" and the event type "Conflict.Attack" triggered by it, sentence-level event argument extraction needs to extract the arguments "American" and "Saddam Hussein" contained in it, and their roles "Attacker" and "Target". For document-level event argument extraction, the arguments are not only contained in one sentence, but may span multiple sentences. The present invention models both sentence-level and document-level event argument extraction. For document-level event argument extraction, the present invention concatenates the sentences in the document paragraphs into long sentences in order.

[0004] Currently, the methods for event argument extraction can be roughly divided into three categories: 1) based on relation extraction: this kind of method directly models the relationship between trigger words and event arguments, and then converts event argument extraction into a relation extraction task to extract trigger words and event argument pairs. This kind of method generally includes two steps, first, identifying candidate arguments, and then classifying them. However, a significant problem is the existence of cascading errors. 2) based on question and answer: some studies convert event argument extraction into a question and answer task to identify the start and end boundaries of the argument. However, this method can only extract one argument at a time, which is not efficient. In addition, this method requires designing specific questions for events and argument roles. 3) based on generation: there are also some works that treat event argument extraction as a text generation task. In addition, with the popularity of prompt learning and its excellent performance, some methods convert event argument extraction into a generation task based on event prompt templates. However, these methods ignore the explicit modeling of entities in the input. According to the Automatic Context Extraction (ACE) evaluation for the definition of event arguments: an argument of an event is an entity contained in the event. Only designing prompt templates for events, without explicitly modeling entities in the input, cannot well realize the interaction between argument roles and entities. Therefore, explicitly modeling entities in the input is also very important. SUMMARY

[0005] The purpose of the present application is to overcome the deficiencies in the prior art and provide an entity representation enhanced double prompt interaction event argument extraction.

[0006] The method explicitly models the entities in the input, proposes to design appropriate input and event prompt templates to prompt entities and argument roles respectively, so that the pre-trained model can better realize the interaction between the two. In addition, in addition to the input prompt template explicitly prompting the entity, the method proposes an entity representation enhancement method with an adaptive selection mechanism to enhance the representation of the entity in the input. Finally, based on the event prompt template, the argument range prediction is used to detect the start and end positions of the argument corresponding to the argument role in the input. The invention also designs a position marker to solve the problem of multiple occurrences of the same argument role, which has a certain improvement in the F1 value evaluation index compared to the current optimal model.

[0007] The purpose of the present application is realized by the following technical solutions:

[0008] An entity representation enhanced double prompt interaction event argument extraction method, comprising the following steps:

[0009] (1) Double prompt template creation, including input and event prompt template creation, respectively used to prompt entities and argument roles;

[0010] (2) Entity representation enhancement, for each entity in the input, first mask it, then get the representation of the masked word through the pre-trained masked language model, and then aggregate different masked representations as the enhanced representation of the entity according to the position of the multiple occurrences of the entity in the corpus;

[0011] (3) Dual prompt interaction, input and event prompt template are respectively taken as the input of the encoder and decoder of the pre-trained language model BART to realize the interaction of entity and argument role; in addition, the enhanced entity representation is adaptively integrated into BART to enhance the entity representation; finally, the argument range prediction is used to detect the start and end positions of the corresponding argument in the input for each argument role.

[0012] Further, step (1) specifically comprises:

[0013] (101) Input prompt template creation

[0014] The input prompt template is used to explicitly model the entity in the input, and the implementation strategy is to insert special words before and after the entity; specifically, given the input X = [x1, …, xn], where X represents a sentence or a document, if it is a document, the sentences in the document are spliced into a long sentence in order, n is the number of input words, xi is the i-th word; assuming that xi is an entity, the input prompt template is as follows: i ,…, xn] (1) n i i

[0015] X' = [x1, …, <e> ,x i ,< / e> ,…, xn] (1) n

[0016] Where X' represents the input prompt template, <e>and< / e> is a special word used to prompt the entity; in order to model different types of entities more finely, the label of the entity is introduced as a special word; specifically, assuming that the original input is the sentence "The essence of the American case for war against Saddam Hussein" in the dataset, which contains two entities "American" and "Saddam Hussein", and the entity types are "GPE" and "PER" respectively, then the final input prompt template is:

[0017] “American <gpe>United States< / gpe> ” “Saddam Hussein <per>Saddam Hussein< / per> ” <t>war< / t> ” Wherein “ <gpe>"and"< / gpe> ​​​​" is a special word used to prompt the entity type of "US" as "GPE", <per>"and"< / per> " has a similar function, <t>"and"< / t> " is a special word used to prompt trigger words.

[0018] (102) Event prompt template creation

[0019] The present application uses event prompt templates to model argument roles, each event has an event prompt template, which contains all the predefined argument roles of the event; At the same time, additional position markers are introduced in the event prompt template to solve the problem of multiple occurrences of the same argument role; Specifically, given the event type "Conflict.Attack", the argument roles include "{Attacker, Target, Victim, Instrument, Place}", and the final event prompt template P e The form is as follows: < arg01> Attacker attacked <arg02>< / arg02> Target hurting <arg03>< / arg03> Victim < / arg03> using <arg04>< / arg04> Instrument at <arg05>< / arg05> Place

[0020] Among them, the special word marked by the underscore is the position marker, and it can be seen that each argument role has a pair of unique special markers before and after it; Even if a certain argument role appears multiple times, it can still be distinguished by the position marker; The predefined event pattern only defines the argument roles contained in each event, and does not define the number of times each argument role may appear, the present application takes the maximum value of the number of occurrences of each argument role in the training corpus as the number of occurrences of the argument role of the event; In addition, the special markers in the input and the event prompt template are added to the word table of the pre-trained model as updatable parameters, and are updated continuously with the training of the model.

[0021] Further, step (2) specifically comprises:

[0022] Given a single sentence S=[x1,…,e,…,x l ] in the corpus, where l is the number of words in the sentence S; Assuming e is an entity, first replace it with [MASK] to get the masked sentence S'=[x1,…,[MASK],…,x lS' is input into a pre-trained mask language model, and a representation of a last layer [MASK] position of the pre-trained mask language model is regarded as a representation of the entity e, as shown in formula (2):

[0023] r e =PMLM(S') (2)

[0024] wherein r e represents a representation of the entity e, PMLM represents the pre-trained mask language model, and in the implementation process, the pre-trained mask language model can be selected according to the situation, such as RoBERTa; for other positions where the entity e appears, a corresponding representation thereof can also be obtained, and finally an enhanced representation of the entity e is obtained through aggregation of all r e , as shown in formula (3):

[0025]

[0026] wherein E(e) represents the enhanced representation of the entity e, S e represents all sentences containing the entity e; in this way, the application creates an enhanced entity representation for each entity offline; for corpus without entity annotation, the application obtains an entity annotation result through an automatic tool such as spaCy.

[0027] Further, the step (3) specifically includes:

[0028] Given an input, the input prompt template X' and the event prompt template P e are obtained through the step (1), and interaction between the two is realized through a BART pre-trained language model; specifically, firstly, the input prompt template X' is encoded through a BART encoder:

[0029] H enc =BART-Encoder(X') (4)

[0030] wherein H enc represents an encoder representation of the input prompt template X', and each word in the template is encoded into a corresponding word embedding vector; then, the enhanced entity representation is integrated into the BART through an adaptive selection mechanism to enhance each word in the entity:

[0031]

[0032]

[0033] wherein e represents an entity, E(e) represents an enhanced representation of the entity e, which is obtained through the step (2), r s represents a position interval of the entity e in the input prompt template, represents an encoder representation of an i-th word of the entity e, the encoder representation of the jth word of entity e, W enc denotes the weight parameter, exp denotes the exponential function, a v denotes the score of the ith word of entity e; the enhanced encoder representation of the ith word of entity e, for words that are not entities, no enhancement is performed; then the decoder representation of the input and event prompt template is obtained through the BART decoder:

[0034]

[0035]

[0036] where H dec denotes the decoder representation of the input prompt template X', H pe denotes the decoder representation of the event prompt template P e , denotes the encoder representation of the enhanced input prompt template X'; for the argument roles in the event prompt template P e surrounded by position markers, the position-aware argument role representation is obtained through the start and end positions, in the form as follows:

[0037]

[0038] where denotes the representation of the kth argument role in the event prompt template, start k and end k denote the start and end positions of the kth argument role in the event prompt template, including position markers, d denotes the hidden layer output dimension of the BART decoder;

[0039] After obtaining the representation of all argument roles, then the start and end positions of the argument corresponding to each argument role in the input are detected through argument range prediction; given the argument role representation first obtain the feature representation of the start and end positions:

[0040]

[0041]

[0042] where h start and h end denote the feature representation of the start and end positions of the kth argument role, W start , b start , W end , b endThese are the weight parameters, ReLU represents the activation function, and d represents the hidden output dimension of the BART decoder; before obtaining the feature representations at the start and end positions, first... The shape changes from 3×d to 1×3d;

[0043] Argument range prediction aims to predict from H dec Obtaining the argument role representation The range of corresponding arguments (s) k ,e k ), where s k and e k represent The start and end positions of the corresponding arguments in the input; not all argument roles of an event appear in the event, if an argument role has no corresponding argument in the input, then its range is (0,0); before predicting the range, the enhanced entity representation is also adaptively incorporated into H. dec :

[0044]

[0045]

[0046] Where e represents the entity in the input prompt template, and E(e) refers to the enhanced representation of entity e, obtained from step (2). s This indicates the position range of entity e in the input prompt template. The decoder representation of the i-th word of entity e. The decoder representation of the j-th word of entity e, W dec β represents the weighting parameter, exp represents the exponential function, and β represents the weighting parameter. i This represents the score of the i-th word in entity e; The decoder representation of the i-th word of entity e is enhanced. Words that are not entities are not enhanced. The start and end positions are then obtained as follows:

[0047]

[0048]

[0049] in start and h end These refer to the feature representations corresponding to the start and end positions of the k-th argument, s start and s end The probability distribution representing the start and end positions. The decoder representation of the enhanced input prompt template X′, where n represents the number of input words and Softmax represents the normalization function;

[0050] To optimize the model, the following loss function is adopted:

[0051] L k (X)=-(s k logs start +e k logs end ) (16)

[0052]

[0053] Where D represents all inputs in the training corpus, s k and e k represent the probability distribution (one-hot vector of length n) of s k and e k respectively.

[0054] Compared with the prior art, the technical scheme of the present application has the beneficial effects that:

[0055] (1) For explicit modeling of entities in the input, an entity representation enhanced double prompt interaction event argument extraction method is proposed, which realizes better interaction of argument roles and entities;

[0056] (2) An entity representation enhancement method with adaptive selection is introduced to enhance each word in the entity;

[0057] (3) Position markers are designed to distinguish multiple occurrences of argument roles;

[0058] (4) Verification is carried out on multiple sentence-level and document-level event argument extraction datasets, and the experimental results prove the effectiveness of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 An entity representation enhanced double prompt interaction event argument extraction method architecture provided by the present application. It contains three sub-modules, which are: 1) double prompt template creation, 2) entity representation enhancement, and 3) double prompt interaction. DETAILED DESCRIPTION

[0060] The present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0061] The present application provides an entity representation enhanced double prompt interaction event argument extraction method, and the overall framework of the method is as shown in Figure 1 The entire system includes data preprocessing, double prompt template creation, entity representation enhancement, and double prompt interaction.

[0062] The specific steps are as follows:

[0063] (1) To verify the effectiveness of the present invention, experiments were conducted on multiple datasets, including ACE2005. [1] RAMS [2] and WIKIEVENT [3] Dataset.

[0064] ACE2005 is a sentence-level event argument extraction dataset containing 599 annotated English documents, 33 event types, 22 argument roles, and 7 entity types.

[0065] RAMS is a document-level event argument extraction dataset containing 3993 annotated documents with 9124 examples, each containing 5 sentences. RAMS defines 139 event types and 65 argument roles. However, RAMS does not provide entity annotations; therefore, this invention first uses spaCy for preprocessing and to obtain entity annotation results. After data preprocessing, 18 entity types were defined.

[0066] WIKIEVENT is another document-level event argument extraction dataset, which includes 246 English documents, 59 argument roles, and 17 entity types.

[0067] Furthermore, for each dataset, this invention employs the same data partitioning method as previous work.

[0068] (2) Creating a double prompt template

[0069] (201) Input prompt template creation

[0070] This invention employs input prompt templates to explicitly model entities in the input, and its implementation strategy involves inserting special words before and after the entities; specifically, given input X = [x1,…,x...] i ,…,x n ], where X represents a sentence or document. If it is a document, the sentences in the document are concatenated into a long sentence in order. n is the number of input words, x i It is the i-th word; assume x i If it is an entity, the input prompt template will look like this:

[0071] X′=[x1,…, <e> ,x i ,< / e> ,…,x n (1)

[0072] Where X′ represents the input prompt template. <e>and< / e>These are special words used to prompt entities; to achieve more granular modeling and distinguish different types of entities, this invention introduces entity labels as special words; specifically, assuming the original input is the sentence "The essence of the American case for war against Saddam Hussein" from the dataset, which contains two entities "American" and "Saddam Hussein," with entity types "GPE" and "PER" respectively, then the final input prompt template is:

[0073] “ <gpe>United States< / gpe> be opposed to <per>Saddam Hussein< / per> launch <t>war< / t> "in" <gpe>"and"< / gpe> " is a special word used to indicate that the entity type of "United States" is "GPE". <per>"and"< / per> The function is similar to that of "". <t>"and"< / t> " is a special word used to indicate the trigger word;

[0074] (202) Event Prompt Template Creation

[0075] This invention uses event hint templates to model argument roles. Each event has its own event hint template, containing all predefined argument roles for that event. Additionally, extra position markers are introduced into the event hint templates to address the issue of the same argument role appearing multiple times. Specifically, given an event type of "Conflict.Attack" and argument roles including "{Attacker,Target,Victim,Instrument,Place}", the final event hint template P... e The format is as follows: < arg01> Attacker attacked <arg02>< / arg02> Target hurting <arg03>< / arg03> Victim < / arg03> using <arg04>< / arg04> Instrument at <arg05>< / arg05> Place

[0076] The special words marked by underlines are position markers, and it can be seen that each argument role is followed by a pair of unique special markers; even if an argument role appears multiple times, it can still be distinguished by the position markers; the pre-defined event pattern only defines the argument roles contained in each event, and does not define the number of times each argument role may appear, and the application takes the maximum value of the number of times each argument role appears in the training corpus as the number of times of the argument role of the event; in addition, the special markers in the input and event prompt templates are added to the word table of the pre-training model as updatable parameters, and are updated continuously with the training of the model.

[0077] (3) Entity representation enhancement

[0078] Given a single sentence S = [x1, …, e, …, x l ] in the corpus, where l is the number of words of the sentence s; assuming e is an entity, first replace it with [MASK] to obtain the masked sentence S' = [x1, …, [MASK], …, x l ], then input S' into the pre-trained mask language model, and take the representation of the last layer [MASK] position of the pre-trained mask language model as the representation of the entity e, as shown in formula (2):

[0079] r e = PMLM (S') (2)

[0080] Where r e is the representation of the entity e, and PMLM is the pre-trained mask language model, which can be selected according to the situation in the implementation process, such as RoBERTa; for other positions of the entity e, the corresponding representation can also be obtained, and finally the enhanced representation of the entity e is obtained by aggregating all r e , as shown in formula (3):

[0081]

[0082] Where E(e) is the enhanced representation of the entity e, and S e represents all sentences containing the entity e; in this way, the application creates an enhanced entity representation for each entity offline; for the corpus without entity annotation, the application obtains the entity annotation result through an automated tool such as spaCy.

[0083] (4) Double prompt interaction

[0084] Given the input, the input prompt template X' and the event prompt template P e are obtained through step (1), and the interaction between the two is realized through a BART pre-training language model; specifically, first encode the input prompt template X' through the BART encoder:

[0085] H enc = BART-Encoder(X') (4)

[0086] where H enc denotes the encoder representation of the input prompt template X', each word in the template is encoded as a corresponding word embedding vector; then the enhanced entity representation is incorporated into BART through an adaptive selection mechanism to enhance each word in the entity:

[0087]

[0088]

[0089] where e represents an entity, E(e) denotes the enhanced representation of entity e, obtained by step (2), e s denotes the position interval of entity e in the input prompt template, denotes the encoder representation of the i-th word of entity e, denotes the encoder representation of the j-th word of entity e, W enc denotes the weight parameter, exp denotes the exponential function, a i denotes the score of the i-th word of entity e; denotes the enhanced encoder representation of the i-th word of entity e, for the words that are not entities, no enhancement is performed; then the decoder representation of the input and event prompt template is obtained through the BART decoder:

[0090]

[0091]

[0092] where H dec denotes the decoder representation of the input prompt template X', H pe denotes the decoder representation of the event prompt template P e , denotes the encoder representation of the enhanced input prompt template X'; for the argument roles in the event prompt template P e surrounded by position markers, the position-aware argument role representation is obtained through the start and end positions, in the form as follows:

[0093]

[0094] where denotes the representation of the k-th argument role in the event prompt template, start k and end kThis indicates the start and end positions of the k-th argument in the event cue template, including the position marker, and d represents the hidden output dimension of the BART decoder;

[0095] After obtaining the representations of all argument roles, argument range prediction is then used to detect the start and end positions of the arguments corresponding to each argument role in the input; given the argument role representations... First, obtain the feature representations of its start and end positions:

[0096]

[0097]

[0098] Where h start and h end The feature representation of the start and end positions of the k-th argument role, W start b start W end b end These are the weight parameters, ReLU represents the activation function, and d represents the hidden output dimension of the BART decoder; before obtaining the feature representations at the start and end positions, first... The shape changes from 3×d to 1×3d;

[0099] Argument range prediction aims to predict from H dec Obtaining the argument role representation The range of corresponding arguments (s) k ,e k ), where s k and e k represent The start and end positions of the corresponding arguments in the input; not all argument roles of an event appear in the event, if an argument role has no corresponding argument in the input, then its range is (0,0); before predicting the range, the enhanced entity representation is also adaptively incorporated into H. dec :

[0100]

[0101]

[0102] Where e represents the entity in the input prompt template, and E(e) refers to the enhanced representation of entity e, obtained from step (2). s This indicates the position range of entity e in the input prompt template. The decoder representation of the i-th word of entity e. The decoder representation of the j-th word of entity e, W dec β represents the weighting parameter, exp represents the exponential function, and β represents the weighting parameter.i denotes the score of the i-th word of entity e; denotes the enhanced decoder representation of the i-th word of entity e, and for words that are not entities, no enhancement is performed; the start and end positions are then obtained by the following way:

[0103]

[0104]

[0105] where h start and h end denote the feature representation corresponding to the start and end positions of the k-th argument role, s start and s end denote the probability distribution of the start and end positions, denotes the decoder representation of the enhanced input prompt template X', n represents the number of input words, and Softmax represents a normalization function;

[0106] In order to optimize the model, the following loss function is used:

[0107] L k (X) = - (s k logs start + e k logs end ) (16)

[0108]

[0109] where D denotes all inputs in the training corpus, s k and e k represent the probability distribution (one-hot vector of length n) of s k and e k , respectively.

[0110] In the specific implementation process, the BART Base and Large models are used for experiments. By setting various hyperparameters in advance, for ACE2005, the batch sizes of the Base and Large models are set to 32 and 16, respectively. For RAMS and WIKIEVENT, the batch sizes of the Base and Large models are set to 8 and 4, respectively. The learning rate is set to 2e-5, the optimizer is AdamW, and the Warmup is set to 0.1. The epoch is set to 50, and the early stop is set to 8. For ACE2005, RAMS, and WIKIEVENT, the lengths of the input prompt templates are set to 240, 550, and 550, respectively, and the lengths of the event prompt templates are set to 50, 50, and 80, respectively.

[0111] To verify the effectiveness of the method of the present application, the method of the present application (DPIERE) is compared with current advanced and representative models (EEQA, RCEE ER, FEAE, DocMRC, BART-Gen, PAIE), which can be roughly divided into two categories:

[0112] Question and answer based method:

[0113] 1) EEQA: Du et al. [4] A new event extraction paradigm is introduced, which is transformed into a question and answer task (QA) to solve the sentence-level event argument extraction task in an end-to-end manner.

[0114] 2) RCEEE ER: Liu et al. [5] The sentence-level event extraction is explicitly transformed into a machine reading comprehension (MRC) problem, which includes an unsupervised question generation process, followed by a BERT-based question and answer to retrieve the answer as the event extraction result.

[0115] 3) FEAE: Wei et al. [6] The document-level event argument extraction is transformed into a question and answer problem, and a machine reading comprehension-based model is used to extract argument ranges, and a curriculum learning knowledge distillation is introduced to drive the final model.

[0116] 4) DocMRC: Liu et al. [7] A new perspective is adopted to solve the data scarcity problem faced by document-level implicit event argument extraction, which combines event argument extraction with machine reading comprehension, and designs two data enhancement mechanisms for event argument extraction through machine reading comprehension.

[0117] Generation-based method:

[0118] 5) BART-Gen: Li et al. [3] The event argument extraction task is transformed into a conditional generation task based on event prompt templates, and an end-to-end document-level event argument extraction model is created.

[0119] 6) PAIE: Ma et al. [8] By designing appropriate event prompt templates, all arguments and their argument roles are extracted at one time, taking into account efficiency and performance, and are used for sentence-level and document-level event argument extraction.

[0120] The specific evaluation indicators are precision (P), recall (R), and F1 value to evaluate the performance of the experiment.

[0121] Table 1 shows the experimental results of the model and all comparison methods on the sentence-level event argument dataset ACE2005, and Table 2 shows the experimental results of Model 1 and all comparison methods on the document-level event argument extraction datasets RAMS and WIKIEVENT. The higher the F1 score value, the better the performance of the model. The "-b" in the PLM column represents the Base model, the "-l" represents the Large model, and the "-" represents that the relevant results are not given in the original paper. In addition, for DPIERE, the present application takes the average of 5 experimental results as the final experimental result.

[0122] Table 1 Performance of the present application method and other methods on the sentence-level event argument extraction dataset ACE2005

[0123]

[0124] Table 2 Performance of the present application method and other methods on the document-level event argument extraction datasets RAMS and WIKIEVENT

[0125]

[0126] From the experimental results in Tables 1 and 2, it can be seen that:

[0127] 1) Through the entity representation enhanced dual prompt interaction, DPIERE can better capture the interaction between argument roles and entities and outperform most benchmark models. The present application method can exceed most models on both sentence-level and document-level corpora by designing appropriate input and event prompt templates to prompt entities and argument roles, respectively. In addition, the present application introduces position markers in the event prompt template, which can distinguish argument roles that appear multiple times.

[0128] 2) Through explicit modeling of entities in the input, DPIERE outperforms other sequence generation methods with prompt templates. Compared with PAIE, DPIERE achieves better performance through additional input prompt templates and entity representation enhancement. Specifically, for sentence-level event argument extraction, the F1 of the Base and Large models of DPIERE is improved by 0.7% and 1.5%, respectively. The Base and Large models of DPIERE on the RAMS dataset are improved by 0.7% and 0.4%, respectively, and on the WIKIEVENT dataset are improved by 0.9% and 1.1%, respectively. It is worth noting that the improvement effect of DPIERE on RAMS is not as good as that on ACE2005 and WIKIEVENT, and one possible reason is that the entity annotation results of the RAMS dataset are not completely accurate.

[0129] 3) Compared with the question and answer based method, the generation based method has better performance with the same size of parameters by designing appropriate templates, indicating that they can better utilize the knowledge of pre-trained language models.

[0130] In addition, in order to verify the effect of each module, the application implements a series of comparative experiments. Table 3 shows the degradation experiment results of the model on the sentence-level event argument extraction dataset ACE2005, and table 4 shows the degradation experiment results of the model on the document-level event argument extraction dataset RAMS and WIKIEVENT. Among them, "w / o PM" means removing the position mark, "w / o IPT" removes the input prompt template, and directly uses the original input, "w / o ERE" means removing the entity representation enhancement, and "w / o ALL" means removing "PM", "IPT", "ERE".

[0131] Table 3 Degradation experiment results of the method of the application on the ACE2005 dataset

[0132] Model PLM P(％) R(％) F1 (%) DPIERE BART-b 68.6 72.4 70.6 w / o PM BART-b 70.0 69.8 69.9 w / o IPT BART-b 66.8 70.2 68.5 w / o ERE BART-b 69.2 68.1 68.7 w / o ALL BART-b 66.5 66.5 66.6

[0133] Table 4 Degradation experiment results of the method of the application on the RAMS and WIKIEVENT data

[0134]

[0135] Table 5 F1 values of different entity modeling methods on each dataset

[0136] Model PLM ACE2005 RAMS WIKIEVENT DPIERE BART-b 70.6 50.2 62.6 DPIERE-I BART-b 70.1 49.7 61.9 DPIERE-E BART-b 69.4 49.8 61.5 DPIERE-D BART-b 68.9 49.4 61.3 DPIERE-S BART-b 69.2 49.5 61.6

[0137] Table 6 F1 values of the model on the standard and non-standard entity annotation dataset

[0138] Model PLM ACE2005 WIKIEVENT DPIERE-G BART-b 70.6 62.6 DPIERE-N BART-b 70.2 62.3

[0139] From the experimental results of table 3 and table 4, it can be seen that the position mark, the input prompt template and the entity representation enhancement are indispensable, verifying the effect of each module. In order to observe the influence of different entity modeling methods on the model results, table 5 shows the influence of different entity modeling methods on the model results. Among them, "DPIERE-I" means that "DPIERE" is used in the input prompt template, and "DPIERE-II" means that "DPIERE" is used in the input prompt template and the entity representation enhancement. <e>"and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and" "and"< / e>The entity is prompted by "DPIERE-G", "DPIERE-E" means that entity representation enhancement is only performed in the BART encoder part, "DPIERE-D" means that entity representation enhancement is only performed in the BART decoder part, and "DPIERE-S" means that entity representation enhancement is performed in the BART encoder and decoder parts, but the adaptive selection mechanism is removed. In order to further observe the influence of entity annotation quality on the model results, the present application carries out experiments on the standard entity annotation and the automatic entity annotation data sets, and the results are shown in Table 6. Among them, DPIERE-G means that the model uses standard entity annotation, and DPIERE-N means that the model uses automatic entity annotation, and the automatic entity annotation is completed by spaCy. It can be known from the experimental results that the method of the present application is effective.

[0140] The above is intended to illustrate the technical solutions of the present application, and the present application is not limited to the embodiments described above. Without departing from the scope of the present application and the scope protected by the claims, those skilled in the art can make many forms of specific changes under the inspiration of the present application, and these all belong to the protection scope of the present application.

[0141] Reference:

[0142] [1] Doddington G R, Mitchell A, Przybocki M A, et al. The automatic content extraction (ACE) program-tasks, data, and evaluation [C] / / Lrec. 2004, 2(1): 837-840.

[0143] [2] Ebner S, Xia P, Culkin R, et al. Multi-Sentence Argument Linking [C] / / Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 8057-8077.

[0144] [3] Li S, Ji H, Han J. Document-Level Event Argument Extraction by Conditional Generation [C] / / Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021: 894-908.

[0145] [4] Du X, Cardie C. Event Extraction by Answering (Almost) Natural Questions [C] / / Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020: 671-683.

[0146] [5] Liu J, Chen Y, Liu K, et al. Event extraction as machine reading comprehension [C] / / Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020: 1641-1651.

[0147] [6] Wei K, Sun X, Zhang Z, et al. Trigger is not sufficient: Exploiting frame-aware knowledge for implicit event argument extraction [C] / / Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021: 4672-4682.

[0148] [7] Liu J, Chen Y, Xu J. Machine reading comprehension as data augmentation: A case study on implicit event argument extraction [C] / / Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 2716-2725.

[0149] [8] Ma Y, Wang Z, Cao Y, et al. Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction [C] / / Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022: 6759-6774.

Claims

1. A method for extracting event arguments in entity representation-enhanced dual-cue interaction, characterized in that, Includes the following steps: (1) Dual-hint template creation, including input and event hint template creation, used to hint at entity and argument roles respectively; specifically including: (101) Input prompt template creation This invention employs input prompt templates to explicitly model entities in the input, and its implementation strategy involves inserting special words before and after the entities; specifically, given input X = [x1,…,x...] i ,…,x n ], where n represents a sentence or document. If it is a document, the sentences in the document are concatenated into a long sentence in order. n is the number of input words, x i It is the i-th word; assume x i If it is an entity, the input prompt template will look like this: X'=[x1,…, <e> ,x i ,< / e> ,…,x n ] (1) Where X' represents the input prompt template. <e> and< / e> These are special words used to prompt entities; to achieve more granular modeling and distinguish different types of entities, this invention introduces entity labels as special words; specifically, assuming the original input is "The essence of the American case for war against Saddam Hussein", which contains two entities "American" and "Saddam Hussein", with entity types "GPE" and "PER" respectively, then the final input prompt template is: "The essence of the <gpe> American< / gpe> case for <t> war< / t> against <per> Saddam Hussein< / per> ” in" <gpe> "and"< / gpe> "This is a special word used to indicate that the entity type of American is "GPE". <per> "and"< / per> " has a similar function," <t> "and"< / t> " is a special word used to indicate the trigger word; (102) Event Prompt Template Creation This invention uses event hint templates to model argument roles. Each event has its own event hint template, containing all predefined argument roles for that event. Additionally, extra position markers are introduced into the event hint templates to address the issue of the same argument role appearing multiple times. Specifically, given an event type of "Conflict.Attack" and argument roles including "{Attacker,Target,Victim,Instrument,Place}", the final event hint template P... e The format is as follows: <arg01 >Attacker< / arg01 >attacked <arg02 >Target< / arg02 >hurting <arg03 >Victim < / arg03 >using <arg04 >Instrument< / <arg04 >at <arg05 >Place< / <arg05 > The underlined special words are position markers; each argument character has a unique pair of special markers before and after it. Even if an argument character appears multiple times, it can still be distinguished by the position markers. The predefined event patterns only define the argument characters included in each event, but not the possible frequency of each argument character. This invention uses the maximum frequency of argument characters for each event in the training corpus as the frequency of argument characters for that event. In addition, the special markers in the input and event prompt templates are added to the vocabulary of the pre-trained model as updatable parameters and are continuously updated as the model is trained. (2) Entity representation enhancement: For each entity in the input, it is first masked, and then the representation of the masked word is obtained through a pre-trained masked language model. Then, different mask representations are aggregated according to the position of the entity in the corpus as the enhanced representation of the entity; specifically including: Given a single sentence S = [x1,…,e,…,x] in a corpus l ], where l is the number of words in sentence S; assuming e is an entity, first replace it with [MASK] to obtain the masked sentence S' = [x1,…,[MASK],…,x l Then, S' is fed into the pre-trained masked language model, and the representation of the last layer [MASK] position of the pre-trained masked language model is regarded as the representation of entity e, as shown in formula (2): r e =PMLM(S′) (2) Where r e The representation of entity e is defined as PMLM, which stands for Pre-trained Masked Language Model. During implementation, this model can be selected based on the specific circumstances, such as RoBERTa. Similarly, for other locations where entity e appears, its corresponding representation can also be obtained. Finally, by aggregating all representations of r... e To obtain the enhanced representation of entity e, as shown in formula (3): Where E(e) refers to the enhanced representation of entity e, and S e This represents all sentences containing entity e; in this way, the present invention creates an enhanced entity representation for each entity offline; for corpora without entity annotations, the present invention obtains entity annotation results through automated tools such as spaCy for data preprocessing. (3) Dual-cue interaction: Input and event cue templates are used as inputs to the pre-trained language model BART encoder and decoder, respectively, to achieve interaction between entities and argument roles; in addition, enhanced entity representations are adaptively integrated into BART to enhance entity representations; finally, argument range prediction is used to detect the start and end positions of the corresponding arguments in the input for each argument role; specifically including: Given input, obtain the input prompt template X' and the event prompt template P through step (1). e The interaction between the two is achieved through a BART pre-trained language model; specifically, the input prompt template X' is first encoded using a BART encoder: H enc =BART-Encoder(X′) (4) Where H enc The encoder representation of the input prompt template X' is such that each word in the template is encoded as a corresponding word embedding vector; Then, an adaptive selection mechanism is used to incorporate the enhanced entity representation into BART to enhance each word in the entity: Where e represents an entity, and E(e) refers to the enhanced representation of entity e, obtained from step (2), e s This indicates the position range of entity e in the input prompt template. The encoder representation of the i-th word of entity e. The encoder representation of the j-th word of entity e, W enc α represents the weighting parameter, exp represents the exponential function, and α represents the weighting parameter. i This represents the score of the i-th word in entity e; The encoder representation of the i-th word of entity e is enhanced; words that are not entities are not enhanced. Then, the decoder representation of the input and event prompt template is obtained through the BART decoder. Where H dec The decoder representation of input prompt template X', H pe Refers to event prompt template P e The decoder indicates that The encoder representation of the enhanced input prompt template X'; for the event prompt template P e Argument roles surrounded by position markers are represented by their start and end positions to obtain position-aware argument role representations, as follows: in The representation of the k-th argument in the event prompt template, start k and end k This indicates the start and end positions of the k-th argument in the event cue template, including the position marker, and d represents the hidden output dimension of the BART decoder; After obtaining the representations of all argument roles, argument range prediction is then used to detect the start and end positions of the arguments corresponding to each argument role in the input; given the argument role representations... First, obtain the feature representations of its start and end positions: Where h start and h end The feature representation of the start and end positions of the k-th argument role, W start b start W end b end These are the weight parameters, ReLU represents the activation function, and d represents the hidden output dimension of the BART decoder; before obtaining the feature representations at the start and end positions, first... The shape changes from 3×d to 1×3d; Argument range prediction aims to predict from H dec Obtaining the argument role representation The range of corresponding arguments (s) k ,e k ), where s k and e k represent The start and end positions of the corresponding arguments in the input; not all argument roles of an event appear in the event, if an argument role has no corresponding argument in the input, then its range is (0,0); before predicting the range, the enhanced entity representation is also adaptively incorporated into H. dec : Where e represents the entity in the input prompt template, and E(e) refers to the enhanced representation of entity e, obtained from step (2). s This indicates the position range of entity e in the input prompt template. The decoder representation of the i-th word of entity e. The decoder representation of the j-th word of entity e, W dec β represents the weighting parameter, exp represents the exponential function, and β represents the weighting parameter. i This represents the score of the i-th word in entity e; The decoder representation of the i-th word of entity e is enhanced. Words that are not entities are not enhanced. The start and end positions are then obtained as follows: Where h start and h end These refer to the feature representations corresponding to the start and end positions of the k-th argument, s start and s end The probability distribution representing the start and end positions. The decoder representation of the enhanced input suggestion template X', where n represents the number of words in the input and Softmax represents the normalization function; To optimize the model, the following loss function is used: L k (X)=-(s k log s start +e k log s end ) (16) Where D represents all inputs in the training corpus, s k and e k They represent s respectively k and e k The probability distribution (a one-hot vector of length n).

Images