A method, system and medium for extracting event arguments from a document at the chapter level

Abstract

The present invention discloses a method, system and medium for extracting event arguments from a chapter of a document, belonging to the field of text information extraction. For three retrieval methods, namely, context consistency retrieval, pattern consistency retrieval and adaptive hybrid retrieval, a retrieval enhancement generation model based on a T5 model is constructed and a corresponding retrieval enhancement strategy is executed, so that examples that help demonstrate how the model should solve the task can be recalled. Among them, the present invention generates reference vectors as depth clues through an adaptive hybrid retrieval enhancement paradigm, which can improve the analogy ability of the model.

Description

Technical Field

[0001] The present invention belongs to the field of text information extraction, and in particular relates to a method, system and medium for extracting event arguments at the chapter level of a document. Background Art

[0002] Converting the massive amount of unstructured text on the Internet into structured event knowledge is a key but unsolved goal in natural language processing, which is especially difficult when dealing with paragraph-level text. Discourse-level event argument extraction is the process of extracting the informative core of events from documents, which is very beneficial for many downstream applications, such as information retrieval, question-answering systems, and event graph reasoning. Discourse-level event argument extraction aims to extract event arguments and determine the roles they play.

[0003] In recent years, retrieval-based methods have been successfully applied to many natural language processing tasks, such as dialogue response generation, machine translation, and information extraction. These methods retrieve additional knowledge from various corpora to enhance text generation, including three main components: retrieval sources, retrieval strategies, and integration methods. At the same time, using additional knowledge as an enhancement signal is a natural way to solve the problem of insufficient information in information extraction, enabling the model to:

[0004] (a) explicitly capture prior external knowledge in a non-parametric way, thus allowing great flexibility;

[0005] (b) Treat the retrieved reference instances as clues for generating text and learn by analogy. These retrieval-based methods use similarity-based retrieval, with the basic assumption that the more similar the retrieved sample is to the original input, the more similar the sample label is to the input label, and thus it will help generate text.

[0006] This assumption is intuitive: for most tasks, similar inputs produce similar outputs. For example, in language modeling, the distribution of the next word of "Dickens is the author of" and "Dickens wrote" is basically the same. However, in paragraph-level event argument extraction, the fact that the retrieved sample is similar to the original input does not guarantee that the sample label and the input label have the same distribution in the label space. In a document, only a few words are event arguments, and other interfering contexts may mislead similarity-based retrieval and cause the sample label to deviate from the input label. In addition, paragraph-level event argument extraction requires not only predicting argument entities, but also determining the correspondence between arguments and roles, which makes it challenging to find samples with the same event label in the original input. According to statistics on the RAMS dataset, only 16.51% of instances can recall samples with the same event pattern through similarity-based retrieval.

[0007] Since chapter-level event argument extraction does not conform to the assumption of similarity-based retrieval, designing a retrieval strategy for chapter-level event argument extraction has become an urgent problem to be solved. Therefore, the present invention explores different retrieval settings for the chapter-level event argument extraction task and proposes a chapter-level event argument extraction method based on hybrid retrieval enhancement. Summary of the invention

[0008] In order to overcome the deficiency that chapter-level event argument extraction does not conform to the assumption of similarity-based retrieval, the present invention proposes corresponding chapter-level event argument extraction methods for three different retrieval settings, and proposes a retrieval enhancement strategy for chapter-level event argument extraction from the perspective of input and label distribution, which can recall examples that help demonstrate how the model should solve the task. Among them, the present invention generates reference vectors as deep clues through an adaptive hybrid retrieval enhancement paradigm, which can improve the model's analogy ability.

[0009] The technical solution adopted by the present invention is as follows:

[0010] A method for extracting event arguments from a document at the chapter level comprises the following steps:

[0011] 1) Building a retrieval enhancement generation model based on the T5 model, including an encoder and a decoder; the retrieval enhancement generation model takes the context of the target document and the connection of its event pattern as the input sequence, and generates the target sequence through encoding and decoding operations;

[0012] 2) For contextual consistency retrieval, the steps performed include:

[0013] For the target document, the sample documents most relevant to the target document are retrieved from the training corpus as discrete samples;

[0014] Input the above discrete samples into the encoder to generate discrete sample embeddings;

[0015] Input the input sequence constructed based on the target document into the encoder, concatenate the output of the encoder with the above discrete sample embedding, and input it into the decoder to generate the target sequence and its probability;

[0016] 3) For pattern consistency retrieval, the steps performed include:

[0017] Using the input event label as the query, retrieve the sample labels related to the input label;

[0018] Input the above sample labels into the encoder to generate sample label embedding;

[0019] Input the input sequence constructed based on the target document into the encoder, concatenate the output of the encoder with the above sample label embedding, and input it into the decoder to generate the target sequence and its probability;

[0020] 4) For adaptive hybrid retrieval, the steps performed include:

[0021] For the target document, the k sample documents most relevant to the target document are retrieved from the training corpus as k discrete samples;

[0022] Input the k discrete samples into the encoder to generate k discrete sample embeddings;

[0023] Based on the k discrete sample embeddings, k event semantic regions are defined in the continuous space;

[0024] Gaussian sampling is performed in the k event semantic regions to generate k pseudo samples;

[0025] The input sequence constructed based on the target document is input into the encoder, and the output of the encoder is connected with the above k pseudo samples and input into the decoder to generate the target sequence and its probability.

[0026] Furthermore, the structure of the input sequence is: <s>Event Pattern [SEP] Document Context< / s> ;in <s> For the start character,< / s> is the terminator, and [SEP] is the separator.

[0027] Furthermore, in step 2), the S-BERT model is used to retrieve sample documents that are most relevant to the target document.

[0028] Furthermore, in step 3), the S-BERT model is used to retrieve the sample labels most relevant to the target document.

[0029] Furthermore, the output of the encoder in steps 2) and 3) is concatenated with the discrete sample embedding or the sample label embedding to be expressed as [h d ;h e ;h x ], the output of the encoder in step 4) is connected with k pseudo samples and expressed as [v; h e ;h x ],in,

[0030] h e ,h x =T5-Encoder([e;x]),

[0031] h d =T5-Encoder(d)

[0032] Among them, T5 represents the T5 model, Encoder represents the encoder, e represents the event template, x represents the input sequence, d represents the sample document or sample label, v represents k pseudo samples, h represents e The embedding representation vector representing the event template, hx Embedding representation vector representing the input sequence.

[0033] Furthermore, the Gaussian sampling step in step 4) includes:

[0034] Calculate the deviation vector b = h x -h e , using the scaling vector ω (i) Convert the deviation vector b to ω (i) ⊙b, where ⊙ represents an element-wise product operation;

[0035] Construct pseudo sample v (i) =h e +ω (i) ⊙b;

[0036] Assume ω (i) Obey the Gaussian distribution and use the reconstruction parameterization method to reconstruct the Gaussian distribution, make the sampling process differentiable, and generate sampling samples.

[0037] Furthermore, in step 4), it is assumed that ω (i) The Gaussian distribution is as follows:

[0038]

[0039] Among them, diag means The vector is normalized to a diagonal matrix, It is used to normalize the importance of each dimension in b. |·| means taking the absolute value of each element in the vector. (i) =||h x -h d (i) ||,R=||h x -h e ||.

[0040] Further, the step of reconstructing the Gaussian distribution using the reconstruction parameterization method in step 4) includes:

[0041] From the standard normal distribution Sample a noise variable ε in ;

[0042] Will Formally expressed as ω (i) =μ+ε·σ, where

[0043] A computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and the steps of the above method are implemented when the processor executes the program.

[0044] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] 1) By introducing the concepts of event semantic regions and pseudo-samples in continuous space, the semantic information of events can be expressed and learned more comprehensively and accurately.

[0047] 2) The present invention can not only improve the performance of event-related tasks, but also be extended to other natural language processing fields, such as information retrieval, question-answering systems, etc.

[0048] 3) The present invention also has good scalability and adaptability, and can be flexibly adjusted and optimized according to the characteristics of specific tasks and data.

[0049] 4) The present invention provides an innovative solution for processing event semantic information in text data, which has broad application prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0051] Figure 1 It is a schematic diagram of the overall process of a method for extracting event arguments at the chapter level from a document of the present invention.

[0052] Figure 2 It is a schematic diagram of the Gaussian sampling stage of the model.

[0053] Figure 3 It is a schematic diagram of the model Gaussian sampling algorithm. DETAILED DESCRIPTION

[0054] In order to make the above features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

[0055] The present invention studies three different search settings:

[0056] 1) Setting 1: Retrieve examples in the input space to maintain contextual semantic consistency as discrete examples. This setting is to explore the necessity of pursuing the similarity of retrieved examples with the original input during the retrieval process under the condition that similar documents cannot guarantee the same distribution of event labels.

[0057] 2) Setting 2: Recall the sample labels in the label space to make them close to the input labels as discrete samples to alleviate the difficulty of learning the complex event patterns of the input labels. This setting is to verify the necessity of directly retrieving sample labels similar to the input labels as reference samples under the condition that the essence of the above assumption is to pursue sample labels similar to the input labels.

[0058] 3) Setting 3: Generate pseudo samples by sampling from the intersection of the original input and the neighboring regions of the input label, thereby maintaining the consistency of context and event patterns on the input. The pseudo samples retrieved by this setting have equal distribution with the input document in the input and label space, and thus can be used as deep cues to guide the model.

[0059] The present invention uses the T5 language model as a basic framework to generate target sequences and then decode them into event arguments. The goal of the present invention is to extract document-level event arguments for retrieval enhancement, first retrieving the top k potentially useful examples (discrete or continuous), and then integrating them into the decoder to generate role records.

[0060] 1.1 Retrieval-augmented generative model framework

[0061] The T5 model is used as the basic structure, which is an encoder-decoder pre-trained model. The encoder-decoder language model simulates the generation process, given the previous tag y (<i) (the first i tags of the character record) and the encoder input [e; x] (r represents the event template and x represents the input sequence), select a new tag y (i) Therefore, given the input [e; x], the total probability of generating output y p(y|x,e) is calculated as follows:

[0062] The input sequence is the concatenation of the document context and its event pattern, constructed as " <s> Event Pattern [SEP] Document Context< / s> The output y is a role record, which is represented by the connection between each argument and its event role, that is, the target sequence.

[0063] In this paper, the modeling of p(y|x,e) is decomposed into two steps: retrieval and prediction. Given a query document, we first retrieve the first k potentially useful examples d from the training corpus and model them as sampling from the distribution p(d|x). Then, we use the twin network structure to obtain meaningful embedding representations for the input sequence [e;x] and the example d:

[0064] h e ,h x =T5-Encoder([e;x]),

[0065] h d =T5-Encoder(d)

[0066] In the formula, h e ,h x The embedding representation vectors representing the event template and the input sequence respectively.

[0067] Then, the conditional generation of output y is performed based on the retrieved d and the original input [e; x], which is modeled as p(y|d,x,e). Specifically, by embedding k examples into h d ={h d (1) ,h d (2) ,...,h d (k)}concatenates the encoder outputs in all decoder layers and feeds them all into the decoder:

[0068] y=T5-Decoder( <bos>; [h d ;h e ;h x ])

[0069] here <bos>is the start marker of the decoder, [h d ;h e ;h x ] represents the encoder output constructed for the decoder input. In Setting 3, [v; h e ;h x ], v represents the k pseudo samples retrieved.

[0070] To get the overall probability of generating y, treat d as a latent variable and get:

[0071] p(y|x,e)=p(y|d,x,e)*p(d|x)

[0072] The main challenge of example retrieval is to design a suitable retrieval strategy to recall examples that can help demonstrate how the model solves the task. This paper explores various retrieval settings and divides the retrieval settings into three categories: (1) context consistency retrieval; (2) pattern consistency retrieval; (3) adaptive hybrid retrieval. The goal of all retrieval settings is to find k examples (discrete or continuous). If context consistency is considered, choose setting 1; if event pattern consistency is considered, choose setting 2; if both context consistency and event pattern consistency are considered, choose setting 3. The entire retrieval framework and process are as follows: Figure 1 shown.

[0073] 1.2 Contextual Consistency Retrieval

[0074] Since similar documents are not guaranteed to have the same event label distribution, Setting 1 aims to answer the question of pursuing sample documents x during retrieval. r Is the similarity with document x meaningful? In this setting, based on the query document x, sample documents x related to the original input document are retrieved from the training corpus. r , as discrete examples d. For retrieval, S-BERT is used to retrieve example documents in the training corpus that are semantically similar to the original input documents. In this way, relevant documents that are similar in content to the query document can be obtained as references to demonstrate how the model should solve the task.

[0075] 1.3 Schema consistency retrieval

[0076] The goal of Setting 2 is to explore whether conditional constraints on the label space can help improve performance and alleviate the difficulty of learning complex event patterns y. In this setting, the input event label y is used as a query, and S-BERT retrieves sample labels y related to the input label from the training corpus. r During inference, the query is the event pattern e of the test sample. In this way, related instances with similar label patterns can be found, thus providing the model with more information about the event pattern.

[0077] 1.4 Adaptive Hybrid Retrieval

[0078] To find ideal examples with similar distributions on the input and label spaces, a novel adaptive hybrid retrieval strategy is proposed to improve the model's analogy ability by sampling pseudo examples from a continuous space.

[0079] First, the top k most relevant documents are retrieved from the training corpus, which are considered to be very helpful in solving the problem involved. These k documents are used as helpful discrete examples, which can represent the semantic information of the event in a discrete space and maintain the consistency of the context and event pattern. However, the number of discrete examples is limited and cannot fully cover all semantic changes.

[0080] Then, based on these k discrete samples, k event semantic regions of each training instance are adaptively determined. These regions are represented in continuous space and can more accurately represent the semantic features of the event, including the following steps:

[0081] 1) Define an event semantic region and regard the points in the region as key states of event semantic equivalence. Specifically, in order to consider the consistency of context and event pattern, the adjacent regions of document and event pattern are first determined by setting their adjacent radius. In addition, the intersection of their adjacent regions is defined as an event semantic region, which describes an accurate replacement that is consistent with the original context and event semantic meaning. Here, there are k discrete sample embeddings h d , corresponding to k adjacent radii r, which determines k event semantic regions. For each event semantic region, the following Gaussian sampling process is performed, such as Figure 2 shown.

[0082] 2) Perform Gaussian sampling. In order to obtain diverse and accurate pseudo samples from event semantic regions, the Gaussian sampling strategy is applied to k event semantic regions, and a clustered vector is sampled from each region. These pseudo samples can serve as a supplement to training instances, helping the model to better understand and learn the semantic features of events.

[0083] like Figure 3 As shown, first use the scaling vector ω (i) The deviation vector b = h x -h e Convert to ω (i) ⊙b, where ⊙ represents the element-wise product operation. Then, a new sample v is constructed (i) =h e +ω (i) ⊙b is used as a pseudo-sample. Therefore, the goal of the sampling strategy is to find a set of scaling vectors ω = {ω (1) ,ω (2) ,...,ω (k) }. Intuitively, we can assume that ω (i) It obeys Gaussian distribution, which can be expressed formally as:

[0084]

[0085] Where diag means The vector is normalized to a diagonal matrix, It is used to normalize the importance of each dimension in b. |·| means taking the absolute value of each element in the vector, indicating that the larger the value, the more information it contains. The sampling scope is limited to the event semantic area.

[0086] Since sampling is a non-differentiable operation that will truncate the gradient, the present invention uses a reparameterization method to construct First, from the standard normal distribution Then, we will Formally:

[0087] ω (i) =μ+ε·σ

[0088] in,

[0089] Now the gradient is differentiable.

[0090] Finally, k pseudo samples v are sampled from the k event semantic regions to enhance text generation, i.e., v = {v (1) ,v (2) ,...,v (k) }. k is a hyperparameter for the number of sampled vectors, which is determined by the number of discrete samples. For a clearer presentation, Algorithm 1 summarizes the sampling process. In this way, deep cues in the continuous space can be used to provide reference information that is consistent with the input document in the input and label space to guide the model to better perform analogical reasoning and solve tasks. This adaptive hybrid retrieval strategy can enhance the model's reasoning and generalization capabilities, further improving its performance.

[0091] Note that the above three settings are intended to design retrieval strategies to extract sample examples that are relevant to the input documents. These sample examples can help demonstrate how the model solves a specific task and provide references about event patterns and label distributions. By choosing appropriate retrieval settings, different retrieval conditions and strategies can be explored to better guide and train the model and improve its performance and accuracy.

[0092] 2.1 Experimental Verification

[0093] This experiment intends to use the public datasets RAMS and WikiEvents as data. RAMS provides 9,124 annotated news examples, including 139 event types and 65 roles. WikiEvents provides 246 annotated documents, including 50 event types and 59 roles.

[0094] 2.2 Evaluation Criteria

[0095] It can be seen from the experimental data that the data set used in the present invention is relatively balanced, so classification accuracy and F1 value are used as evaluation criteria.

[0096]

[0097]

[0098]

[0099]

[0100] Among them, TP, TN, FP, and FN are the numbers of true positive examples, true negative examples, false positive examples, and false negative examples, respectively.

[0101] 2.3 Summary and Analysis

[0102] Through a large number of experiments on RAMS and WikiEvents datasets, the method used in the present invention has achieved better results in the comparison of the generative model and the question-answering model, proving the effectiveness of the retrieval model newly introduced by the present invention. In the comparison of the generative model, when the adaptive hybrid retrieval mechanism is used, the effect is generally better than the index value of the generative method using the manually constructed template. In the comparison of the question-answering model, in each retrieval strategy involved in the experiment, the argument extraction effect using the adaptive hybrid retrieval mechanism is significantly better than the question-answering model.

[0103] Through the above experiments, the effectiveness of the present invention in the task of extracting event arguments at the paragraph level has been demonstrated, and good results have been achieved on both datasets.

[0104] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.< / bos> < / bos>

Claims

1. A method for extracting event arguments from a document at the chapter level, characterized in that: The following steps are involved: 1) Building a retrieval enhancement generation model based on the T5 model, including an encoder and a decoder; The retrieval enhancement generation model takes the context of the target document and the connection of its event pattern as the input sequence, and generates the target sequence through encoding and decoding operations; 2) For contextual consistency retrieval, the steps performed include: For the target document, the sample documents most relevant to the target document are retrieved from the training corpus as discrete samples; Input the above discrete samples into the encoder to generate discrete sample embedding; Input the input sequence constructed based on the target document into the encoder, concatenate the output of the encoder with the above discrete sample embedding, and input it into the decoder to generate the target sequence and its probability; 3) For pattern consistency retrieval, the steps performed include: Using the input event label as the query, retrieve the sample labels related to the input label; Input the above sample labels into the encoder to generate sample label embedding; Input the input sequence constructed based on the target document into the encoder, concatenate the output of the encoder with the above sample label embedding, and input it into the decoder to generate the target sequence and its probability; 4) For adaptive hybrid retrieval, the steps performed include: For the target document, the k sample documents most relevant to the target document are retrieved from the training corpus as k discrete samples; Input the k discrete samples into the encoder to generate k discrete sample embeddings; Based on the k discrete sample embeddings, k event semantic regions are defined in the continuous space; Gaussian sampling is performed in the k event semantic regions to generate k pseudo samples; Input the input sequence constructed based on the target document into the encoder, concatenate the output of the encoder with the above k pseudo samples, and input it into the decoder to generate the target sequence and its probability; The output of the encoder in steps 2) and 3) is connected with the discrete sample embedding or sample label embedding to be expressed as [h d ;h e ;h x ], the output of the encoder in step 4) is connected with k pseudo samples and expressed as [v; h e ;h x ],in, h e ,h x =T5-Encoder([e;x]), h d =T5-Encoder(d) Among them, T5 represents the T5 model, Encoder represents the encoder, e represents the event template, x represents the input sequence, d represents the sample document or sample label, v represents k pseudo samples, h represents e The embedding representation vector representing the event template, h x Embedding representation vector representing the input sequence; The steps of Gaussian sampling in step 4) include: Calculate the deviation vector b = h x -h e , using the scaling vector ω (i) Convert the deviation vector b to ω (i) ⊙b, where ⊙ represents an element-wise product operation; Construct pseudo sample v (i) =h e +ω (i) ⊙b; Assume ω (i) Obeying Gaussian distribution, use reconstruction parameterization method to reconstruct Gaussian distribution, make the sampling process differentiable, and generate sampling samples; In step 4), we assume that ω (i) The Gaussian distribution is as follows: Among them, diag means The vector is normalized to a diagonal matrix, It is used to normalize the importance of each dimension in b. |·| means taking the absolute value of each element in the vector. (i) =||h x -h d (i) ||, R = ||h x -h e ||; Step 4) of reconstructing the Gaussian distribution using the reconstruction parameterization method includes: From the standard normal distribution A noise variable ε is sampled in; Will Formally expressed as ω (i) =μ+ε·σ, where 2. The method according to claim 1, characterized in that The structure of the input sequence is: <s> Event Pattern [SEP] Document Context< / s> ;in <s> For the start character,< / s> is the terminator, and [SEP] is the separator.

3. The method according to claim 1, characterized in that In step 2), the S-BERT model is used to retrieve sample documents that are most relevant to the target document.

4. The method according to claim 1, characterized in that In step 3), the S-BERT model is used to retrieve the sample labels that are most relevant to the target document.

5. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium, characterized in that: A computer program is stored, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images