A document-level event extraction method based on co-reference resolution

Abstract

The present application relates to natural language processing and deep learning technical field, especially relates to a kind of based on co-reference resolution document-level event extraction method.The main technical scheme of the present application includes: obtaining all named entities in document by named entity recognition;Co-reference resolution is used to group entities representing the same thing together to obtain co-reference entity set;The document is processed using entity interaction, sentence interaction and co-reference relationship to obtain a homogeneous graph with sentences as nodes;Sentences with similar semantics or themes are grouped together, and a sentence-level attention mechanism is used for noise reduction to obtain a sentence group;The sentence group is classified to obtain event types and further corresponding arguments.The extraction method of the present application can effectively address the problem that existing document-level event extraction methods do not consider co-reference resolution and the influence of noise within the sentence group.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and deep learning, and in particular to a document-level event extraction method based on coreference disambiguation. Background Technology

[0002] As a crucial component of information extraction, event extraction technology is a key technique for rapidly understanding the essential content of events. An event itself is a specific form of information, referring to an event that occurs at a specific time and place, involving one or more participants, each playing a different role. Event extraction technology aims to extract this type of information from unstructured natural text and assemble it into structured knowledge. Specifically, it is defined as identifying all events of the target event type in a given natural language sentence based on pre-specified event types and argument roles, and extracting the corresponding arguments (i.e., entities that play certain roles in the event). For example: "Wang Xiaoming was born on September 27, 2022, in Nanjing, Jiangsu." The event type in this sentence is "birth," and the argument roles include name, date of birth, and place of birth. The arguments corresponding to these roles in this sentence are "Wang Xiaoming," "September 27, 2022," and "Nanjing, Jiangsu."

[0003] Based on data granularity, event extraction can be divided into sentence-level and document-level. Compared to single-sentence event extraction, document-level event extraction is more intuitive, but correspondingly, it is also more difficult. The main challenges of document-level event extraction are: the arguments of an event may be scattered across multiple sentences in a document; and multiple different events are mixed in a document, requiring differentiation and identification of corresponding arguments for different events.

[0004] To address the aforementioned challenges, on the one hand, Zheng et al. proposed a new dataset, ChFinAnn, which is 10 times larger than the previous dataset, addressing the issue that a document in the previous DCFEE dataset contained only one key event and could not effectively solve the problem of scattered arguments. On the other hand, Zheng et al. proposed a novel end-to-end model, Doc2EDAG, which generates an entity-based directed acyclic graph for document-level event extraction. This transforms the form-filling task into several sequentially expanding path subtasks, which are more tractable. Furthermore, the extraction task is treated as not requiring trigger words, simplifying document-level event annotation.

[0005] To alleviate the difficulty of encoding long document sequences, in 2021, Huang et al. utilized the entity interaction characteristics of long texts—that is, entities existing in the same sentence have a higher probability of becoming arguments for the same event—and the sentence interaction characteristics—that is, sentences containing the same entity tend to describe the same event—to transform each document into an undirected, unweighted graph. Each event can be represented as a subgraph called a sentence community. Graph attention networks are used to extract multiple events, and the role overlap problem—that is, the phenomenon that an argument can play multiple roles—is alleviated by predicting arguments based on roles.

[0006] It is worth noting that document data, especially formal documents such as announcements and notices, contains a large amount of linguistic coreference, meaning that multiple expressions of an entity all refer to the same entity in real life. Current methods often fail to address this, leading to information omissions in the document graph constructed using entity interaction characteristics, and also resulting in a single reference answer for the model, increasing the learning difficulty. Furthermore, current methods, when detecting sentence groups—that is, identifying sets of sentences expressing the same event—assume that any sentence containing the target entity belongs to that sentence group. This approach is too crude, introducing noisy sentences that interfere with event extraction from sentence groups. Summary of the Invention

[0007] The purpose of this invention is to address the problems of existing technologies neglecting coreference in documents and sentence group noise. On the one hand, a document-level event extraction method based on coreference disambiguation aims to construct a more complete document graph. On the other hand, a sentence-level attention mechanism is used to suppress noise, thereby improving the performance of the event extraction model.

[0008] To achieve the above objectives, this invention provides a document-level event extraction method based on coreference disambiguation, the method comprising:

[0009] S1. For the input document, obtain all named entities in the document through named entity recognition and form a named entity set;

[0010] S2. For the input document, obtain several sets of core reference entities using the core reference disambiguation method;

[0011] S3. Using named entity sets, sentence interactions, and core reference sets, transform the document into an isomorphic graph with sentences as nodes;

[0012] S4. Extract features within the sentence to obtain sentence nodes s i eigenvector v i ;

[0013] S5. Apply an attention mechanism to the feature vector of each sentence node on the isomorphic graph to group sentences with similar semantics or themes together to generate sentence groups.

[0014] S6. Classify sentence groups to obtain event types and determine argument roles and arguments.

[0015] Further, step S2 includes:

[0016] S21. Perform word segmentation on each sentence in the document, and then encode it to generate word representations;

[0017] S22. Construct heterogeneous graphs based on words and constituents for documents using syntactic dependency trees and constituent trees;

[0018] S23. Input the heterogeneous graph into the convolutional neural network to obtain a fused representation of words that integrates local syntactic information and global document features;

[0019] S24. Based on the word fusion representation, obtain the span representation of the entire document;

[0020] S25. Enumerate each span representation, obtain the antecedent corresponding to the current span representation based on the coreference score, group entities representing the same thing together, and obtain a sequence of coreference entity sets.

[0021] Furthermore, step S22, which involves constructing a heterogeneous graph, includes:

[0022] (1) Each word representation is treated as a word node in the heterogeneous graph, and the other nodes in the syntactic component tree besides the words are treated as component nodes. The representation of the component node is calculated from the representation of the word node:

[0023]

[0024] in, It is component node c i The embedding representation of the first word below it in the constituent tree. It is component node c i The embedding representation of the last word below it in the constituent tree. It is component node c i The corresponding component type embedding representation;

[0025] (2) Edges in the heterogeneous graph are represented by the adjacency matrix A∈R(M+T)×(M+T), where an element of 0 indicates that the nodes are not connected. Edges in the syntactic dependency tree and the syntactic component tree constitute syntactic dependency edges and syntactic component edges, respectively. If a word node is located at the leftmost or rightmost position of a component node in the syntactic component tree, then a syntactic component edge is formed (e.g., ...). Figure 3(As shown by the dashed line); connect the syntactic root of a sentence to the root of the previous and next sentences to form adjacent sentence edges; connect each word to the previous and next words to form adjacent word edges; form self-loop edges on all nodes of the graph.

[0026] Further, step S23 includes:

[0027] In a graph convolutional network, the fused representation of each node i is iteratively updated.

[0028]

[0029] in, Let v(i) represent the fused representation of the i-th word generated by the k-th layer of the graph convolutional neural network, and v(i) represent a set of neighboring nodes of the i-th node. and The parameters of the k-th layer are represented when the edge type between node i and node u is l, where l takes values ​​from 0 to 5, and f is the ReLU activation function.

[0030] Furthermore, the span in step S24 is represented as: Where, x start(i) and x end(i) The node representation of the start and end boundaries of the span. It is a span i The sum of weights represented by all nodes in the attention mechanism is calculated using the following formula:

[0031] α t =FFNN α (x t )

[0032]

[0033]

[0034] Among them, FFNN α The first feedforward neural network is used to fuse the representation x of each word. t This is mapped to a non-standard attention score.

[0035] Further, in step S25, the step of obtaining the antecedent includes:

[0036] For span i Select the j-th span preceding it. j The co-reference score s(i,j) for the two spans is calculated using the following formula:

[0037]

[0038] s m (i)=FFNN m (g i )

[0039] s(i,j)=s m (i)+s m (j)+s a (i,j)

[0040] Among them, FFNN sa This is the second feedforward neural network, where ° represents the vector positional multiplication operation, and s m (i) represents the score for an entity mention with span i, FFNN m This is the third feedforward neural network.

[0041] Further, step S4 includes:

[0042] S40. For the s corresponding to the i-th node in the isomorphic graph i Determine all named entities contained within it, and denote the number of named entities as l;

[0043] S41. For the i-th sentence s i Feature encoding is performed, and BERT encoding is used to obtain the embedding matrix B of the last layer. i Max pooling is performed on the embedding matrix (pooling is a current technique and will not be elaborated further). The i-th sentence s i The feature encoding of the t-th entity contained herein is:

[0044] e t =maxpool(B i,j B i,j+1 ,…,B i,k )

[0045] In this sentence, j and k are the first and last words of entity t in sentence s. i Repeat this operation at the position in the middle to obtain sentence s. i The feature codes e1, e2, ..., e of all entities in the array l ;

[0046] S42. Max pool the encoded features of all entities to obtain the entity-level feature vector α. i :

[0047] α i =maxpool(e1,e2,…,e l )

[0048] Similarly, in B i Max pooling is used to obtain the sentence-level feature vector β. i :

[0049] β i =maxpool(B i,1 B i,2 ,…,B i,r )

[0050] Where r is sentence s i The number of words, i.e., the number of tokens in the BERT encoding;

[0051] S43. Using the second BiLSTM network, obtain the feature vector v of the node. i :

[0052] v i =Bi-LSTM(α) i ||β i )

[0053] In this context, || represents a concatenation operation.

[0054] S44. Repeat steps S41-S43 to obtain the feature vector corresponding to each sentence node in the isomorphic graph.

[0055] Further, step S5 includes:

[0056] S51. The input to the graph attention mechanism includes the adjacency matrix A of the isomorphic graph. uv The node eigenvectors v = [v1, v2, ..., v n In the graph attention mechanism, the importance of neighbor node j to the current node i is calculated using an attention score α. ij The formula is as follows:

[0057]

[0058] Where σ represents the LeakyReLU activation function, f represents the fully connected layer in the attention mechanism, and N i Indicates all neighboring nodes of node i;

[0059] S52. To capture more information, a multi-head attention mechanism with K heads is adopted:

[0060]

[0061] Here, || represents the concatenation operation. This represents the score of node j under the attention of the k-th graph with respect to the current node i;

[0062] The output v′ of the K-header attention i The values ​​of i = 1, 2, ..., n are superimposed to obtain a prediction matrix Z. This matrix is ​​then fed into a multilayer perceptron (MLP) and finally normalized to obtain the node v.i The predicted probability of belonging to the j-th sentence group:

[0063]

[0064] If node v i The probability of the group corresponding to the j-th sentence If it exceeds the predetermined value, then node v will be... i Assigned to the group of sentences;

[0065] Repeat the above steps. Sentences with similar semantics or information will be grouped into the same sentence group. Finally, m sentence groups will be output, with each sentence group representing an event.

[0066] S53. Construct a membership matrix F = {0, 1} for the document. n×m If the i-th sentence is within the group of the j-th sentences, then set F. i,j =1, otherwise 0.

[0067] Further, step S6 includes:

[0068] S61. Calculate the feature encoding sequence of all nodes v = [v1, v2, ..., v...]. n ] and membership matrix F∈{0,1+ n×m After multiplying by their positions, we get a matrix sequence C = [c1, c2, ..., c m ]:

[0069]

[0070] Among them, c i The feature vectors of all sentences in the i-th sentence group are retained. This indicates positional multiplication;

[0071] S62. For each sentence group c i A sentence-level attention mechanism is used to obtain the score α for each sentence. j :

[0072] g j =tanh(W sa c i,j +b sa )

[0073]

[0074] Among them, W sa and b sa For the parameters of the attention mechanism, g j The score before normalization;

[0075] S63. Calculate the output sentence community c through a fully connected layer.i Fusion coding representation and event type probability distribution The formula is:

[0076]

[0077]

[0078] Where y ET Indicates the event type label, W ET and b ET These are the parameters for a fully connected network.

[0079] Highest probability Corresponding y ET This refers to the event type corresponding to the sentence group. By using the predefined correspondence between event types and argument roles, the argument roles for each event type are obtained.

[0080] Repeat the above steps to obtain the event type and argument role corresponding to each sentence group;

[0081] S64. Use the attention score α obtained in step S62. j By weighted summation of attention scores, the final representation E of entities in the sentence group is obtained. entity :

[0082]

[0083] The final representation E of the entities in the sentence group entity The input is fed into the sigmoid function to simulate the relative scores of argument classification:

[0084]

[0085] Where W and b are fully connected parameters. For each argument role, the entity with the highest score that exceeds a predetermined threshold is selected as the argument corresponding to that argument role.

[0086] Repeat the above steps to output the arguments corresponding to each argument role.

[0087] Furthermore, if the i-th sentence is within the group of the j-th sentences, then set F. i,j =1, otherwise 0.

[0088] The document-level event extraction method based on coreference disambiguation provided by this invention has the following advantages compared with existing document-level event extraction methods:

[0089] (1) This invention takes into account the phenomenon of entity coreference disambiguation and reuses important information that has been ignored.

[0090] (2) This invention introduces a sentence-level attention mechanism, which suppresses sentence noise introduced during event classification and argument classification, reduces interference from irrelevant sentences, and effectively solves the problem of argument dispersion in document-level event extraction.

[0091] (3) The sentence groups obtained by the present invention can provide explanations for the extracted events. Attached Figure Description

[0092] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0093] Figure 1 This is a flowchart illustrating a document-level event extraction method based on coreference disambiguation according to an embodiment of the present invention.

[0094] Figure 2 This is a schematic diagram of the named entity recognition process according to an embodiment of the present invention;

[0095] Figure 3 This is a schematic diagram of component nodes in a syntactic component tree according to an embodiment of the present invention;

[0096] Figure 4 This is a schematic diagram of the coreference disambiguation process according to an embodiment of the present invention. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0098] This invention provides a document-level event extraction method based on coreference disambiguation, such as... Figure 1 As shown, the method includes the following steps:

[0099] S1. For the input document, obtain all named entities in the document through named entity recognition and form a named entity set;

[0100] S2. Obtain several sets of core reference entities through the core reference disambiguation method; the entities in each set of core reference entities represent the same thing, and they are treated as the same entity when performing event extraction;

[0101] S3. Using named entity sets, sentence interactions, and core reference entity set sequences (i.e., core reference relations, which use the output of S2), the document is transformed into an isomorphic graph with sentences as nodes.

[0102] S4. Extract features within the sentence, including entity-level features and sentence-level features, to obtain sentence nodes s. i eigenvector v i ;

[0103] S5. Apply an attention mechanism to the feature vector of each sentence node on the isomorphic graph to group sentences with similar semantics or themes together to obtain sentence groups.

[0104] S6. Classify sentence groups, introduce sentence-level attention mechanisms to mitigate the impact of noise, obtain event types, and find the corresponding arguments based on predefined argument roles.

[0105] In step S1, named entity recognition aims to obtain all named entities in the document. This step is fundamental to event extraction because an event must contain at least one subject. The specific steps are as follows: Figure 2 As shown, it includes:

[0106] S11. The sentence is segmented into multiple words. The word sequence is then encoded using a transformer encoder to obtain the vector representation of each word, i.e., the word vector.

[0107] S12. Input the word vectors into the first BiLSTM network. The first BiLSTM network learns the context information and uses the BIO annotation method to output the score of each word for each label. In the BIO annotation method, B represents the beginning of a noun phrase, I represents the middle of a noun phrase, and O represents a non-noun phrase. B and I can be further subdivided, for example, B-Person represents the beginning of a person's name, and I-Organization represents the middle of an organization's name. Output the score of each word for the corresponding BIO label.

[0108] S13. Input the score output by the first BiLSTM network into the Conditional Random Field (CRF) layer. The purpose of the CRF is to use the order dependency information between labels to filter out some erroneous labeled sequences, such as the case where "there is no B label before the I label". The output of the CRF layer is the score of different sequence labels, and the one with the highest score is the BIO label sequence corresponding to the word.

[0109] S14. Obtain named entities through the BIO tag sequence. Identify the BIO tag sequence; when a B tag is detected, it represents the beginning of a named entity. Continue searching until an O tag (excluding O) is found, representing the end of a named entity. Traversing the entire BIO tag sequence yields all named entities in the document.

[0110] In step S2, the purpose of coreference disambiguation is to identify different expressions in a text that refer to the same entity. For example, an article might begin with "Shanghai" or "the city that never sleeps," and later might say "this city." These different expressions actually refer to the same entity. Coreference disambiguation can link different expressions scattered throughout a document to the same event element, enhancing text understanding and aiding event extraction. Specific steps are as follows... Figure 4 As shown, it includes:

[0111] S20. Construct a syntactic dependency tree and a syntactic component tree for each sentence in the document. This is existing technology and will not be described in detail here.

[0112] S21. Segment the sentence into words, resulting in multiple words. Encode each word using a BERT encoder to obtain a vector representation of the word (for English, a variant of the BERT encoder, SpanBERT, can be used; for Chinese data, the standard BERT encoder can be used). After segmentation, a document W = (w1, w2, ..., w...) containing T words... T ), where w i Representing the i-th word, using SpanBERT or BERT as the encoder, the final output word representation is H. W =(h1,h2,…,h T ) represents the representation with the largest context.

[0113] S22. Construct heterogeneous graphs based on words and constituents for documents using syntactic dependency trees and constituent trees;

[0114] S23. Apply a convolutional neural network to the heterogeneous graph, fuse local syntactic information and global document features, and output a fused representation of each word;

[0115] S24. Based on the fusion representation of each word obtained in S23, obtain the representation of all spans;

[0116] S25. Enumerate the span before each span, obtain the antecedent corresponding to the current entity based on the coreference score, and group the entities representing the same thing together to obtain the coreference entity set.

[0117] In step S22, a heterogeneous graph is constructed for the document, including the following steps:

[0118] Represent the word H WEach word in the graph represents a word node in the heterogeneous graph, while constituent nodes are other nodes besides words in the syntactic constituent tree (such as...). Figure 3 As shown in the figure, it is represented as C = (c1, c2, ..., c M The representation of a component node is calculated from the representation of a word node:

[0119]

[0120] in, It is component node c i The embedding representation of the first word below it in the constituent tree. It is component node c i The embedding representation of the last word below it in the constituent tree. The component node c is obtained from the lookup table. i The lookup table is a predefined structure consisting of component types and their corresponding embeddings. Components come in various types, including subject, object, attributive, adverbial, complement, headword, and "Unknown." If a component type is known, its corresponding embedding can be found in the lookup table. Finally, the representation of the component nodes is obtained.

[0121] The edges of the heterogeneous graph are constructed below. The edges in the heterogeneous graph are connected through the adjacency matrix A∈R. (M+T)×(M+T) This indicates that 0 represents a non-connected edge, and other numbers represent different types. There are five main types of edges: syntactic dependency edges, syntactic component edges, adjacent sentence edges, adjacent word edges, and self-loop edges. Syntactic dependency edges and syntactic component edges originate from the syntactic dependency tree and syntactic component tree of each sentence in the document. Besides the edges already present in the trees, if a word node is located at the leftmost or rightmost position of a component node in the syntactic component tree, it constitutes a syntactic component edge (e.g., ...). Figure 3 (As shown by the dashed lines); adjacent sentence edges connect the syntactic root of a sentence (the syntactic roots of the syntactic component tree and the syntactic dependency tree are the same) to the root of the previous and next sentences; adjacent word edges (the words in the syntactic component tree and the syntactic dependency tree are also the same, because they are trees constructed from the same sentences) connect each word to the previous and next words; finally, self-looping edges are formed on all nodes of the graph. This yields a heterogeneous graph represented by the adjacency matrix A.

[0122] In step S23, a convolutional neural network is applied to the heterogeneous graph. In the graph convolutional neural network, parameter values ​​are set for each edge type: syntactic dependency edges, syntactic component edges, adjacent sentence edges, adjacent word edges, and self-loop edges have parameter values ​​of 1 to 5, respectively; the parameter value is 0 if there is no edge. Typically, graph convolutional neural networks only distinguish whether an edge exists between two nodes using 0 and 1, without specifically setting the edge values. The five edge types are one of the innovations of this invention. Among them, syntactic dependency edges, syntactic component edges, adjacent word edges, and self-loop edges are used to represent the syntactic information of the document, considering local syntactic information. Adjacent sentence edges are used to represent information between sentences, i.e., the global features of the document. Integrating them into the node representation can fully utilize the semantic and structural information of the nodes, enhancing the understanding of the text.

[0123] In a graph convolutional network, the fused representation of each node i is iteratively updated.

[0124]

[0125] in, Let v(i) represent the i-th fused representation generated by the k-th layer of the graph convolutional neural network, and v(i) represent a set of neighboring nodes of the i-th node. and This represents the parameters of the k-th layer when the edge type between node i and node u is l (l takes values ​​from 0 to 5), and f is the ReLU activation function. Output the fused representation for each word.

[0126] In this step, the fusion representation corresponding to a word combines word node representation and constituent node representation, which means it combines the global features and syntactic features of the document.

[0127] In step S24, for the fusion representation of each word, a span representation is generated according to the order of words in the document.

[0128] (1) Take the first word in the document, and then set the span length to 1 to L (L represents the maximum length limit of the span, which is to reduce the amount of calculation. L is a hyperparameter that can be adjusted by yourself), to obtain a set of span representations starting with the first word;

[0129] (2) Similarly, we get a span representation starting with the second word... until we get a span representation starting with the nth word and ending with the nth word (n is the number of words in the document).

[0130] (3) Form a span representation sequence in the above order.

[0131] A span refers to a continuous sequence of words or characters in a text, usually indicated by a start and end position. For example, all possible spans in the sentence "I love Shanghai" are "I, I love, I love Shanghai, love, love Shanghai, Shanghai, Shanghai, sea".

[0132] For the i-th span i Its span is expressed as: Where, x start(i) and x end(i) The node representation of the span's beginning and end boundaries, such as "I love Shanghai", is a node representation where the beginning boundary is "I" and the end boundary is "sea". It is a span i The sum of weights representing all nodes in the attention mechanism is calculated using the following formula to fully consider the contextual information across the span:

[0133] α t =FFNN α (x t )

[0134]

[0135]

[0136] Among them, FFNN α The first feedforward neural network is used to fuse the representation x of each word. t This is mapped to a non-standard attention score. During training, this neural network, along with other parameters in this method, is updated using the backpropagation algorithm.

[0137] In step S25, the span representation sequence is traversed. For each span representation, its coreference score with the preceding span representation is calculated to obtain the antecedent of that span representation, i.e., the preceding span representation that has the same meaning as that span representation. The steps for obtaining the antecedent of a span representation are described below, including:

[0138] For span i Select the j-th span preceding it. j The co-reference score s(i,j) for the two spans is calculated using the following formula:

[0139]

[0140] s m (i)=FFNN m (g i )

[0141] s(i,j)=s m(i)+s m (j)+s a (i,j)

[0142] Among them, FFNN sa This is the second feedforward neural network, where ° represents the vector positional multiplication operation. s m (i) represents the score for an entity mention with span i, FFNN m This is the third feedforward neural network.

[0143] The coreference score is influenced by three factors, s m (i) The score represents the span, indicating whether i is an entity; s m (j) The score represents the span, indicating whether j is an entity; s a (i,j) represents whether span representation j is an antecedent of span representation i. The span representation with the highest coreference score that is above a predetermined threshold is considered an antecedent of the current span representation. In this case, the span representation is a named entity. Therefore, the entities corresponding to these two span representations are put into a coreference entity set, and it is assumed that they refer to the same entity.

[0144] After traversing all spans, several sets of core-referenced entities are obtained, forming a sequence of core-referenced entity sets.

[0145] In step S3, the document is transformed into an isomorphic graph with sentences as nodes by utilizing named entity sets, sentence interactions, and sequences of coreference entity sets (i.e., coreference relations). Using isomorphic graphs with sentences as nodes to model long-sequence text can effectively capture the document's logical structure information, and the graph's structure has the ability to transfer information from distant sentences to related sentences, thus alleviating long-term dependency problems.

[0146] (1) Named entity set: Multiple entities that exist in the same sentence have a high probability of becoming arguments of the same event. For example, "A certain company holds how many shares of this company, and what is the price of each share". Although the number of shares and the price per share are different entities, they actually describe the same event and are arguments of the event.

[0147] (2) Sentence interaction: Several sentences containing the same entity tend to describe the same event;

[0148] (3) Entity set sequence: Different entities may actually represent the same thing.

[0149] The specific steps for constructing an isomorphic graph G = {V, E} for the document include:

[0150] Each sentence in the document is a node in the graph; construct edges. The set of edges E = {(u,v)∈V×V:A} uv =1+, where Auv ={0,1} N×N It is a binary adjacency matrix; for two sentences s i and s j Compare A with the named entity set in step S1 and the core reference entity set sequence in step S2 respectively. If they contain the same named entity (sentence interaction feature) or core reference entities representing the same thing, assign A... i,j Assign a value of 1 otherwise, and add a self-loop to the adjacency matrix A (entity interaction characteristic, entities within the same sentence have a high probability of describing the same event), i.e., A i,i =1.

[0151] In step S4, based on the isomorphism graph, features within the sentence are extracted, including entity-level features and sentence-level features, to obtain sentence node s. i eigenvector v i The specific steps include:

[0152] S40. For the s corresponding to the i-th node in the isomorphic graph i Determine all named entities contained within it, and denote the number of named entities as l;

[0153] S41. For the i-th sentence s i Feature encoding is performed, and BERT encoding is used to obtain the embedding matrix B of the last layer. i Max pooling is performed on the embedding matrix (pooling is a current technique and will not be elaborated further). The i-th sentence s i The feature encoding of the t-th entity contained herein is:

[0154] e t =maxpool(B i,j B i,j+1 ,…,B i,k )

[0155] In this sentence, j and k are the first and last words of entity t in sentence s. i In the sentence "Wang Xiaoming was born in Nanjing", the entity "Wang Xiaoming" has j=1 and k=3, while the entity "Nanjing" has j=7 and k=8.

[0156] Repeat this operation to obtain sentence s. i The feature codes e1, e2, ..., e of all entities in the array l

[0157] S42. Max pool the encoded features of all entities to obtain the entity-level feature vector α. i :

[0158] α i =maxpool(e1,e2,…,el )

[0159] Similarly, in B i Max pooling is used to obtain the sentence-level feature vector β. i

[0160] β i =maxpool(B i,1 B i,2 ,…,B i,r )

[0161] Where r is sentence s i The number of words, i.e., the number of tokens in the BERT encoding;

[0162] S43. Using the second BiLSTM network, obtain the feature vector v of the node. i :

[0163] v i =Bi-LSTM(α) i ||β i )

[0164] In this context, || represents a concatenation operation.

[0165] S44. Repeat steps S41-S43 to obtain the feature vector corresponding to each sentence node in the isomorphic graph.

[0166] In step S5, within the isomorphic graph, an attention mechanism is applied to the feature vector corresponding to each sentence node to group sentences with similar semantics or themes together, resulting in sentence groups; specifically, this includes:

[0167] S51. The input to the graph attention mechanism includes the adjacency matrix A of the isomorphic graph. uv The node eigenvectors v = [v1, v2, ..., v n In the graph attention mechanism, the importance of neighbor node j to the current node i is calculated using an attention score α. ij The formula is as follows:

[0168]

[0169] Where σ represents the LeakyReLU activation function, f represents the fully connected layer in the attention mechanism, and N i This represents all neighboring nodes of node i.

[0170] S52. To capture more information, a multi-head attention mechanism with K heads is adopted:

[0171]

[0172] Here, || represents the concatenation operation. Let represent the score of node j under the attention of the k-th graph with respect to the current node i.

[0173] The output v′ of the K-header attention i The values ​​of i = 1, 2, ..., n are superimposed to obtain a prediction matrix Z. This matrix is ​​then fed into a multilayer perceptron (MLP) and finally normalized to obtain the node v. i The predicted probability of belonging to the j-th sentence group:

[0174]

[0175] If node v i The probability of the group corresponding to the j-th sentence If the value is greater than a predetermined value (e.g., 0.5), then node v will be... i Assigned to the group of sentences.

[0176] Repeat the above steps. Sentences with similar semantics or information will be grouped into the same sentence group. Finally, m sentence groups will be output, with each sentence group representing an event.

[0177] S53. Construct a membership matrix F = {0, 1} for the document. n×m If the i-th sentence is within the group of the j-th sentences, then set F. i,j =1, otherwise 0.

[0178] In step S6, sentence groups are classified to obtain corresponding arguments, specifically including:

[0179] S61. Calculate the feature encoding sequence of all nodes v = [v1, v2, ..., v...]. n The membership matrix F∈{0,1} n×m After multiplying by their positions, we get a matrix sequence C = [c1, c2, ..., c m ]:

[0180]

[0181] Among them, c i The feature vectors of all sentences in the i-th sentence group are retained. This indicates positional multiplication.

[0182] S62. For each sentence group c i A sentence-level attention mechanism is used to obtain the score α for each sentence. j :

[0183] g j =tanh(W sa c i,j +bsa )

[0184]

[0185] Among them, W sa and b sa For the parameters of the attention mechanism, g j This is the score before normalization.

[0186] S63. Calculate the output sentence community c through a fully connected layer. i Fusion coding representation and event type probability distribution The formula is:

[0187]

[0188]

[0189] Where y ET Indicates the event type label, W ET and b ET These are the parameters for a fully connected network.

[0190] Highest probability Corresponding y ET This refers to the event type corresponding to the sentence group. By using the predefined correspondence between event types and argument roles, the argument roles for each event type are obtained.

[0191] Repeat the above steps to obtain the event type and argument role corresponding to each sentence group.

[0192] S64. Use the attention score α obtained in step S62. j By weighted summation of attention scores, the final representation E of entities in the sentence group is obtained. entity :

[0193]

[0194] To mitigate the role overlap problem—that is, an entity may play multiple argument roles—the final representation E of entities in a sentence group is determined by... entity Instead of a regular softmax classifier, the input is fed into a sigmoid function to simulate the relative scores of argument classification:

[0195]

[0196] Where W and b are fully connected parameters, and the fully connected parameters are different for different argument roles. These parameters will be updated along with other network parameters in the method of this invention during training.

[0197] For each argument role, the entity with the highest score that exceeds a predetermined threshold is selected as the argument corresponding to that argument role.

[0198] Repeat the above steps to output the arguments corresponding to each argument role.

[0199] The networks and their parameters involved in the above steps need to be trained before use. The networks involved include first and second BiLSTMs, Conditional Random Fields (CRFs), feedforward neural networks, graph convolutional neural networks, and fully connected layers. They are trained on the training set of the dataset, and the network parameters are uniformly adjusted by optimizing the loss function.

[0200] The following uses data from the ChFinAnn dataset as an example to illustrate the document-level event extraction method based on coreference disambiguation. Some parameters during the training process are shown in Table 1.

[0201] Before training the model, the ChFinAnn dataset was first divided into training and testing sets. All data were manually annotated by crowdsourced reviewers to ensure an accuracy rate higher than 95%. Training was performed on the training dataset. Input documents were processed using BERT encoding, BiLSTM, and Conditional Random Field (CRF) to obtain all entities within the documents. Then, SpanBERT encoding, convolutional neural networks, and attention mechanisms were applied to the documents to obtain different entity sets, where entities within each set represent the same thing. The documents were then transformed into isomorphic graphs, and sentences with the same semantics or topic were grouped together to form sentence groups. Noise reduction was performed using sentence-level attention mechanisms. Finally, the sentence groups were classified to obtain event types and corresponding arguments.

[0202] After training the entire model using the above steps, a validation and comparison experiment was conducted using a segmented test set. The experimental results are shown in Table 2. Table 2 shows that the document-level event extraction method based on coreference disambiguation proposed in this invention improves the F1 score by 21.7 compared to earlier sentence-level event extraction models; and by 0.9 compared to the latest document-level event extraction model. This indicates that the method proposed in this invention effectively improves the performance of document-level event extraction through coreference disambiguation and sentence-level attention mechanisms.

[0203] Table 1 Training parameter settings

[0204]

[0205] Table 2 shows the experimental results (the last row shows the results obtained using the method of this invention).

[0206]

[0207] Those skilled in the art will understand that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for document-level event extraction based on co-reference resolution, characterized in that, The method comprises: S1, for the input document, obtaining all named entities in the document through named entity recognition to form a named entity set; S2, for the input document, obtaining a plurality of co-reference entity sets through a co-reference resolution method; S3, using the named entity set, sentence interaction and co-reference entity set, converting the document into an isomorphic graph with sentences as nodes; S4, extract features inside the sentence, obtain the sentence node s i v i ; S5, using an attention mechanism on the feature vector of each sentence node in the isomorphic graph, grouping sentences with similar semantics or themes together to generate a sentence group; S6, classifying the sentence group to obtain the event type, determining the argument role and argument; The step S2 comprises: S21, performing word segmentation processing on each sentence in the document, and then encoding to generate a word representation; S22, constructing a heterogeneous graph based on words and components for the document using a syntactic dependency tree and a constituent tree; S23, inputting the heterogeneous graph into a convolutional neural network to obtain a fusion representation of the word that fuses local syntactic information and global document features; S24, obtaining all span representations of the document according to the fusion representation of the word; S25, enumerating each span representation, obtaining the antecedent corresponding to the current span representation according to the co-reference score, and dividing entities representing the same thing together to obtain a sequence of co-reference entity sets; The step S5 comprises: S51, the input of the graph attention mechanism includes the adjacency matrix A of the isomorphic graph uv , the node feature vector v = [v1, v2, …, v n ], in the graph attention mechanism, the importance of the neighbor node j to the current node i is represented by the attention score α ij , and the formula is: wherein σ denotes a LeakyReLU activation function, f denotes a fully connected layer in the attention mechanism, N i denotes all neighbor nodes of node i; S52, in order to capture more information, a multi-head attention mechanism with K heads is used: wherein || denotes a concatenation operation, denotes the score of the k-th graph attention for the current node i from the node j. The output v′ of the K-head graph attention i are stacked to obtain a prediction matrix Z, which is then put into a multi-layer perception (MLP) and normalized to obtain the node v i The prediction probability belonging to the jth sentence group: If node v i is assigned to the sentence group corresponding to the jth sentence group if the probability is greater than a predetermined value; and i assigning node v Repeat the above steps, sentences with similar semantics or information will be divided into the same sentence group, and finally output m sentence groups, one sentence group representing one event; S53, construct membership matrix F = {0, 1} for the document n×m F = 1 if the ith sentence is within the jth sentence cluster, otherwise 0. i,j F = 1 if the ith sentence is within the jth sentence cluster, otherwise 0.

2. The method of claim 1, wherein, The step S22 of constructing the heterogeneous graph comprises: (1) taking each word representation as a word node in the heterogeneous graph, taking other nodes in the syntactic constituent tree except the words as component nodes, and obtaining the representation of the component nodes from the representation of the word nodes: wherein, is an embedding representation of the component node c i is an embedding representation of the first word under c in the component tree, is an embedding representation of the component node c i is an embedding representation of the last word under c in the component tree, is an embedding representation of the component node c i is an embedding representation of the corresponding component type; (2) The edges in the heterogeneous graph are represented by an adjacency matrix A ∈ R (M+T)×(M+T) where the element is 0 represents not connected, the edges in the syntax dependency tree and the syntax constituent tree constitute the syntax dependency edge and the syntax constituent edge, if the word node is located at the leftmost or rightmost of the constituent node in the syntax constituent tree, a syntax constituent edge is formed; the syntax root of the sentence and the root of the previous and next sentence are connected to form adjacent sentence edges; each word and the previous and next word are connected to form adjacent word edges; the self-loop edges are formed on all nodes of the graph.

3. The method of claim 1, wherein, The step S23 comprises: In a graph convolution network, the fused representation of each node i is iteratively updated wherein, denotes the fusion representation of the i-th word generated by the k-th layer of the graph convolutional neural network, v(i) denotes a set of adjacent nodes of the i-th node, and denotes the parameter of the k-th layer when the edge type between the node i and the node u is l, the value of l is 0-5, and f is a ReLu activation function.

4. The method of claim 1, wherein, The span in the step S24 is expressed as: where x start(i) and x end(i) represent the node representations of the head and tail boundaries of the span, is the sum of the weight of all node representations in the span span i is calculated according to the attention mechanism as follows: a t = FFNN α (x t ) where FFNN α is a first feed-forward neural network for mapping the fused representation x t of each word into non-standard attention scores.

5. The method of claim 1, wherein, In step S25, the step of obtaining the antecedent comprises: For span i , the jth span j before it is selected, and the co-reference score s(i,j) of the two spans is calculated as follows: s m (i) = FFNN m (g i ) s(i,j) = s m (i) + s m (j) + s a (i,j) where FFNN sa is a second feed-forward neural network, denotes a vector-by-phase multiplication operation, s m (i) denotes a score for span i being an entity mention, FFNN m is a third feed-forward neural network.

6. The method of claim 1, wherein, The step S4 comprises: S40、For the s corresponding to the i-th node in the isomorphic graph i , judge all the named entities contained, and the number of named entities is recorded as l; S41, the i-th sentence s i The feature encoding is performed, and the embedding matrix B of the last layer is obtained by using BERT encoding i The embedding matrix is subjected to a maximum pooling operation, and the feature encoding of the t-th entity contained in the i-th sentence s i The feature encoding of the t-th entity contained in the i-th sentence s e t = maxpool(B i,j ,B i,j+1 ,…,B i,k ) where j and k are the positions of the first and last words of entity t in sentence s i , and repeating the operation to obtain the feature encoding e1, e2, …, e i of all entities in sentence s l ; S42, maximum pooling of the encoding features of all entities to obtain an entity-level feature vector a i : a i = maxpool(e1, e2,..., en) l ) Similarly, in B i The sentence-level feature vector β is obtained by using max-pooling on B i : β i = maxpool(B i,1 ,B i,2 ,…,B i,r ) Where r is sentence s i The number of words, i.e., the number of tokens in the BERT encoding; S43, using the second BiLSTM network, obtaining the feature vector v of the node i : v i = Bi-LSTM(a i || β i ) Wherein, || represents the concatenation operation; S44, repeating steps S41-S43 to obtain the feature vector corresponding to each sentence node in the isomorphic graph.

7. The method of claim 1, wherein, Step S6 comprises: S61. Calculate the feature encoding sequence of all nodes v = [v1, v2, ..., v...]. n The membership matrix F∈{0,1} n×m After multiplying by their positions, we get a matrix sequence C = [c1, c2, ..., c m ]: where c i all sentence feature vectors in the i-th sentence group are reserved, denotes a multiplication of the phase. S62, for each sentence group c i , get the score a j for each sentence using a sentence-level attention mechanism. g j = tanh(W sa c i,j + b sa ) where W sa and b sa are parameters of the attention mechanism, and g j is the score before normalization. S63, compute output sentence cluster c by fully connected layer i of the fusion encoding representation and event type probability distribution is given by: where y ET represents the event type label, W ET and b ET are fully connected parameters; Most probable Corresponding y ET That is, the event type corresponding to the sentence group, and the argument role of each event type is obtained through the predefined correspondence between the event type and the argument role. Repeat the above operation to obtain the event type and argument role corresponding to each sentence group; S64, using the attention scores a obtained in step S62 j , by weighted summing the attention scores, obtaining the final representation E of the entity in the sentence group entity : Final representation E of an entity in a sentence community entity Input into the sigmoid function, which models the relative score of the argument classification: Wherein, W and b are full connection parameters, for each argument role, selecting the entity with the highest score and exceeding a predetermined threshold as the argument corresponding to the argument role; Repeat the above operation to output the argument corresponding to each argument role.

8. The method of claim 7, wherein, If the ith sentence is inside the jth sentence cluster, set F i,j = 1, otherwise 0.

Images