Hierarchical label event extraction method and system based on global attention and grid marking
By introducing global attention and grid labeling technology into the hierarchical label event extraction model, and using the BERT model and gating mechanism, we solve the problem of failing to fully utilize the hierarchical label semantic information in the existing technology, and achieve more efficient and accurate events. Extraction.
Patent Information
- Application Number
- CN202411492914.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-01-24
AI Technical Summary
When processing hierarchical label event extraction model fails to fully utilize the semantic information contained in hierarchical label text, resulting in error propagation and overlap when processing trigger words and arguments, which affects the accuracy of event extraction. and efficiency.
Adopt the level labeling event drawing method based on global attention and grid marking, convert the text of the hierarchical label event into character-words-to-line sequences through the BERT model, obtain hidden layers of the level label event, and use door control mechanisms and candidate events characteristics characteristics characteristics Integrate, build a grid labeling network to minimize the loss function to train the extraction model.
The semantic information of hierarchical label events is effectively used to reduce the dissemination of the errors of triggers and the component, improve the accuracy and efficiency of event extraction, and lays the foundation for building a richer and dynamic knowledge map.
Smart Images
Figure CN119357308A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information extraction in natural language processing, and in particular to a method and system for extracting hierarchical label events based on global attention and grid marking. Background Art
[0002] Hierarchical label event extraction refers to extracting specific event elements from unstructured text with hierarchical labels and presenting them in a structured form. It plays an important role in applications such as information retrieval and knowledge graph construction. However, the current event extraction model is based on character-level features, and the extraction method usually includes two stages: first, the identification of trigger words, and second, the extraction of arguments. Although this staged processing method can achieve good results in some cases, it also has an obvious problem, namely error propagation. Once the trigger word is recognized incorrectly, it will directly affect the extraction of subsequent arguments, resulting in a decrease in the accuracy of the entire event extraction. In addition, when processing hierarchical label text, existing event extraction technologies often fail to fully utilize the rich semantic information contained in hierarchical labels. In hierarchical label text, the trigger words and arguments of events may overlap, that is, the same word or phrase may play the role of both trigger words and arguments in different contexts. This phenomenon is relatively rare in text without hierarchical labels, but it is very common in hierarchical label text. Due to the lack of effective use of hierarchical label semantic information, existing event extraction models are often unable to cope with such texts and find it difficult to accurately identify the trigger words and arguments of events, thus affecting the quality and efficiency of event extraction. Summary of the invention
[0003] The main purpose of the present invention is to propose a hierarchical label event extraction method and system based on global attention and grid marking, which effectively utilizes semantic information to achieve more efficient and accurate hierarchical label event extraction.
[0004] The present invention is achieved through the following technical solutions: The hierarchical label event extraction method based on global attention and grid marking includes the following steps: Step S1, converting the hierarchical label event text composed of characters into a character-word pair sequence, using the BERT model to obtain character hidden layer vectors and character matching word vectors for characters and words respectively, and fusing the character hidden layer vectors and character matching word vectors to obtain the hierarchical label event text hidden layer vector; Step S2, vectorize the event type of the hierarchical label event to obtain an event type feature set, obtain a global attention representation of the hierarchical label event, fuse the global attention representation with the known event text hierarchical label representation to obtain a hierarchical label fused global attention representation, fuse the hierarchical label event text hidden vector with the hierarchical label fused global attention representation using a gating mechanism, and integrate the fused result with the candidate event type feature using a gating mechanism to obtain a fused word vector sequence of the hierarchical label event text, wherein the candidate event type feature belongs to the event type feature set; Step S3, constructing a grid labeling network, the loss function of which involves the span relationship score and role relationship score of the characters of the hierarchical label event text for the candidate event type features, the span relationship score and the role relationship score are obtained according to the scoring scores obtained by fusion of the two vectors of the word vector sequence and their respective position encodings, and the grid labeling network is trained to minimize the loss function to obtain an extraction model.
[0005] Furthermore, the step S1 specifically includes the following steps: Step S11: Set the level label event text Query dictionary with characters D trie Match and get the characters c i The corresponding matching word set ws i , and then get the character-word pair sequence, that is, ,in, i =1,2,…, n ; Step S12: c i Input the BERT model and get the Transformer layer output of the BERT model, recorded as the character hidden layer vector ; Step S13: c i The set of matching words ws i Input the BERT model to get the corresponding word vector set ,in, m Representing characters c i The number of words matched; Step S14: According to the formula Word vector collection Perform a nonlinear transformation, where W 1 Yes d c and dw Multiply, W 2 Yes d c and d c Multiply, b 1 and b 2 is the deviation, d w is the dimension of word embedding for character matching, d c is the character hidden layer vector Dimensions; Step S15: According to the formula calculate Middle j The confidence level of the word a ij ,in, W attn is the weight matrix of the bilinear mapping, for The j Quantity, softma (.) is the normalization function; Step S16: According to the formula Take the weighted sum of the matching words of each character and get the character matching word vector ; Step S17: According to the formula The character hidden layer vector Match word vectors with characters Fusion, get the level label event text hidden vector h i .
[0006] Furthermore, the step S2 specifically includes the following steps: Step S21: Express the event type of the hierarchical label event in vector form to obtain an event type feature set. , M is the number of event types; Step S22: Use the self-attention mechanism to obtain the global attention representation of the hierarchical label event , and obtain the hierarchical label fusion global attention representation ,in, Attention (.) represents the self-attention mechanism, W q , W k , W v is the trainable parameter of the self-attention mechanism, E hl It is the event text level label representation; Step S23: Use the gating mechanism to transform the level label event text hidden layer vector h i The global attention representation is fused with the level label to obtain the global attention character representation ,in, represents the gating mechanism; Step S24: Use the gating mechanism to represent the global attention character and candidate event types e t The features are integrated to obtain the fused word vector sequence of the hierarchical label event text ,in, .
[0007] Furthermore, in step S2, the gating mechanism is ,in, , p and q is the input vector, and [;] are element-wise dot product and concatenation operations, respectively. and are trainable parameters of the gating mechanism.
[0008] Furthermore, in step S3, the loss function of the grid labeling network is ,in, is the span relationship loss function, according to the formula calculate, is the role relationship loss function, calculated according to the formula , For character pairs About candidate event types e t The span relationship score is calculated according to the formula calculate, For character pairs About candidate event types e t The role relationship score is calculated according to the formula calculate, , W s1 , W s2 , W r1 , W sr is a trainable parameter, and is the fused word vector sequence corresponding to the character pair Two vectors of and is the position encoding of the two vectors,E for all event types.
[0009] The present invention is also achieved through the following technical solutions: A hierarchical label event extraction system based on global attention and grid labeling, including: Hierarchical label event text hidden vector acquisition module: used to convert the hierarchical label event text composed of characters into a character-word pair sequence, use the BERT model to obtain character hidden layer vectors and character matching word vectors for characters and words respectively, and merge the character hidden layer vectors and character matching word vectors to obtain the hierarchical label event text hidden layer vector; A fused word vector sequence acquisition module is used to express the event type of the hierarchical label event in vector form, obtain an event type feature set, obtain a global attention representation of the hierarchical label event, fuse the global attention representation with the hierarchical label representation of the known event text, obtain a hierarchical label fused global attention representation, fuse the hierarchical label event text hidden vector with the hierarchical label fused global attention representation using a gating mechanism, and integrate the fused result with the candidate event type feature using a gating mechanism to obtain a fused word vector sequence of the hierarchical label event text, wherein the candidate event type feature belongs to the event type feature set; Grid labeling network construction module: used to construct the loss function of the grid labeling network. The loss function involves the span relationship score and role relationship score of the characters of the hierarchical label event text for the candidate event type features. The span relationship score and role relationship score are obtained based on the scoring scores obtained by fusion of the two vectors of the word vector sequence and their respective position encodings. The grid labeling network is trained to minimize the loss function to obtain the extraction model.
[0010] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects: The present invention firstly uses the BERT model to obtain character hidden layer vectors and character matching word vectors for characters and words respectively, and then obtains the hidden layer vector of the hierarchical label event text, then vectorizes the event type of the hierarchical label event to obtain the event type feature set, and obtains the hierarchical label fusion global attention representation, and then uses the gating mechanism to obtain the fused word vector sequence, and finally constructs a grid labeling network, and designs a loss function according to the span relationship score and role relationship score of the characters of the hierarchical label event text for the candidate event type features, and then obtains the extraction model, so that the semantic information of the hierarchical label event can be effectively utilized, the problems of trigger word overlap and argument overlap in event extraction can be solved, and more efficient and accurate hierarchical label event extraction can be achieved, laying a solid foundation for building a richer and more dynamic knowledge graph and improving the performance of information retrieval systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0012] Figure 1 It is a flow chart of the present invention.
[0013] Figure 2 This is a schematic diagram of the process of predicting the text event type and event elements of nursing activity events according to the present invention. DETAILED DESCRIPTION
[0014] like Figure 1 As shown in Figure 1, the hierarchical label event extraction method based on global attention and grid marking includes the following steps: Step S1, converting the hierarchical label event text composed of characters into a character-word pair sequence, using the BERT model to obtain character hidden layer vectors and character matching word vectors for characters and words respectively, and fusing the character hidden layer vectors and character matching word vectors to obtain the hierarchical label event text hidden layer vector; In this embodiment, the experimental data comes from a self-built nursing activity event data set, which contains 11,870 nursing activity event texts, 16 event types, and 15,159 nursing activity events. Among them, there are 1,742 samples with overlapping trigger words, 1,690 samples with overlapping arguments, and 1,378 samples with overlapping trigger words and arguments. Table 1 shows an example of event types and event elements for the nursing activity text "record and report the health status of adults or children in accordance with community requirements".
[0015] Table 1
[0016] At the same time, the nursing activity text has three levels: domain, measure category, and measure name. The domain layer contains 7 tags, the measure layer contains 30 categories, and the measure name layer contains 583 measure names. Examples of nursing activity text and its hierarchical labels are shown in Table 2. "##" is the separator of the hierarchical label.
[0017] Table 2
[0018] Wherein, step S1 specifically includes the following steps: Step S11: Set the level label event text Query dictionary with characters D trie Match and get the characters c i The corresponding matching word set ws i , and then get the character-word pair sequence, that is, ,in,i =1,2,…, n ; In this embodiment, the level label event text is S ={record and report the health of adults or children as required by law} as an example, the resulting character-word pair sequence is Sc ={(according to,(according to, according to the law)),(according to,(according to, according to the law)),(law,(according to the law, law, legal requirements)),...,(item,(event))}; Step S12: c i Input the BERT model and get the Transformer layer output of the BERT model, recorded as the character hidden layer vector ; Step S13: c i The set of matching words ws i Input the BERT model to get the corresponding word vector set ,in, m Representing characters c i The number of matched words is insufficient m Fill with 0 when Step S14: Aligning character hidden layer vectors And word vector collection Two different expressions, according to the formula Word vector collection Perform a nonlinear transformation, where W 1 Yes d c and d w Multiply, W 2 Yes d c and d c Multiply, b 1 and b 2 is the deviation, d w is the dimension of word embedding for character matching, d c is the character hidden layer vector Dimensions; Step S15: According to the formula calculate Middle j The confidence level of the word a ij ,in,W attn is the weight matrix of the bilinear mapping, for The j Quantity, softma (.) is the normalization function; Step S16: According to the formula Take the weighted sum of the matching words of each character and get the character matching word vector ; Step S17: According to the formula The character hidden layer vector Match word vectors with characters Fusion, get the level label event text hidden vector h i .
[0019] Step S2, vectorize the event type of the hierarchical label event to obtain an event type feature set, obtain a global attention representation of the hierarchical label event, fuse the global attention representation with the known event text hierarchical label representation to obtain a hierarchical label fused global attention representation, fuse the hierarchical label event text hidden vector with the hierarchical label fused global attention representation using a gating mechanism, and integrate the fused result with the candidate event type feature using a gating mechanism to obtain a fused word vector sequence of the hierarchical label event text, wherein the candidate event type feature belongs to the event type feature set; In order to obtain the hidden vector of each level label event text The global attention representation is based on the Self-Attention in Transformer, and an attention mechanism is introduced. The calculation formula is as follows: , where Q is the query, K is the key value, and V is the value. d h are the dimensions of Q and K, is a scaling factor to prevent the dot product from being too large.
[0020] Then step S2 specifically includes the following steps: Step S21: Randomly initialize the embedding vector expression of the event type of the hierarchical label event to obtain the event type feature set , M =16 is the number of event types; Step S22: Use the self-attention mechanism to obtain the global attention representation of the hierarchical label event , incorporating event text level labels into the global attention representation In the above example, we get the hierarchical label fusion global attention representation ,in, Attention (.) represents the self-attention mechanism, W q , W k , W v is the trainable parameter of the self-attention mechanism, E hl is the event text level label representation. In this embodiment, the event text level label is E hl = {Health Systems##Information Management##Records}; Step S23: Use the gating mechanism to transform the level label event text hidden layer vector h i The global attention representation is fused with the level label to obtain the global attention character representation ,in, represents the gating mechanism; The gating mechanism is , where the gate vector Generated by the fully connected layer of the sigmoid function, it is used to control the input flow, that is, , p and q is the input vector, and [;] are element-wise dot product and concatenation operations, respectively. and are trainable parameters of the gating mechanism; Step S24: Use the gating mechanism to represent the global attention character and candidate event types e t The features are integrated to obtain the fused word vector sequence of the hierarchical label event text ,in, , .
[0021] Step S3, constructing a grid labeling network, wherein the loss function of the grid labeling network involves the span relationship score and the role relationship score of the characters of the hierarchical label event text with respect to the candidate event type features, wherein the span relationship score and the role relationship score are obtained according to the scoring scores obtained by fusion of two vectors of the word vector sequence and their respective position encodings, and the grid labeling network is trained to minimize the loss function to obtain an extraction model; Specifically, the steps are as follows: Step S31: For character pairs , calculate the score to measure its span relationship Relationship with roles The possibility of S represents a span relation set,R Represents a span relationship set. Since there are multiple character pairs, these two sets are formed): The fused word vector sequence Corresponding to the character pair Two vectors of and , with their respective position codes and Combine them and get the score by calculating their dot product, that is, , and then get the character pair About candidate event types e t The span relationship score Relationship score with character : , ,in, W s1 , W s2 , W r1 , W sr is a trainable parameter; Step S32: To solve the multi-event classification problem, a sub-loss function is designed while considering the convergence targets of relation pairs and non-relation pairs. , where * refers to span relationship or role relationship. For span relationship loss function , when the span relationship score Greater than the set threshold And the score of non-relationship pair is less than the threshold When , the span relation loss function Gradually converge, for the role relationship loss function , when the span relationship score Greater than the set threshold And the score of non-relationship pairs is less than the threshold When , the span relation loss function Gradually converge, in this embodiment, the threshold Set to 0.
[0022] The loss function of the grid labeling network is ,in, is the span relationship loss function, according to the formula calculate, is the role relationship loss function, according to the formula calculate, For character pairs About candidate event types e t The span relationship score is calculated according to the formula calculate, For character pairs About candidate event types e t The role relationship score is calculated according to the formula calculate, It is a collection of event type features.
[0023] For span relation loss function , when the span relationship score Greater than the set threshold And the score of non-relationship pairs is less than the threshold When , the span relation loss function Gradually converge, for the role relationship loss function , when the span relationship score Greater than the set threshold And the score of non-relationship pair is less than the threshold When , the span relation loss function Gradually converge. Among them, if the event type character pair If the pairing result really exists in the same event, it is a relational pair, otherwise, it is a non-relational pair. The judgment of whether the pairing result exists in the same event can be determined based on the annotation of the training data.
[0024] The hierarchical label event text to be tested is input into the extraction model. When the model predicts all the tags of a certain event type, it first obtains the start and end indexes of the trigger words and arguments, and obtains the trigger words and arguments based on S-*. At the same time, it matches the relationship between the trigger words and arguments based on R-*, where * is the argument role, and finally obtains a complete event structure.
[0025] like Figure 2 The model shown extracts the nursing activity event text "recording and reporting the health status of adults or children in accordance with legal requirements" of the "monitoring" event type and event element process. Since the event pattern defines the unique argument role of each event type, Figure 1 There are two "SP" role relationships in the event, so it is identified as two events. Repeating the above process can obtain all event elements of event type "record".
[0026] Correspondingly, the hierarchical label event extraction system based on global attention and grid labeling network includes: Hierarchical label event text hidden vector acquisition module: used to convert the hierarchical label event text composed of characters into a character-word pair sequence, use the BERT model to obtain character hidden layer vectors and character matching word vectors for characters and words respectively, and merge the character hidden layer vectors and character matching word vectors to obtain the hierarchical label event text hidden layer vector; A fused word vector sequence acquisition module is used to express the event type of the hierarchical label event in vector form, obtain an event type feature set, obtain a global attention representation of the hierarchical label event, fuse the global attention representation with the hierarchical label representation of the known event text, obtain a hierarchical label fused global attention representation, fuse the hierarchical label event text hidden vector with the hierarchical label fused global attention representation using a gating mechanism, and integrate the fused result with the candidate event type feature using a gating mechanism to obtain a fused word vector sequence of the hierarchical label event text, wherein the candidate event type feature belongs to the event type feature set; Grid labeling network construction module: used to construct the loss function of the grid labeling network. The loss function involves the span relationship score and role relationship score of the characters of the hierarchical label event text for the candidate event type features. The span relationship score and role relationship score are obtained based on the scoring scores obtained by fusion of the two vectors of the word vector sequence and their respective position encodings. The grid labeling network is trained to minimize the loss function to obtain the extraction model.
[0027] The present invention is further described below through specific implementation modes.
[0028] In the present invention, the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. In the description, the directions or positional relationships indicated by "upper", "lower", "left", "right", "front", and "back" are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention, and are not intended to indicate or imply that the device referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the scope of protection of the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0029] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0030] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.
Claims
1. A hierarchical label event extraction method based on global attention and grid marking, characterized by: The steps include: Step S1, converting the hierarchical label event text composed of characters into a character-word pair sequence, using the BERT model to obtain character hidden layer vectors and character matching word vectors for characters and words respectively, and fusing the character hidden layer vectors and character matching word vectors to obtain the hierarchical label event text hidden layer vector; Step S2, vectorize the event type of the hierarchical label event to obtain an event type feature set, obtain a global attention representation of the hierarchical label event, fuse the global attention representation with the event text hierarchical label representation to obtain a hierarchical label fused global attention representation, fuse the hierarchical label event text hidden vector with the hierarchical label fused global attention representation using a gating mechanism, and integrate the fused result with the candidate event type feature using a gating mechanism to obtain a fused word vector sequence of the hierarchical label event text, wherein the candidate event type feature belongs to the event type feature set; Step S3, constructing a grid labeling network, the loss function of which involves the span relationship score and role relationship score of the characters of the hierarchical label event text for the candidate event type features, the span relationship score and the role relationship score are obtained according to the scoring scores obtained by fusion of the two vectors of the word vector sequence and their respective position encodings, and the grid labeling network is trained to minimize the loss function to obtain an extraction model.
2. The hierarchical label event extraction method based on global attention and grid marking according to claim 1 is characterized in that: The step S1 specifically includes the following steps: Step S11: Set the level label event text and character query dictionary D trie Match and get character c i The corresponding matching word set ws i , and then get the character-word pair sequence, that is, , where i=1,2,…,n; Step S12: character c i Input the BERT model and get the Transformer layer output of the BERT model, recorded as the character hidden layer vector ; Step S13: character c i The matching word set ws i Input the BERT model to get the corresponding word vector set , where m represents the character c i The number of words matched; Step S14: According to the formula Word vector collection Perform nonlinear transformation, where W1 is d c With d w Multiply, W2 is d c With d c Multiply, b1 and b2 are deviations, d w is the dimension of character matching word embedding, d c is the character hidden layer vector Dimensions; Step S15: According to the formula calculate The confidence a of the jth word in ij , where W attn is the weight matrix of the bilinear mapping, for The j-th component in , softma(.) is the normalization function; Step S16: According to the formula Take the weighted sum of the matching words of each character and get the character matching word vector ; Step S17: According to the formula The character hidden layer vector Match word vectors with characters Fusion, get the level label event text hidden layer vector h i .
3. The hierarchical label event extraction method based on global attention and grid marking according to claim 2 is characterized in that: The step S2 specifically includes the following steps: Step S21: Express the event type of the hierarchical label event in vector form to obtain an event type feature set. , M is the number of event types; Step S22: Use the self-attention mechanism to obtain the global attention representation of the hierarchical label event , and obtain the hierarchical label fusion global attention representation , where Attention(.) represents the self-attention mechanism, W q , W k , W v is the trainable parameter of the self-attention mechanism, E hl It is the event text level label representation; Step S23: Use the gating mechanism to transform the level label event text hidden layer vector h i The global attention representation is fused with the level label to obtain the global attention character representation ,in, represents the gating mechanism; Step S24: Use the gating mechanism to represent the global attention character and candidate event type e t The features are integrated to obtain the fused word vector sequence of the hierarchical label event text ,in, .
4. The hierarchical label event extraction method based on global attention and grid marking according to claim 3 is characterized in that: In step S2, the gating mechanism is ,in, , p and q are input vectors, and [;] are element-wise dot product and concatenation operations, respectively. and are trainable parameters of the gating mechanism.
5. The hierarchical label event extraction method based on global attention and grid marking according to claim 1, 2 or 3, characterized in that: In step S3, the loss function of the grid labeling network is ,in, is the span relationship loss function, according to the formula calculate, is the role relationship loss function, calculated according to the formula , For character pairs About candidate event type e t The span relationship score is calculated according to the formula calculate, For character pairs About candidate event type e t The role relationship score is calculated according to the formula calculate, , W s1 , W s2 , W r1 , W sr is a trainable parameter, and is the fused word vector sequence corresponding to the character pair Two vectors of and are the position codes of the two vectors, and E is all event types.
6. A hierarchical label event extraction system based on global attention and grid labeling, characterized by: include: Hierarchical label event text hidden vector acquisition module: used to convert the hierarchical label event text composed of characters into a character-word pair sequence, use the BERT model to obtain character hidden layer vectors and character matching word vectors for characters and words respectively, and merge the character hidden layer vectors and character matching word vectors to obtain the hierarchical label event text hidden layer vector; A fused word vector sequence acquisition module is used to express the event type of the hierarchical label event in vector form, obtain an event type feature set, obtain a global attention representation of the hierarchical label event, fuse the global attention representation with the hierarchical label representation of the known event text, obtain a hierarchical label fused global attention representation, fuse the hierarchical label event text hidden vector with the hierarchical label fused global attention representation using a gating mechanism, and integrate the fused result with the candidate event type feature using a gating mechanism to obtain a fused word vector sequence of the hierarchical label event text, wherein the candidate event type feature belongs to the event type feature set; Grid labeling network construction module: used to construct the loss function of the grid labeling network. The loss function involves the span relationship score and role relationship score of the characters of the hierarchical label event text for the candidate event type features. The span relationship score and role relationship score are obtained based on the scoring scores obtained by fusion of the two vectors of the word vector sequence and their respective position encodings. The grid labeling network is trained to minimize the loss function to obtain the extraction model.