An efficient event extraction method based on controllable pattern generation
By using a controllable pattern generation method and an event pattern generation model for event extraction, the problems of complex template definition and low generation efficiency in existing technologies are solved. This achieves efficient and controllable event extraction, improving the adaptability and robustness of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2024-02-21
- Publication Date
- 2026-07-24
Smart Images
Figure CN118093777B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of artificial intelligence and natural language processing, and relates to an efficient event extraction method based on controllable pattern generation. Background Technology
[0002] Event extraction aims to present event information in text in a structured form. This task includes important subtasks such as named entity recognition and relation extraction, and has accumulated a wealth of research and achievements. It is not only the foundation for various tasks in natural language processing, such as information retrieval and knowledge graphs, but also has wide applications in industries such as government, biology, and finance, thus attracting increasing attention in recent years.
[0003] With the superior performance of pre-trained language models, most studies have adopted representation-based generative methods for event extraction. For example, event patterns are customized into questions described in natural language, transforming the event extraction task into question-answering. This approach amplifies the semantic information embedded in event types and roles and more effectively utilizes the rich knowledge contained in pre-trained language models. However, how to effectively construct question templates and improve answering efficiency for different event types remain potential challenges. Other research uses cue-based learning, encoding designed cue along with text as input and decoding the corresponding event sequences in a fill-in-the-blank process. Cue-based methods alleviate the difficulty of defining discrete question templates through automatically learned continuous templates, but they also suffer from output efficiency issues. Furthermore, a sequence-to-structure event extraction paradigm was proposed. This paradigm utilizes end-to-end learning of (text, event records) and generates all trigger words, arguments, and their role category labels as natural language words in a unified manner. This generative approach has higher data efficiency, but at the same time, the model training is more difficult. This is because there are a large number of indicator symbols in the event structure, which lack semantic information and appear frequently. They do not follow the syntactic constraints of natural language and can mislead the model learning process.
[0004] Therefore, existing technologies suffer from problems such as complex definition, low generation efficiency, poor controllability, non-standard results, and poor adaptability when it comes to templates for different event types. Summary of the Invention
[0005] To address the above technical problems, this invention employs an efficient event extraction method based on controllable pattern generation, comprising:
[0006] S1. Obtain the text sequence to be tested, extract entities from the text sequence to obtain the event entity set, trigger word set and position vector sequence;
[0007] S2. Construct multiple events based on the event entity set and the trigger word set;
[0008] S3. Input both the event and location vector sequences into the trained event pattern generation model to obtain the event pattern;
[0009] S4. Merge and filter the event patterns to obtain the event structure.
[0010] Entity extraction of the text sequence under test includes:
[0011] S11. Convert each character in the text sequence to be tested into a position vector to obtain a position vector sequence;
[0012] S12. Perform word embedding on the position vector sequence to obtain the embedding vector;
[0013] S13. Use a feature extractor to extract features from the embedded vector to obtain a feature sequence;
[0014] S14. Predict the feature sequence and obtain the prediction probability;
[0015] S15. Obtain the set of event entities and the set of trigger words based on the predicted probabilities.
[0016] Constructing events based on a set of event entities and a set of trigger words involves combining the set of event entities with each trigger word in the set of trigger words to obtain multiple events.
[0017] The event pattern generation model includes: an input module, an encoding module, a forward noise-adding module, a backward noise-denoising module, an autoregressive decoder, and an output module; the training process of the event pattern generation model includes:
[0018] S31. Obtain the training text, including the event and position vector sequences. Input the events into the input module to obtain the conditional text W. x and target text W y ;
[0019] S32, conditional text W x Target text W y The position vector sequence is input into the encoding module to obtain the encoded vector;
[0020] S33. Input the encoded vector into the forward noise addition module to obtain the noise vector;
[0021] S34. Input the noise vector into the reverse denoising module to obtain the denoised vector;
[0022] S35. Input the denoised vector into the autoregressive decoder to obtain the output text. and event patterns;
[0023] S36. Based on the denoised vector and the output text Calculate the loss function value, update the event pattern generation model based on the loss function value, and complete the model training when the loss function value is minimized.
[0024] The input module processes events by: adding an event type hint [TYPE] to the end of the event entity set, adding an event role hint [ROLE] before each entity in the event entity set, resulting in a new event entity set; and concatenating the new event entity set with the trigger word to obtain the conditional text W. x ; Obtain the event role for each entity in the event, resulting in an event role set; add the event role hint symbol [ROLE] before each event role in the event role set to obtain a new event role set; obtain the event type of the event, and concatenate the new event role set and the event type to obtain the target text W. y .
[0025] The encoding module for conditional text W x Target text W y And processing of the position vector sequence includes: processing the conditional text W x and target text W y The concatenation is performed to obtain the concatenated result; the concatenation result and the position vector sequence are processed using an embedding function to obtain the embedding vector h. e and h s h e with h s Perform average pooling to obtain the encoding vector h. xy The embedding vector h e Including conditional text W x Embedding vector h x and target text W y Embedding vector h y The encoded vector h xy Including conditional text W x The encoding vector h′ x and target text W y The encoding vector h′ y .
[0026] The forward noise module processes the encoded vector by: processing the encoded vector h step by step. xy vector h′ in y Adding noise yields the noise vector z. T .
[0027] The inverse denoising module processes the noise vector by using a prediction model and a classifier based on the noise vector z. T Calculate the denoised vector:
[0028]
[0029]
[0030] Wherein, at each time step t, z t Conditional text W x The encoding vector is replaced with the encoding vector h. xy vector h′ in x T is the last time step of the forward noise-adding module, z t Let be the denoised vector at time step t, and p be the probability distribution of the reverse denoising process. This represents the classification result of the classifier.
[0031] The autoregressive decoder processes the denoised vector in the following ways:
[0032] S351, Based on the conditional text W x Calculate the output words The conditional probability;
[0033] S352, Based on the output word Conditional probability, target text W y The encoding vector h′ yi and conditional text W x The encoding vector h′ x The output word is calculated using a dual attention mechanism.
[0034] S353, Based on the output word The constraint decoding algorithm is used to calculate the event role and event type, and the event role and event type are combined to obtain the event pattern.
[0035] Merging and filtering event patterns includes:
[0036] Set a threshold ε1. When two events have the same event type and the ratio of the number of identical event entities in the two events to the total number of event entities in the two events is greater than ε1, merge the event patterns of the two events. Set a threshold ε2, calculate the score of the event entities, and delete the event entities with scores lower than ε2 from the event patterns to obtain the event structure.
[0037] Beneficial effects:
[0038] 1. This invention utilizes a generative model to learn the definition process of event patterns based on candidate mentioned entities, thereby automatically generating event patterns and avoiding modeling all predefined event types, thus improving generation efficiency. 2. This invention employs a partial diffusion strategy, simulating the conditional language modeling of traditional autoregressive models by adding and removing noise only to the target text, enabling the target text to be generated according to certain specifications based on the conditional text. 3. This invention additionally trains an event type classifier during the denoising process to further model the semantics, ensuring the robustness of pattern generation and achieving fine-grained controllable pattern generation. 4. This invention sets two special symbols ([ROLE], [TYPE]) to prompt the input structure, which can be applied to the extraction of new event types beyond predefined event patterns, improving the model's adaptability. 5. This invention designs a decoding constraint algorithm to fine-grainedly control the pattern generation process, preventing non-standard results and ensuring that the number of generated roles corresponds to the number of entities. Attached Figure Description
[0039] Figure 1 This is a schematic diagram of an efficient event extraction method based on controllable pattern generation provided by an embodiment of the present invention;
[0040] Figure 2 A flowchart of an efficient event extraction method based on controllable pattern generation provided in an embodiment of the present invention;
[0041] Figure 3 This is a structural diagram of the event pattern generation module based on the diffusion model provided in an embodiment of the present invention;
[0042] Figure 4 This is a schematic diagram of the constraint decoding algorithm provided in an embodiment of the present invention. Detailed Implementation
[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] An embodiment of the present invention provides an event extraction method based on a continuous text diffusion model, such as... Figure 1 , Figure 2 As shown, the method includes the following four main steps:
[0045] Step 1: Obtain the text sequence to be tested, extract entities from the text sequence to obtain the event entity set and the trigger word set;
[0046] Using the BIO sequence labeling mode, each character in the text is labeled. The specific labeling process is as follows: Set the following labels: BT, IT, BE, IE, O, where BT represents the current character w. i As the beginning of the trigger word, IT represents the current character w. i For the middle or end of the trigger word, BE and IE are interpreted the same as above, and O represents the current character w. i It is neither a trigger word nor an event entity; for example, the annotation result for the input text "At noon on the 23rd, the reporter received a call from some villagers, and thousands of primary school students had entered the school." is as follows: 2 / BE 3 / IE at noon on the 23rd, / O reporter / IE received / BT / IT some / O villagers / IE's / O phone call / IE, / O thousands / O primary school / BE students / IE have / O entered / BT / IT school / BE campus / IE. / O, from which the set of trigger words can be obtained. The candidate event entity set ε = {"23rd", "reporter", "villager", "primary school student", "campus"}.
[0047] The joint identification of trigger words and event entities is carried out as follows:
[0048] 1. Convert each character in the text sequence to be tested into a position vector corresponding to the word in the vocabulary, i.e., S = (w1w2…w ... n ), where n represents the total number of characters in the sentence, w i This represents the position vector corresponding to each character. The length of this vector is the length of the vocabulary. The character vectors are initialized. In one embodiment, for the input sentence sequence S = (w1w2…w…) n The pre-trained model BERT is used for word embedding, i.e., (h1, h2, ..., h...). n =BERT(w1,w2,…,w) n ), where h i For each character w i The embedding vector.
[0049] 2. The word embedding vector is input into a feature extractor for further feature extraction. The feature extractor automatically learns the effective features in the input vector and reduces or increases the dimension of the input vector to fuse and reorganize the original features into new features for subsequent probability calculation and classification. The feature extractor can use attention mechanisms, convolutional neural networks, or recurrent neural networks; the vector (a1, a2, ..., a...) after passing through the feature extractor... n It has the same dimension as the word embedding vector.
[0050] 3. The feature sequence (a1, a2, ..., a...)n Input to the CRF layer via formula To calculate the scores of the predicted sequence labeling results; where E is the emission matrix, used to calculate the score for each word w. i Corresponding to its label y i The score, T is the transition matrix, used to calculate the label y. i-1 To tag y i The transition score between them, where CRF is a conditional random field.
[0051] 4. According to the formula: Calculate the probability of all sequence labeling results and select the one with the highest probability as the final sequence labeling result; where y′ is the true label sequence of the sentence and y is the predicted label sequence.
[0052] 5. Based on the definition of the BIO pattern, the trigger word sets can be obtained according to the final annotation results. and the set of event arguments p and q are the number of entities and the number of trigger words, respectively, and e and t are both composed of any consecutive w in S.
[0053] Step 2: Construct an event based on the event entity set and the trigger word set. The event includes the relationship between the entities and the trigger words. The specific process is as follows: Construct all entities {e1, e2, ..., e...} p} respectively with the set of trigger words Each trigger word t in i By matching these events, we can create q possible events.
[0054] The number of trigger words extracted in the previous step, q, determines the number of event types in the text. In the next generation process, only q event pattern models need to be performed. There is no need to build templates for all predefined event types, so as to improve generation efficiency.
[0055] Step 3: Input the relationship between entities and trigger words into the trained event pattern generation model to obtain event patterns. The event pattern generation model includes an input module, an encoding module, a diffusion model, and an output module. The diffusion model includes a forward denoising module and a backward denoising module. The diffusion model is used to add noise to the candidate entities mentioned above. During the inference process, the noise is denoised into the corresponding event roles and event types to obtain the complete event pattern. That is, for the set of event entities... Assign an event role r to each entity i This yields the corresponding set of event roles. And the event type Type to which this event belongs = {C i}. For example above, given the conditional text input: {Entities: "23rd", "Reporter", "Villager", "Elementary School Student", "Campus"; Trigger words: "Received", "Entered"}, the output target text is: {Role: "Time", "Contact Person", "Contact Person", "Contact Location"; Event Type: "Contact - Phone"}.
[0056] like Figure 3 As shown, the specific training process of the event pattern generation model is as follows:
[0057] 1. Considering that event roles within the same event may have interdependent relationships, this invention designs a joint generation strategy to simultaneously generate the event roles and event types of all entities in the same sentence; it sets two prompts "[ROLE]" and "[TYPE]", adding the event type prompt "[TYPE]" to the end of the input sequence to ensure that the event type includes all global information, including roles, in the generation; that is, it constructs the conditional text W. x =[[ROLE],e1,[ROLE],e2,…,[ROLE],e p [TYPE],t i ], construct the corresponding target text W y =[[ROLE],r1,[ROLE],r2,…,[ROLE],r p [TYPE],C i For modeling other event types, simply add the conditional text W. x The corresponding trigger word part is modified to add subsequent generated conditional perturbation terms.
[0058] 2. Set the conditional text W x and target text W y The discrete text is concatenated and then embedded into a continuous vector space using the embedding function EMB(w) on both the original text sequence S and the concatenated text sequence S, resulting in embedding vectors. h s The embedding function EMB(w) can directly use pre-trained language model vectors such as BERT.
[0059] 3. with h s Mean-pooling incorporates semantic information, improving model accuracy and yielding the encoded vector h. xy Encoding vector h xy Including conditional text W x The encoding result h′ x and target text W y The encoding result h′ y .in, p is the length of the conditional text, q is the length of the target text, and d is the dimension of the embedding vector; Mean-pooling is average pooling.
[0060] 4. h xy As the input z0 at time t=0, noise is continuously added to generate z. t+1 Gradually, z0 is noise-added to standard Gaussian noise. This process only applies to the target text h′ y Noise is added to a portion of the code to ensure it is suitable for conditional language modeling. Since this noise-adding process depends only on the previous step, it can be viewed as a Markov chain, i.e. The probability q(z) at time t t |z t-1 ) can be represented as Among them, variance Used to control noise levels, where I is the identity matrix, and z... t Let t be the noisy text vector at time step t.
[0061] 5. Use a transformer as the prediction model for each time step, and denoise the z-axis. T To directly restore z0, that is The calculation process can be approximated as p(z) t-1 |z t )·p(c|z t-1 ,z t ), p(z t-1 |z t This can be viewed as a distribution. Therefore, the reverse denoising module mainly learns the magnitude of the noise added at time t during the forward denoising process, i.e., the distribution parameters mean μ and variance σ, to gradually restore the true distribution z0 of the data; where, to ensure h′ x Not to be changed, z t Conditional text W x The encoded result is partially replaced with the encoded vector h. xy Conditional text W x The encoding vector h′ x , To use a pre-trained event type classifier, a transformer model is employed, with additional gradient perturbations added at each time step t during denoising. Further ensure the generation of controllable modes.
[0062] 6. Use an autoregressive decoder to obtain the event role and event type corresponding to the final candidate entity based on the final denoised vector;
[0063] The encoded vector of the target text in the final denoised vector is further used to model the word order through an autoregressive decoder. During the modeling process, the word order is modeled by analyzing the output words. The conditional probability is modeled to obtain the current output word. The overall probability calculation for the score and the decoder input / output sequence is as follows: This invention employs a dual attention mechanism, which includes an encoded vector containing the target text at each decoding time step. Hidden state The encoding vector h′ of the conditional text x The hidden state h′ x Cross attention, i.e.:
[0064]
[0065] in, For the cross-attention mechanism algorithm, Decoder represents a constrained decoder that integrates a decoding constraint algorithm to fine-grainedly control the pattern generation process. This algorithm is as follows: Figure 4 As shown, since this invention sets two prompts "[ROLE]" and "[TYPE]" to indicate and distinguish the jointly generated event roles and event types, the number of prompts in the target text should be consistent with the condition text.
[0066] Decoding constraint algorithm control mode generation includes:
[0067] When the terminator "" is generated <eos>When the number of "[ROLE]" is less than the number of event roles in the conditional text, then "[ROLE]" will be used. <eos>Replace "[ROLE]" with "[ROLE]"; if the number of "[ROLE]" is equal to the number of event roles in the conditional text and "[TYPE]" is less than 1, then replace "[ROLE]" with "[ROLE]"; <eos>Replace "" with "[TYPE]"; otherwise, pattern generation ends.
[0068] When generating the prompt "[TYPE]", if the number of "[ROLE]" is less than the number of event roles in the conditional text, then "[TYPE]" is replaced with "[ROLE]"; if "[TYPE]" is greater than 1, then pattern generation ends.
[0069] When generating the prompt "[ROLE]", if the number of "[ROLE]" exceeds the number of event roles in the conditional text and "[TYPE]" is less than 1, then the last prompt "[ROLE]" is replaced with "[TYPE]". This algorithm ensures that the number of generated roles corresponds to the number of entities.
[0070] 7. Calculate the loss function value based on the event role and event type. The loss function consists of two weighted parts, namely... λ is a hyperparameter used to adjust the relative importance of the two parts of the loss; The training loss of the diffusion model is... This equation is calculated in vector space over the entire denoising time step T. The target text vector z0 and the final denoised vector f θ (z t The sum of the distances between f and t) θ () represents the denoising process of the diffusion model; The training loss of the Decoder model, i.e. Output text With conditional text W x The log-likelihood between the two values is used as the loss; when the loss function value When the minimum value is reached, model training is complete.
[0071] Once the model training is complete, given the text embedding EMB(W) of the test text, x The embedding of the text sequence to be tested, EMB(S), is obtained by sampling h from a standard Gaussian distribution. y The two are concatenated and then embedded into the text sequence to be tested for average pooling, i.e. At each time step t, the transformer model can be used to denoise z. T This process continues until a denoised vector is obtained. The denoised vector is then input into an autoregressive decoder to obtain the event pattern.
[0072] Step 4: Merge and filter the event patterns to obtain a complete, high-quality event structure;
[0073] The specific steps are as follows:
[0074] 1. Compare events of the same type in pairs.
[0075] 2. Set a threshold ε1. When two events are of the same type and the ratio of the number of identical event entities contained in each event to the total number of events is greater than ε1, the events are merged. Specifically, when the entity offsets of two event entities are exactly the same and the event roles are the same, the event entities are considered to be the same.
[0076] 3. Set a threshold ε2, and calculate the score of the event entity G(e) = α*Escore(e,y) + (1-α)Gscore(e,r), where, α is a parameter, k is the total number of characters contained in entity e, n is the total number of characters in the text, and P(y) i |e i P(r) represents the probability output when classifying the label of the i-th character of entity e, t represents the total number of characters contained in the event role r corresponding to entity e, and P(r) represents the probability output when classifying the label of the i-th character of entity e. i |r 1:i-1 ) represents the probability output when generating the i-th character of the entity role; if the score is lower than ε2, the event entity is deleted.
[0077] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / eos> < / eos> < / eos>
Claims
1. A method for efficient event extraction based on controllable pattern generation, characterized in that, include: S1. Obtain the text sequence to be tested, extract entities from the text sequence to obtain the event entity set, trigger word set and position vector sequence S; S2. Construct multiple events based on the event entity set and the trigger word set; S3. Input both the event and location vector sequence S into the trained event pattern generation model to obtain the event pattern; The event pattern generation model includes: an input module, an encoding module, a forward noise-adding module, a backward noise-denoising module, an autoregressive decoder, and an output module; the training process of the event pattern generation model includes: S31. Obtain the training text, including the event and position vector sequences. Input the events into the input module to obtain the conditional text. and target text ; S32, Conditional Text Target text The position vector sequence is input into the encoding module to obtain the encoded vector; S33. Input the encoded vector into the forward noise addition module to obtain the noise vector; S34. Input the noise vector into the reverse denoising module to obtain the denoised vector; S35. Input the denoised vector into the autoregressive decoder to obtain the output text. and event patterns; S36. Based on the denoised vector and the output text Calculate the loss function value, update the event pattern generation model based on the loss function value, and complete the model training when the loss function value is minimized; S4. Merge and filter the event patterns to obtain the event structure.
2. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, Entity extraction of the text sequence under test includes: S11. Convert each character in the text sequence to be tested into a position vector to obtain a position vector sequence; S12. Perform word embedding on the position vector sequence to obtain the embedding vector; S13. Use a feature extractor to extract features from the embedded vector to obtain a feature sequence; S14. Predict the feature sequence and obtain the prediction probability; S15. Obtain the set of event entities and the set of trigger words based on the predicted probabilities.
3. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, Constructing events based on a set of event entities and a set of trigger words involves combining the set of event entities with each trigger word in the set of trigger words to obtain multiple events.
4. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, The input module handles events in the following ways: Add the event type hint [TYPE] to the end of the event entity set, and add the event role hint [ROLE] before each entity in the event entity set to obtain a new event entity set; concatenate the new event entity set with the trigger word to obtain the conditional text. ; Obtain the event role for each entity in the event, resulting in an event role set; add the event role hint symbol [ROLE] before each event role in the event role set to obtain a new event role set; obtain the event type of the event, and concatenate the new event role set and the event type to obtain the target text. .
5. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, The encoding module for conditional text Target text And the processing of position vector sequences includes: condition text and target text The concatenation is performed to obtain the concatenated result; the concatenation result and the position vector sequence are processed using an embedding function to obtain the embedding vector. and ;Will and Perform average pooling to obtain the encoding vector. The embedding vector Including conditional text Embedded vector and target text Embedded vector The encoded vector Including conditional text Encoding vector and target text Encoding vector .
6. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, The forward noise module processes the encoded vector by: processing the encoded vector step by step. vectors in Add noise to obtain a noise vector. .
7. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, The inverse denoising module processes the noise vector by: using the prediction model and classifier based on the noise vector. Calculate the denoised vector: = In each time step ,Will Conditional text Replace the encoding vector with the encoding vector Conditional text Encoding vector , This is the last time step of the forward noise-adding module. for Denoising vector at each time step for The probability distribution of the process, This represents the classification result of the classifier.
8. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, The autoregressive decoder processes the denoised vector in the following ways: S351, Based on the conditional text Calculate the output words The conditional probability; S352, Based on the output word conditional probability, target text Encoding vector and conditional text Encoding vector The output word is calculated using a dual attention mechanism. ; S353, Based on the output word The constraint decoding algorithm is used to calculate the event role and event type, and the event role and event type are combined to obtain the event pattern.
9. The efficient event extraction method based on controllable pattern generation according to claim 1, characterized in that, Merging and filtering event patterns includes: Set threshold When two events have the same event type, and the ratio of the number of identical event entities in the two events to the total number of event entities in the two events is greater than... When merging the event patterns of two events, set a threshold. Calculate the score of the event entity and remove those with scores lower than a certain threshold from the event pattern. From the event entities, we obtain the event structure.