A pre-training question and answer event argument extraction method based on a multi-pool attention mechanism

Abstract

The application is a pre-training question and answer event extraction method based on a multi-pool attention mechanism. The method comprises the following steps: obtaining an event mention description, obtaining an event type classification, and retrieving all event roles corresponding to the event type; constructing a pre-training question and answer text and encoding it using a pre-training model; obtaining a feature vector using a multi-pool attention mechanism; linearly weighting the encoding result using the feature vector to extract argument corresponding to the event role; training the model using labeled data; and deducing the role argument corresponding to the event mention using unlabeled samples. The application provides an effective natural language processing event extraction method, improves the extraction capability when the same entity type acts as an event subject or object, accurately obtains an event extraction result, and is suitable for knowledge graph construction with less labeled data.

Description

Technical Field

[0001] This invention belongs to the field of air quality detection and control, specifically a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism, used to solve the structured representation of air pollutant control policies. Background Technology

[0002] Industrial production areas in cities are one of the main sources of volatile organic compounds (VOCs), the primary component of ozone. While national guidelines exist, these guidelines cover five major industries and over 20 sub-production processes, involving various VOC emission standards. Due to the natural interactions between VOCs, simply reducing emissions of a single VOC cannot effectively lower air pollution levels, making it impossible for relevant workers to achieve their goals through simplistic prevention and control measures. Therefore, a structured approach to governance can establish relationships between governance solutions for various equipment in industrial production, thereby enabling better decision-making through the construction of knowledge graphs.

[0003] Event extraction is a solution for structured text processing, and event argument extraction is an important subtask within event extraction. Its goal is to extract argument representations from event mentions given a natural language description of the event and its type, based on the event roles provided in the event pattern. For industrial production, obtaining a model with limited training resources and using it with lightweight resources or even no training GPU resources is quite challenging. Furthermore, machine question-answering models perform poorly when the amount of training data is reduced, especially when the event contains multiple arguments with the same role or when different roles constitute a subject-object relationship. Summary of the Invention

[0004] To improve the performance of pre-trained models without significantly increasing model parameters, this invention proposes a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism. This invention inputs information such as event mentions, event types, and event roles described in Chinese natural language into the model. After encoding by the pre-trained model and using a multi-pool attention mechanism to obtain weighted feature vectors, the corresponding event roles are extracted by calculating argument probabilities through parameter weights.

[0005] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0006] A pre-trained question-answering event argument extraction method based on multi-pool attention mechanism includes the following steps:

[0007] 1) Obtain event mention descriptions from the air pollutant control database, determine the event type, and retrieve all event roles corresponding to the event type;

[0008] 2) Construct pre-trained question-and-answer text using the acquired data, and encode it using a pre-trained model;

[0009] 3) Based on the encoding results, a multi-pool attention mechanism is used to obtain feature vectors;

[0010] 4) Use feature vectors to linearly weight the encoding results and extract the arguments corresponding to the event roles;

[0011] 5) Train a multi-pool attention mechanism using labeled data from the database;

[0012] 6) Based on the trained multi-pool attention mechanism model, the role arguments corresponding to the event mentions are inferred using unlabeled samples.

[0013] The event mention description refers to a sentence describing a certain type of event using Chinese natural language; the event type refers to the category of events described using Chinese natural language; and the event role refers to information related to the event that may appear in the event.

[0014] The pre-trained question-and-answer text is constructed by emphasizing event trigger words in event mentions to create enhanced event mention text. The question-and-answer text is obtained by combining event type and event role.

[0015]

[0016]

[0017] C={[CLS],Q,[SEP],X,[SEP],[PAD],···,[PAD]}

[0018] in, Indicates the original event mention text x s The i-th segment after word segmentation, where i = 1, 2, ..., n, and n is the length of the original event mention text. The text represents the event trigger word, j = 1, 2, ..., m, where m is the length of the event trigger word text, and X represents the enhanced event mention text. ts Indicates the event type, The j-th segment after event type segmentation, r represents the event argument role information obtained through event pattern. iFor the i-th segment after word segmentation, Q represents the question template, C represents the pre-trained question-answer template, [CLS] represents the start marker, [SEP] represents the separator marker, used to separate the question template and the enhanced question-answer text and to mark the end of the text, and [PAD] represents the padding marker, used to keep all question-answer texts of the same length.

[0019] The encoding using a pre-trained model specifically involves:

[0020]

[0021] Where H represents the pre-trained model encoding the machine question-and-answer result, i.e., the encoding result. The value represents the corresponding length, |C| represents the length of the input C, d represents the dimension of the hidden layer representation, and BERT(.) represents the open-source pre-trained model BERT.

[0022] Step 3) includes the following steps:

[0023] 3.1) Max pooling is performed on the encoding result to obtain the global feature vector, whose dimension is the same as that of the encoding hidden layer:

[0024]

[0025]

[0026] Among them, H :i Let p represent a vector of length |C| in the i-th hidden layer of the pre-trained encoding, and p represent the global feature vector. i This represents the maximum value of the question-and-answer text in the i-th hidden layer of the pre-trained encoding;

[0027] 3.2) The encoding result is split according to the position of the trigger word, and the feature vectors are obtained by max pooling each part. The weighted sum is then used to obtain the context feature vector:

[0028]

[0029]

[0030]

[0031]

[0032] Where, p h This indicates the encoding pooling before the trigger word. p represents the maximum value in the hidden layer feature dimension i before the trigger word. e This indicates the encoding pooling after the trigger word. t represents the maximum value in feature dimension i of the hidden layer after the trigger word.s Indicates the position [SEP] before the trigger word, t e Indicates the [SEP] position after the trigger word, H ji This represents the i-th hidden layer representation of the pre-trained encoding under the j-th word segmentation;

[0033]

[0034] in, It is a trainable parameter, and [;] represents the concatenation operation between vectors. Represents the context feature vector;

[0035] 3.3) Construct multi-pool attention mechanism vectors using global feature vectors and context feature vectors:

[0036]

[0037] Where 'a' represents an intermediate variable. It is a trainable parameter, and [;] represents the concatenation operation between vectors;

[0038]

[0039]

[0040] Among them, a i Let a represent the i-th feature of a, and b represent the multi-pool attention mechanism vector. i This represents the multi-pool attention feature representation of the i-th word segment of the question-and-answer text.

[0041] The linear weighting of the encoding result using feature vectors specifically involves multiplying the multi-pool attention mechanism vector by the corresponding position of the pre-trained encoding, i.e.:

[0042]

[0043]

[0044] in, h represents the pre-trained encoding representation of the i-th word segment of the question-and-answer text after weighting the multi-pool attention mechanism vectors. i Let H represent the encoding of the i-th question-and-answer text segmentation by the pre-trained encoding H. This represents the pre-trained encoding weighted by the multi-pool attention mechanism vectors.

[0045] The extraction of arguments corresponding to event roles specifically involves: weighting the encoded results and using them as the probabilities of the start and end of event arguments, i.e.:

[0046]

[0047]

[0048] in, These are trainable parameters. W a W e The i-th dimension, [;] denotes the vector concatenation operation. θ represents the probability that the i-th position of the machine question-answer text C predicted by the model is the start and end of the argument, respectively, and θ represents all parameters in the model.

[0049] A pre-trained question-answering event argument extraction system based on a multi-pool attention mechanism includes:

[0050] The data acquisition module is used to obtain event mention descriptions from the air pollutant control database, determine the event type, and retrieve all event roles corresponding to the event type;

[0051] The question-and-answer text construction and encoding module is used to construct pre-trained question-and-answer text from the acquired data and encode it using a pre-trained model.

[0052] The feature vector construction module is used to obtain feature vectors based on the encoding results using a multi-pool attention mechanism;

[0053] The argument extraction module is used to extract arguments corresponding to event roles by linearly weighting the encoded results using feature vectors.

[0054] The output module is used to train a multi-pool attention mechanism using labeled data in the database, and based on the trained multi-pool attention mechanism model, infers the corresponding role arguments for the event mentions using unlabeled samples.

[0055] A pre-trained question-answering event argument extraction device based on a multi-pool attention mechanism includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism when the computer program is executed.

[0056] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism.

[0057] The present invention has the following beneficial effects and advantages:

[0058] 1. This invention improves the extraction performance of the pre-trained question answering model under resource scarcity by introducing a multi-pool attention mechanism to weight the pre-trained encoding, enabling the model to complete the structured extraction of ozone governance text.

[0059] 2. Compared with mainstream structured extraction models of similar performance, this model can complete model training with less training resources and can meet the energy consumption constraints of industrial production. Attached Figure Description

[0060] Figure 1 This is a flowchart of a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism.

[0061] Figure 2 A model diagram for calculating the text feature vectors of a multi-pool attention mechanism. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0063] A pre-trained question-answering event argument extraction method based on multi-pool attention mechanism includes the following steps:

[0064] 1) Obtain the event mention description, get the event type classification, and retrieve all event roles corresponding to the event type;

[0065] 2) Construct pre-trained question-and-answer text and encode it using a pre-trained model;

[0066] 3) Use a multi-pool attention mechanism to obtain feature vectors;

[0067] 4) Use feature vectors to linearly weight the encoding results and extract the arguments corresponding to the event roles;

[0068] 5) Train the model using labeled data;

[0069] 6) Use unlabeled samples to infer the role arguments corresponding to the event mentions.

[0070] Step 1) The event mention description refers to a sentence describing a certain type of event using Chinese natural language. The event type refers to the category of events described using Chinese natural language. The event role refers to information related to the event that may appear in the event, such as time and place.

[0071] Step 2) The pre-trained question-and-answer text is constructed by emphasizing event trigger words in event mentions to create enhanced event mention text, which is then combined with event type and event role to form question-and-answer text; the pre-trained model refers to the open-source pre-trained model BERT.

[0072]

[0073] Where, x s Indicates the original event reference text, x tr This represents the event trigger text, and X represents the enhanced event mention text.

[0074]

[0075] C={[CLS],Q,[SEP],X,[SEP],[PAD],···,[PAD]}

[0076] Where, x ts The event type is indicated by 'r', which indicates that event argument role information is obtained through event patterns. 'Q' represents the question template, and 'C' represents the pre-trained question-and-answer template. [CLS] indicates the start of the machine question-and-answer text, and [SEP] separates the question template and the enhanced question-and-answer text. [SEP] is also used to mark the end of the text, and [PAD] serves as a padding symbol to ensure consistent length across all question-and-answer texts.

[0077]

[0078] Where H represents the machine question-and-answer result encoded by the pre-trained model. The value represents its corresponding length, |C| represents the length of the input C, and d represents the dimension of the hidden layer.

[0079] The linear weighting described in step 4) refers to multiplying the corresponding positions of multiple attention feature vectors with the pre-trained encoding.

[0080]

[0081]

[0082] Example

[0083] This invention relates to a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism. The method obtains a global feature vector by max-pooling the pre-trained encoding and a contextual feature vector by using contextual max-pooling representation. These feature vectors are then weighted to obtain multi-pool attention feature vectors that process the pre-trained encoding. The model is then trained using training data to obtain the extraction results. This method effectively addresses the issue of model accuracy when the subject and object roles in event mentions are of the same entity type, especially when training data resources are scarce, thereby improving model accuracy.

[0084] like Figure 1 As shown, the implementation of this method includes the following steps:

[0085] Step 1: Obtain the event mention text, event type, and trigger word information. For example, the event mentions "Dongfeng Yueda Kia recalls nearly 70,000 vehicles" and the event type is "Product Behavior - Recall" with the trigger word "recall".

[0086] Step 2: Add the separator [SEP] to the trigger word part of the event mention to indicate the construction of an enhanced event mention.

[0087] Step 3: Obtain the event roles corresponding to the event type from the event pattern library. For example, the event roles for the event type "Product Behavior - Recall" are "Event", "Recall Content", and "Recall Party".

[0088] Step 4: Build a problem template based on the event type and the role of the event participant.

[0089] Step 5: Combine the question template and enhanced event mentions as input to the pre-trained model, encode them through the pre-trained model, and obtain the encoding result.

[0090] Step 6: Obtain feature vectors from the pre-trained encoding results using a multi-pool attention mechanism and weight the pre-trained encoding.

[0091] Step 7: Use the weighted results to calculate the probability of each entity in the event mention being an event role.

[0092] like Figure 2 As shown, the process of obtaining feature vectors through a multi-pool attention mechanism includes the following steps:

[0093] Step 1: Perform max pooling on the encoding result to obtain the global feature vector, whose dimension is the same as that of the encoding hidden layer.

[0094]

[0095]

[0096] Among them, H :i Let p represent a vector of length |C| in the i-th hidden layer of the pre-trained encoding, and p represent the global feature vector.

[0097] Step 2: Split the encoding results according to the position of the trigger word, and obtain the feature vectors by max pooling. After weighting, obtain the context feature vector.

[0098]

[0099]

[0100]

[0101]

[0102] Wherein, the encoding pooling representation before defining the trigger word is p h ,in The maximum value in the hidden layer feature dimension i before the trigger word is represented by p. Similarly, the encoding pooling representation after the trigger word is defined as p. e , t s Indicates the position [SEP] before the trigger word, t e Indicates the [SEP] position after the trigger word.

[0103]

[0104] in It is a trainable parameter, and [;] represents the concatenation operation between vectors. This represents the context feature vector.

[0105] Step 3: Construct multi-pool attention mechanism vectors using global feature vectors and context feature vectors.

[0106]

[0107] in, It is a trainable parameter, and [;] represents the concatenation operation between vectors.

[0108]

[0109]

[0110] Where b represents the multi-pool attention mechanism vector.

[0111] Wherein, the probability of each entity in the event mention being an event role is:

[0112]

[0113]

[0114] in, These are trainable parameters. W a W e The i-th dimension, [;] denotes the vector concatenation operation. θ represents the probability that the i-th token of the machine question-answer text C predicted by the model is used as the start and end argument, and θ represents all parameters in the model.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This application is described with reference to a process flowchart according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 A device for a function specified in one or more processes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

Claims

1. A pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism, characterized in that, Includes the following steps: 1) Obtain event mention descriptions from the air pollutant control database, determine the event type, and retrieve all event roles corresponding to the event type; 2) Construct pre-trained question-and-answer text using the acquired data, and encode it using a pre-trained model; 3) Based on the encoding results, a multi-pool attention mechanism is used to obtain feature vectors; 4) Use feature vectors to linearly weight the encoding results and extract arguments corresponding to the event roles; 5) Train a multi-pool attention mechanism using labeled data from the database; 6) Based on the trained multi-pool attention mechanism model, the corresponding role arguments for event mentions are inferred using unlabeled samples; Step 3) includes the following steps: 3.1) Max pooling is performed on the encoding result to obtain the global feature vector, whose dimension is the same as that of the encoding hidden layer: in, Indicates the first pre-trained encoding The hidden layer has a dimension length of 1. The vector, Represents the global feature vector. This indicates that the question-and-answer text is in the first stage of the pre-trained encoding. The maximum value of each hidden layer; 3.2) The encoding result is split according to the position of the trigger word, and the feature vectors are obtained by max pooling each part. The weighted sum is then used to obtain the context feature vector: in, This indicates the encoding pooling before the trigger word. This indicates the hidden layer feature dimension before the trigger word. The maximum value in, This indicates the encoding pooling after the trigger word. Indicates the hidden layer feature dimension after the trigger word. The maximum value in, Indicates the preceding trigger word Location, Indicates the trigger word Location, Indicates the pre-trained encoding at the 1st The first word under the segmentation Each hidden layer represents; in, It is a trainable parameter. This represents the concatenation operation between vectors. Represents the context feature vector; 3.3) Construct multi-pool attention mechanism vectors using global feature vectors and context feature vectors: in, Indicates intermediate variables. It is a trainable parameter. This represents a concatenation operation between vectors; in, express The 3D feature representation, Represents the multi-pool attention mechanism vector. This indicates the question and answer text. Multi-pool attention feature representation of each word segmentation; The extraction of arguments corresponding to event roles specifically involves: weighting the encoded results and using them as the probabilities of the start and end of event arguments, i.e.: in, These are trainable parameters. express The dimension, This represents the vector concatenation operation. Machine question-answering text predicted by a model. The Each position represents the probability of the argument starting and ending. This represents all parameters in the model.

2. The pre-trained question-answering event argument extraction method based on multi-pool attention mechanism according to claim 1, characterized in that, The event mention description refers to a sentence describing a certain type of event using Chinese natural language; the event type refers to the category of events described using Chinese natural language; and the event role refers to information related to the event that may appear in the event.

3. The pre-trained question-answering event argument extraction method based on multi-pool attention mechanism according to claim 1, characterized in that, The pre-trained question-and-answer text is constructed by emphasizing event trigger words in event mentions to create enhanced event mention text. The question-and-answer text is obtained by combining event type and event role. in, Indicates the original event reference text After word segmentation, the first Each word, , where n is the length of the original event mention text. Indicates the text of the event trigger word. m is the length of the text that triggered the event. This indicates that the event mention text has been enhanced. Indicates the event type, The first word after event type segmentation Each word, This indicates that event argument role information is obtained through the event pattern. The first word after word segmentation Each word, This indicates a question template. This indicates a pre-trained question-and-answer template. Indicates the start marker, This indicates a separator used to separate the question template and the enhanced question-and-answer text, as well as to mark the end of the text. This indicates a padding tag used to ensure that all question and answer text has a consistent length.

4. The pre-trained question-answering event argument extraction method based on multi-pool attention mechanism according to claim 1, characterized in that, The encoding using a pre-trained model specifically involves: in, This indicates that the pre-trained model encodes the machine question-and-answer results, i.e., the encoded results. Indicates its corresponding length. Indicates input Length, This indicates the dimension of the hidden layer representation. This refers to the open-source pre-trained model BERT.

5. The pre-trained question-answering event argument extraction method based on multi-pool attention mechanism according to claim 1, characterized in that, The linear weighting of the encoding result using feature vectors specifically involves multiplying the multi-pool attention mechanism vector by the corresponding position of the pre-trained encoding, i.e.: in, This indicates that the weighted vectors of the multi-pool attention mechanism are applied to the question-answer text. The pre-trained encoding representation of each word segmentation, Indicates pre-trained encoding For the Encoded representation of a question-and-answer text segmentation, This represents the pre-trained encoding weighted by the multi-pool attention mechanism vectors.

6. A pre-trained question-answering event argument extraction system based on a multi-pool attention mechanism, used to implement the pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism as described in any one of claims 1-5, characterized in that, include: The data acquisition module is used to obtain event mention descriptions from the air pollutant control database, determine the event type, and retrieve all event roles corresponding to the event type; The question-and-answer text construction and encoding module is used to construct pre-trained question-and-answer text from the acquired data and encode it using a pre-trained model. The feature vector construction module is used to obtain feature vectors based on the encoding results using a multi-pool attention mechanism; The argument extraction module is used to extract arguments corresponding to event roles by linearly weighting the encoded results using feature vectors. The output module is used to train a multi-pool attention mechanism using labeled data in the database, and based on the trained multi-pool attention mechanism model, infers the corresponding role arguments for the event mentions using unlabeled samples.

7. A pre-trained question-answering event argument extraction device based on a multi-pool attention mechanism, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism as described in any one of claims 1-5.

Images