Pre-training question and answer event argument extraction method based on multi-pool attention mechanism

Abstract

The invention relates to a pre-training question and answer event extraction method based on a multi-pool attention mechanism. The method comprises the following steps: obtaining event mention description, obtaining event type classification, and retrieving all event roles corresponding to event types; a pre-training question and answer text is constructed, and a pre-training model is used for encoding the pre-training question and answer text; using a multi-pool attention mechanism to obtain a feature vector; performing linear weighting on a coding result by using a feature vector, and extracting an argument corresponding to an event role; training the model by using the annotation data; and performing inference by using an unlabeled sample to obtain a role argument corresponding to the event mention. The invention provides an effective natural language processing event extraction method, the extraction capability when the same entity type serves as the subject and object of the event is improved, the event extraction result is accurately obtained, and the method is suitable for the knowledge graph construction condition with less annotation data.

Description

Technical Field

[0001] The present invention belongs to the field of air quality detection and control, and specifically provides a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism, which is used to solve the structured representation of air pollutant control policies. Background Art

[0002] Industrial production areas in cities are one of the main sources of emissions of volatile organic compounds (VOCs), the main components of ozone. The country has formulated guidance materials, however, these guidance materials involve five major industries and more than 20 sub-production processes, involving a variety of VOC emission standards. Due to the mutual reactions between VOCs in the natural environment, simply reducing the emissions of a certain VOC cannot achieve the effect of reducing air pollution, which makes it impossible for relevant workers to achieve their goals through simple prevention and control measures. Therefore, through the method of structured governance solutions, it is possible to generate correlations between the governance solutions for VOCs corresponding to various types of equipment in industrial production, and then achieve better decision-making results by building the decision-making capabilities of knowledge graphs.

[0003] Event extraction is a solution for structured text processing, and event argument extraction is an important subtask within the event extraction task. Its goal is to obtain argument representations in event mentions based on the event roles provided in the event schema, given a known natural language description of the event mention and its event type. For industrial production, it is difficult to obtain a model with limited training resources and use it with lightweight resources or even without training graphics card resources. Furthermore, when the amount of training data is reduced, machine question answering models perform poorly when the event contains multiple arguments for the same role or when different roles form a subject-object relationship. This is particularly noticeable when the amount of training data is reduced. Summary of the Invention

[0004] To achieve better performance for pre-trained models without significantly increasing model parameters, this paper proposes a method for extracting event arguments from pre-trained question-answering (Q&A) based on a multi-pool attention mechanism. This method inputs information such as event mentions, event types, and event roles described in Chinese natural language into the model. After encoding the pre-trained model and applying the multi-pool attention mechanism, the model extracts the corresponding event roles by calculating argument probabilities using parameter weights.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] A method for extracting event arguments from a pre-trained question-answering system based on a multi-pool attention mechanism, comprising the following steps:

[0007] 1) Obtain event descriptions from the atmospheric pollutant control database, obtain event types, and retrieve all event roles corresponding to the event types;

[0008] 2) Construct pre-trained question-answer text using the acquired data and encode it using the pre-trained model;

[0009] 3) Based on the encoding results, a multi-pool attention mechanism is used to obtain the feature vector;

[0010] 4) Use the feature vector to linearly weight the encoding results and extract the arguments corresponding to the event roles;

[0011] 5) Use the labeled data in the database to train the multi-pool attention mechanism;

[0012] 6) Based on the trained multi-pool attention mechanism model, unlabeled samples are used to infer the corresponding role arguments of event mentions.

[0013] The event mention description refers to a sentence about a certain type of event described in Chinese natural language; the event type refers to an event category described in Chinese natural language; and the event role refers to information related to the event that may appear in the event.

[0014] The pre-trained question-answer text is to emphasize and mark the event trigger words in the event mention to construct an enhanced event mention text, and the question-answer text is obtained by combining the event type and event role, namely:

[0015]

[0016]

[0017] C={[CLS],Q,[SEP],X,[SEP],[PAD],···,[PAD]}

[0018] in, Represents the original event mention text x s The i-th word after word segmentation, i=1,2,···,n, n is the length of the original event mention text, represents the event trigger word text, j = 1, 2, ···, m, where m is the length of the event trigger word text, X represents the enhanced event mention text, x ts Indicates the event type, is the jth word after the event type word segmentation, r represents obtaining event argument role information through event pattern, r iis the i-th word after word segmentation, Q represents the question template, C represents the pre-trained question and answer template, [CLS] represents the start marker, [SEP] represents the separation marker, which is used to separate the question template and the enhanced question and answer text and the end of the text, and [PAD] represents the padding marker, which is used to keep the length of all question and answer texts consistent.

[0019] The encoding is performed using a pre-trained model, specifically:

[0020]

[0021] Among them, H represents the pre-trained model encoding machine question answering result, that is, the encoding result, represents its corresponding length, |C| represents the length of the input C, d represents the dimension of the hidden layer representation, and BERT(.) represents the open source pre-trained model BERT.

[0022] The step 3) comprises the following steps:

[0023] 3.1) Perform maximum pooling on the encoded result to obtain a global feature vector, whose dimension is the same as the encoding hidden layer dimension:

[0024]

[0025]

[0026] Among them, H :i represents the vector with dimension length |C| in the i-th hidden layer of the pre-trained encoding, p represents the global feature vector, p i Represents the maximum value of the question-answer text in the i-th hidden layer of the pre-training encoding;

[0027] 3.2) Split the encoding results according to the location of the trigger word, and obtain the feature vectors by maximum pooling, and then weight them to obtain the context feature vector:

[0028]

[0029]

[0030]

[0031]

[0032] Among them, p h Indicates the encoding pooling before the trigger word, represents the maximum value of the hidden layer feature dimension i before the trigger word, p e Indicates the encoding pooling after the trigger word, Represents the maximum value of the hidden layer feature dimension i after the trigger word, ts Indicates the position before the trigger word [SEP], t e Indicates the position after the trigger word [SEP], H ji represents the i-th hidden layer representation of the pre-trained code under the j-th word segmentation;

[0033]

[0034] in, Is a trainable parameter, [;] represents the connection operation between vectors, represents the context feature vector;

[0035] 3.3) Use the global feature vector and the context feature vector to construct a multi-pool attention mechanism vector:

[0036]

[0037] Among them, a represents the intermediate variable, Is a trainable parameter, [;] represents the connection operation between vectors;

[0038]

[0039]

[0040] Among them, a i represents the i-th dimension feature representation of a, b represents the multi-pool attention mechanism vector, b i Represents the multi-pool attention feature representation of the i-th word in the question and answer text.

[0041] The encoding result is linearly weighted using the feature vector, specifically by multiplying the multi-pool attention mechanism vector with the corresponding position of the pre-trained encoding, that is:

[0042]

[0043]

[0044] in, represents the pre-trained encoding representation of the i-th word of the question-answer text after weighting the multi-pool attention mechanism vector, h i represents the encoding representation of the pre-trained encoding H for the i-th question and answer text segmentation, represents the pre-trained encoding weighted by the multi-pool attention mechanism vector.

[0045] The extraction of arguments corresponding to event roles is specifically as follows: after weighted calculation, the encoding results are used as the probability of the start and end of the event arguments, that is:

[0046]

[0047]

[0048] in, are trainable parameters, W a ,W e The i-th dimension of , [;] represents the vector concatenation operation, is the probability that the i-th position of the machine question answering text C predicted by the model is the beginning and end of the argument, and θ represents all the parameters in the model.

[0049] A pre-trained question-answering event argument extraction system based on a multi-pool attention mechanism, including:

[0050] The data acquisition module is used to obtain event descriptions from the atmospheric pollutant control database, obtain event types, and retrieve all event roles corresponding to the event types;

[0051] The question-answer text construction and encoding module is used to construct pre-trained question-answer text using the acquired data and encode it using the pre-trained model;

[0052] The feature vector construction module is used to obtain the feature vector based on the encoding result using the multi-pool attention mechanism;

[0053] The argument extraction module is used to linearly weight the encoding results using feature vectors to extract the arguments corresponding to the event roles;

[0054] The output module is used to train the multi-pool attention mechanism using labeled data in the database, and based on the trained multi-pool attention mechanism model, use unlabeled samples to infer the corresponding role arguments of event mentions.

[0055] A pre-trained question-answering event argument extraction device based on a multi-pool attention mechanism comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement the pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism when executing the computer program.

[0056] A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for extracting arguments from pre-trained question-answering events based on a multi-pool attention mechanism is implemented.

[0057] The present invention has the following beneficial effects and advantages:

[0058] 1. This invention introduces a multi-pool attention mechanism to perform weighted processing on pre-trained codes, thereby effectively improving the extraction performance of the pre-trained question-answering model when dealing with events where the subject and object belong to the same entity type under resource-scarce conditions, enabling the model to complete the structured extraction of ozone control texts.

[0059] 2. Compared with mainstream structured extraction models with the same performance, this model can complete model training with fewer training resources and can meet the energy consumption constraints of industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flowchart of a method for extracting event arguments from pre-trained question-answering based on a multi-pool attention mechanism.

[0061] Figure 2 Diagram of the model for computing feature vectors for question-answer text using the multi-pool attention mechanism. DETAILED DESCRIPTION

[0062] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0063] A method for extracting event arguments from a pre-trained question-answering system based on a multi-pool attention mechanism, comprising the following steps:

[0064] 1) Get the event mention description, get the event type classification, and retrieve all event roles corresponding to the event type;

[0065] 2) By constructing pre-trained question-answer text and encoding it using a pre-trained model;

[0066] 3) Use multi-pool attention mechanism to obtain feature vectors;

[0067] 4) Use the feature vector to linearly weight the encoding results and extract the arguments corresponding to the event roles;

[0068] 5) Use labeled data to train the model;

[0069] 6) Use unlabeled samples to infer the corresponding role arguments of event mentions.

[0070] The event mention description in step 1) refers to a sentence about a certain type of event described in Chinese natural language, the event type refers to the event category described in Chinese natural language, and the event role refers to information related to the event that may appear in the event, such as time, place, etc.

[0071] The pre-trained question-answer text in step 2) is to emphasize and mark the event trigger words in the event mention to construct an enhanced event mention text, and to form a question-answer text by combining it with the event type and event role; the pre-trained model refers to the open source pre-trained model BERT.

[0072]

[0073] Among them, x s Represents the original event mention text, x tr Represents the event trigger text, and X represents the enhanced event mention text.

[0074]

[0075] C={[CLS],Q,[SEP],X,[SEP],[PAD],···,[PAD]}

[0076] Among them, x ts [CLS] represents the start of the machine Q&A text, and [SEP] separates the question template from the enhanced Q&A text. [SEP] is also used to mark the end of the text, and [PAD] is used as a padding symbol to ensure that all Q&A texts have the same length.

[0077]

[0078] Among them, H represents the pre-trained model encoding machine question answering results, represents its corresponding length, |C| represents the length of input C, and d represents the dimension of hidden layer representation.

[0079] The linear weighting described in step 4) refers to multiplying the corresponding positions of multiple attention feature vectors with the pre-trained encoding.

[0080]

[0081]

[0082] Example

[0083] The present invention relates to a method for extracting arguments from pre-trained question-answering events based on a multi-pooling attention mechanism. This method obtains a global feature vector by performing maximum pooling on the pre-trained code and a context feature vector by performing maximum pooling on the context. These feature vectors are then weighted to obtain a multi-pooling attention feature vector to process the pre-trained code. The model is then trained using training data to obtain extraction results. This method effectively addresses the issue of the model's accuracy in processing event mentions where the subject and object roles are the same entity type, even when training data resources are scarce, thereby improving model accuracy.

[0084] like Figure 1 As shown, the implementation of this method includes the following steps:

[0085] Step 1: Obtain the event mention text, event type, and trigger word information. For example, the event mentions "Dongfeng Yueda Kia recalls nearly 70,000 vehicles," the event type is "Product Action - Recall," and the trigger word is "Recall."

[0086] Step 2: Add a separator [SEP] to the trigger word part of the event mention to construct an enhanced event mention.

[0087] Step 3: Obtain the event roles corresponding to the event type based on the event pattern library. For example, the event roles for the event type "Product Behavior - Recall" are "Event," "Recall Content," and "Recalling Party."

[0088] Step 4: Build a question template based on event type and event role.

[0089] Step 5: The concatenated question template and enhanced event mentions are used as inputs to the pre-trained model, and encoded using the pre-trained model to obtain the encoding result.

[0090] Step 6: Obtain the feature vector of the pre-trained encoding result through the multi-pool attention mechanism and weight the pre-trained encoding.

[0091] Step 7: Use the weighted results to calculate the probability of each entity in the event mention being an event role.

[0092] like Figure 2 As shown, the feature vector is obtained by the multi-pool attention mechanism, including the following steps:

[0093] Step 1: Perform maximum pooling on the encoding result to obtain a global feature vector, whose dimension is the same as the encoding hidden layer dimension.

[0094]

[0095]

[0096] Among them, H :i represents the vector of dimension length |C| in the i-th hidden layer of the pre-trained encoder, and p represents the global feature vector.

[0097] Step 2: Split the encoding results according to the location of the trigger word, and obtain the feature vectors by maximum pooling, and then obtain the context feature vector after weighting.

[0098]

[0099]

[0100]

[0101]

[0102] Among them, the encoding pool before the trigger word is defined as p h ,in Represents the maximum value of the hidden layer feature dimension i before the trigger word. Similarly, the encoding pool after the trigger word is defined as p e , t s Indicates the position before the trigger word [SEP], t e Indicates the [SEP] position after the trigger word.

[0103]

[0104] in Is a trainable parameter, [;] represents the connection operation between vectors, Represents the context feature vector.

[0105] Step 3: Use the global feature vector and the context feature vector to construct a multi-pool attention mechanism vector.

[0106]

[0107] in, Is a trainable parameter, [;] represents the connection operation between vectors.

[0108]

[0109]

[0110] Where b represents the multi-pool attention mechanism vector.

[0111] The probability of each entity in the event mention being an event role is calculated as

[0112]

[0113]

[0114] in, are trainable parameters, W a ,W e The i-th dimension of , [;] represents the vector concatenation operation, is the probability that the i-th token of the machine question answering text C predicted by the model starts and ends as an argument, and θ represents all parameters in the model.

[0115] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The present application is described with reference to the method flow chart according to the embodiment of the present application. It should be understood that each process in the flow chart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate the instructions for implementing the process in the flow chart. Figure 1 A device that specifies functions in a process or multiple processes.

[0117] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. A method for extracting argument from pre-trained question-answering events based on a multi-pool attention mechanism, characterized in that: The following steps are involved: 1) Obtain event descriptions from the atmospheric pollutant control database, obtain event types, and retrieve all event roles corresponding to the event types; 2) Construct pre-trained question-answer text using the acquired data and encode it using the pre-trained model; 3) Based on the encoding results, a multi-pool attention mechanism is used to obtain the feature vector; 4) Use the feature vector to linearly weight the encoding results and extract the arguments corresponding to the event roles; 5) Use the labeled data in the database to train the multi-pool attention mechanism; 6) Based on the trained multi-pool attention mechanism model, unlabeled samples are used to infer the corresponding role arguments of event mentions.

2. The method for extracting arguments from a pre-trained question-answering event based on a multi-pool attention mechanism according to claim 1, wherein: The event mention description refers to a sentence about a certain type of event described in Chinese natural language; the event type refers to an event category described in Chinese natural language; and the event role refers to information related to the event that may appear in the event.

3. The method for extracting arguments from pre-trained question-answering events based on a multi-pool attention mechanism according to claim 1, wherein: The pre-trained question-answer text is to emphasize and mark the event trigger words in the event mention to construct an enhanced event mention text, and the question-answer text is obtained by combining the event type and event role, namely: C={[CLS],Q,[SEP],X,[SEP],[PAD],···,[PAD]} in, Represents the original event mention text x s The i-th word after word segmentation, i=1,2,···,n, n is the length of the original event mention text, represents the event trigger word text, j = 1, 2, ···, m, where m is the length of the event trigger word text, X represents the enhanced event mention text, x te Indicates the event type, is the jth word after the event type word segmentation, r represents obtaining event argument role information through event pattern, r i is the i-th word after word segmentation, Q represents the question template, C represents the pre-trained question and answer template, [CLS] represents the start marker, [SEP] represents the separation marker, which is used to separate the question template and the enhanced question and answer text and the end of the text, and [PAD] represents the padding marker, which is used to keep the length of all question and answer texts consistent.

4. The method for extracting arguments from pre-trained question-answering events based on a multi-pool attention mechanism according to claim 1, wherein: The encoding is performed using a pre-trained model, specifically: Among them, H represents the pre-trained model encoding machine question answering result, that is, the encoding result, represents its corresponding length, |C| represents the length of the input C, d represents the dimension of the hidden layer representation, and BERT(.) represents the open source pre-trained model BERT.

5. The method for extracting arguments from pre-trained question-answering events based on a multi-pool attention mechanism according to claim 1, wherein: The step 3) comprises the following steps: 3.1) Perform maximum pooling on the encoded result to obtain a global feature vector, whose dimension is the same as the encoding hidden layer dimension: Among them, H :i represents the vector with dimension length |C| in the i-th hidden layer of the pre-trained encoding, p represents the global feature vector, p i Represents the maximum value of the question-answer text in the i-th hidden layer of the pre-training encoding; 3.2) Split the encoding results according to the location of the trigger word, and obtain the feature vectors by maximum pooling, and then weight them to obtain the context feature vector: Among them, p h Indicates the encoding pooling before the trigger word, represents the maximum value of the hidden layer feature dimension i before the trigger word, p e Indicates the encoding pooling after the trigger word, Represents the maximum value of the hidden layer feature dimension i after the trigger word, t s Indicates the position before the trigger word [SEP], t e Indicates the position after the trigger word [SEP], H ji represents the i-th hidden layer representation of the pre-trained code under the j-th word segmentation; in, Is a trainable parameter, [;] represents the connection operation between vectors, represents the context feature vector; 3.3) Use the global feature vector and the context feature vector to construct a multi-pool attention mechanism vector: Among them, a represents the intermediate variable, Is a trainable parameter, [;] represents the connection operation between vectors; Among them, a i represents the i-th dimension feature representation of a, b represents the multi-pool attention mechanism vector, b i Represents the multi-pool attention feature representation of the i-th word in the question and answer text.

6. The method for extracting arguments from pre-trained question-answering events based on a multi-pool attention mechanism according to claim 1, wherein: The encoding result is linearly weighted using the feature vector, specifically by multiplying the multi-pool attention mechanism vector with the corresponding position of the pre-trained encoding, that is: in, represents the pre-trained encoding representation of the i-th word of the question-answer text after weighting the multi-pool attention mechanism vector, h i represents the encoding representation of the pre-trained encoding H for the i-th question and answer text segmentation, represents the pre-trained encoding weighted by the multi-pool attention mechanism vector.

7. The method for extracting arguments from pre-trained question-answering events based on a multi-pool attention mechanism according to claim 1, wherein: The extraction of arguments corresponding to event roles is specifically as follows: after weighted calculation, the encoding results are used as the probability of the start and end of the event arguments, that is: in, are trainable parameters, W a ,W e The i-th dimension of [;] represents the concatenation operation of the vectors. is the probability that the i-th position of the machine question answering text C predicted by the model is the beginning and end of the argument, and θ represents all the parameters in the model.

8. A pre-trained question-answering event argument extraction system based on a multi-pool attention mechanism, characterized by: include: The data acquisition module is used to obtain event descriptions from the atmospheric pollutant control database, obtain event types, and retrieve all event roles corresponding to the event types; The question-answer text construction and encoding module is used to construct pre-trained question-answer text using the acquired data and encode it using the pre-trained model; The feature vector construction module is used to obtain the feature vector based on the encoding result using the multi-pool attention mechanism; The argument extraction module is used to linearly weight the encoding results using feature vectors to extract the arguments corresponding to the event roles; The output module is used to train the multi-pool attention mechanism using labeled data in the database, and based on the trained multi-pool attention mechanism model, use unlabeled samples to infer the corresponding role arguments of event mentions.

9. A pre-trained question-answering event argument extraction device based on a multi-pool attention mechanism, characterized in that: It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, it implements a pre-trained question-answering event argument extraction method based on a multi-pool attention mechanism as described in any one of claims 1 to 7.

Images