Method for extracting chapter-level events based on information fusion graph network

Abstract

The invention provides a chapter-level event extraction method based on an information fusion graph network, which comprises the following steps of: 1, fusing Chinese vocabulary characteristics and decoding by using a mask conditional random field to complete event element entity identification of a chapter-level text; and 2, performing text-level text coding on text sentences and event element entities by using an information fusion graph network. According to the method, Chinese vocabularies and position feature information of the Chinese vocabularies are introduced, and the problem of illegal label transfer of BIO sequence labeling is solved by using a mask conditional random field, so that better event element entity recognition performance is realized; according to the method, chapter-level text coding is completed by using the information fusion graph network, event detection is performed by using event type nodes after information fusion, and the interference problem of irrelevant sentences or other event type related sentences is eliminated, so that the chapter-level event extraction performance is improved.

Description

Technical Field

[0001] The present invention relates to an event extraction method, in particular to a chapter-level event extraction method based on an information fusion graph network, and belongs to the technical field of natural language processing. Background Art

[0002] In the era of the Internet, big data technology and artificial intelligence technology, as the core technologies of the current Internet development, are constantly making milestone progress. The degree of informatization of social life is constantly deepening, and people's way of obtaining information has gradually changed from paper texts such as books to network information carriers such as web pages. This change enables people to obtain valuable information more conveniently, but it also makes people exposed to a large amount of irrelevant information every day, making it impossible to obtain key information from it efficiently. Especially in the financial field, massive amounts of enterprise-related information, difficult-to-understand financial professional terms, and long corporate announcements have caused many troubles for people to retrieve important information.

[0003] In order to better solve the above problems, information extraction technology has been proposed and widely studied. The main purpose of information extraction technology is to use natural language processing technology to complete the encoding and semantic analysis of text, and then automatically extract the structured key information that users are concerned about in the text, and store the structured information in a fixed form. On the one hand, by providing direct and effective information, it is convenient for people to retrieve better; on the other hand, it can provide information and technical support for downstream tasks such as knowledge graphs and knowledge questions and answers. According to the different topics of the extracted text information content, information extraction technology can be divided into multiple subtasks, mainly named entity recognition NER for extracting entity nouns, relation extraction RE for extracting relations between entities, and event extraction EE for extracting event information. In the financial field, listed companies need to disclose information regularly in accordance with relevant regulations. Regularly disclosed corporate information helps investors fully understand corporate information, so as to better assess its risk situation and profitability, and assist investors in making decisions. The corporate announcements disclosed by listed companies mainly contain a large amount of economic activity information, such as share changes, equity pledges, financial profits and losses, and a series of corporate-related event information. It can be seen that events are important information transmission carriers in the financial field and are key information that investors need to obtain.

[0004] At present, the research focus of event extraction technology is still on sentence-level event extraction, that is, extracting structured event information from a given sentence-level text, where various neural network models can be used to encode and obtain sentence information. However, compared with sentence-level text, corporate announcement texts disclosed by listed companies in the actual financial field often appear in the form of long texts, and the event information contained in the announcement is often scattered in different sentences in the document and there may be multiple events without obvious boundaries at the same time. Faced with these problems in practical applications, sentence-level event extraction technology cannot well implement event extraction for corporate announcements because it cannot obtain the associated information of the entire document. Therefore, in order to solve the problem that the current sentence-level event extraction cannot model the complex semantics of the document level and the structural association of the event elements therein, the chapter-level event extraction technology has been proposed and has received more and more attention.

[0005] More and more chapter-level event extraction algorithms have been proposed to solve the problems of long texts, scattered event elements, and multiple events. These studies generally divide the chapter-level event extraction task into multiple subtasks, namely event element entity recognition, event detection, and argument extraction. The recognition accuracy of each task will affect the final event extraction effect. However, there are still many deficiencies and problems in the research on chapter-level event extraction for corporate announcements.

[0006] First, the event element entity recognition in the current chapter-level event extraction model has the following main problems. First, the existing event extraction models all use Chinese entity recognition models based on word vectors, without considering Chinese vocabulary information, which plays an important role in identifying the correct event element entities. Second, defining entity recognition as a BIO sequence labeling task and using conditional random fields to output the final entity label has the problem of illegal label conversion;

[0007] Secondly, after completing the event element entity recognition model, it is necessary to perform chapter-level text encoding, event detection, and role classification. However, the current event extraction model still has the following shortcomings. Existing models all use a detection method with unknown event types, that is, event detection is performed on document sentences. This method cannot eliminate the interference of irrelevant sentences or sentences related to other event types;

[0008] To this end, a chapter-level event extraction method based on information fusion graph network is proposed. Summary of the invention

[0009] In view of this, the present invention provides a chapter-level event extraction method based on an information fusion graph network to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial option.

[0010] The technical solution of the embodiment of the present invention is implemented as follows: a chapter-level event extraction method based on an information fusion graph network comprises the following steps:

[0011] Step 1: Integrate Chinese vocabulary features and use masked conditional random fields for decoding to complete event element entity recognition of chapter-level text;

[0012] Step 2: Use the information fusion graph network to perform text encoding on the text sentences and event element entities;

[0013] Step 3: Use the encoded event type node to perform event detection and obtain the time type;

[0014] Step 4: Classify the event role of the event element entity according to the time type.

[0015] Further preferably, in step one, vocabulary information is integrated to construct a vocabulary enhancement model, wherein the vocabulary information includes feature information of the vocabulary itself and position feature information corresponding to the vocabulary, and the vocabulary enhancement model is improved based on the BERT model, and a vocabulary information injection adapter is constructed between the first layer Transfomer and the second layer Transfomer of the BERT model, and the BERT model includes twelve layers of Transfomer.

[0016] Further preferably, in step one, a masked conditional random field is used for label prediction.

[0017] Further preferably, in step 2, an information fusion graph network is used to complete chapter-level text encoding, wherein the graph network adopts a graph convolutional network, and the graph convolutional network mainly includes entity nodes, sentence nodes, event type nodes and role nodes.

[0018] Further preferably, in step three, event detection is performed using the event type nodes after the information fusion graph network fuses the long text information, and event detection is defined as a binary classification task for each event type embedding representation.

[0019] Further preferably, in step 4, the event role classification mainly adopts a directed acyclic graph solution, and the roles included in the defined event type are classified into two categories according to the roles played by the entities.

[0020] Further preferably, in step one, the vocabulary enhancement model uses an Adam optimizer to perform parameter optimization, and the Adam optimizer has the characteristic of an adaptive learning rate.

[0021] Further preferably, the initial learning rate of the vocabulary enhancement model is set to 0.00001, the batch size during training is set to 8, and the maximum number of iterations is 100.

[0022] Further preferably, in step 2, the number of layers of the graph network is 3, and the final node embedding representation of the paragraph text perception is obtained by transforming the embedding representations of all layers.

[0023] Further preferably, in step 3 and step 4, a 4-layer Transformer is used for event detection, the hidden layer word vector dimension of the Transformer is set to 768, the dimension size of the feedforward layer is set to 1024, and the learning rate is set to 5e-5;

[0024] A joint training method is used, and the sum of the loss values ​​of entity recognition, event detection, and role classification in the vocabulary enhancement model is used as the total loss value for training, and the weights are set to 0.05, 0.95, and 0.95 respectively;

[0025] Using the planned sampling method, from the 10th round to the 20th round, the proportion of predicted entity results is linearly increased from 0% to 100%, and the model with the best effect on the validation set is selected to output the test set.

[0026] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:

[0027] First, the present invention introduces Chinese vocabulary and its position feature information, and uses masked conditional random fields to solve the problem of illegal label transfer in BIO sequence annotation, thereby achieving better event element entity recognition performance;

[0028] Second, the present invention uses an information fusion graph network to complete chapter-level text encoding, and uses the event type nodes after information fusion to perform event detection, eliminating the interference problem of irrelevant sentences or sentences related to other event types, thereby improving the performance of chapter-level event extraction;

[0029] 3. Compared with the prior art, the present invention has better effects and can extract more comprehensive and accurate limited domain event information compared with other chapter-level event extraction algorithm models.

[0030] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 It is a flowchart of the steps of the chapter-level event extraction method based on the information fusion graph network of the present invention;

[0033] Figure 2 is a flowchart of the vocabulary enhancement model processing of the present invention;

[0034] Figure 3 Schematic diagram of the mask transfer matrix of the mask conditional random field MCRF used in the present invention;

[0035] Figure 4 A schematic diagram of decoding the directed acyclic graph event record used in the role classification task in the present invention. DETAILED DESCRIPTION

[0036] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0037] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0038] like Figure 1-Figure 4 As shown, the embodiment of the present invention provides a chapter-level event extraction method based on an information fusion graph network, comprising the following steps:

[0039] Step 1: Integrate Chinese vocabulary features and use masked conditional random fields for decoding to complete event element entity recognition of chapter-level text;

[0040] First, construct a character-vocabulary-position matching pair sequence, and then use it as the input of the subsequent vocabulary enhancement model. The vocabulary enhancement model processing flow is as shown in the attached figure. Figure 2 As shown;

[0041] First, event element entity recognition is performed, which mainly includes a vocabulary enhancement encoder that integrates vocabulary information and a decoding layer based on masked conditional random fields. The vocabulary enhancement model is constructed by building a Chinese word-level information fusion layer between the first and second layers of the Transformer in the Chinese BERT model. The input c iRepresents the i-th Chinese character in the input corporate announcement paragraph text;

[0042] Transformer is a neural network structure, and the BERT model is a deep learning model;

[0043] The input of the Chinese word-level information fusion layer mainly consists of three parts. The first part is the character vector: h i is a feature vector of a Chinese character, that is, the output of a certain layer of Transformer in the Chinese BERT model, and the second part is the vocabulary vector: is a collection of vocabulary vectors, where the words are related to the Chinese character c i Matching, the vocabulary vectors in this set are obtained by querying the pre-trained word vector dictionary, the third part is the position vector: is the same as the Chinese character c i The set of position vectors of the matching words;

[0044] Through the above operations, we get three inputs of the vocabulary information injection adapter. Since the dimensions of the pre-trained word vector and the Chinese character feature vector are different, we need to perform nonlinear transformation first. Each Chinese character matches multiple words, and the importance of different words to the character is different. Therefore, the correlation between each character and the matching word is calculated according to the following formula to obtain the weight vector of each word:

[0045]

[0046] Where W attn represents the bilinear attention weight matrix;

[0047] Then the lexical features and the lexical-related positional features are fused as follows:

[0048]

[0049] Therefore, the weighted sum of all vocabulary information including vocabulary features and position features can be obtained, and the calculation formula is as follows:

[0050]

[0051] Finally, the weighted vocabulary features and position features are injected into the Chinese character vector. The calculation formula is as follows:

[0052]

[0053] After completing the text encoding of the fused vocabulary information, it is input into the decoding layer based on the masked conditional random field to predict the entity label, thereby completing the event element entity recognition task;

[0054] The masked conditional random field solves the problem of illegal label transfer by masking part of the label transfer path, as shown in the attached figure. Figure 3 shown.

[0055] Step 2: Use the information fusion graph network to perform text encoding on the text sentences and event element entities;

[0056] The information fusion graph network is used for chapter-level text encoding, which mainly includes the traditional three-layer graph convolutional network, in which entity nodes, sentence nodes, event type nodes and role nodes are mainly constructed. In order to complete the fusion of text information and thus help the subsequent event detection task based on event type nodes, seven types of edges are mainly constructed in the graph network, namely, edges between sentences, edges between sentences and entities, edges between entities within sentences, edges between entities between sentences, edges between roles and entities, edges between roles and event types, and edges between sentences and event types;

[0057] The node information propagation algorithm formula is as follows:

[0058]

[0059] Among them, u represents any node in the graph network, K represents the set of different types of edges connected to the sentence node, is the learnable parameter matrix corresponding to edge type k, N k (u) represents the set of adjacent nodes connected to the node with edge type k, c u,k is the normalization constant;

[0060] Finally, the final embedding representation of the node is obtained according to the following formula:

[0061]

[0062] in, represents the initial embedding representation of node u, W a is a learnable parameter matrix, and L represents the number of layers of the graph convolutional network.

[0063] Step 3: Use the encoded event type node to perform event detection and obtain the time type;

[0064] In order to mitigate the interference of event-irrelevant sentences or sentences related to different types of events, event detection is defined as embedding representation for each event type. The binary classification task is to calculate the loss function L of event detection according to the following formula: ed :

[0065]

[0066] Step 4: Classify the event role of event element entities according to the time type;

[0067] Use as attached Figure 4 The method for generating EDAG shown performs event record decoding and event role classification, which converts event extraction into a form that fills the event table;

[0068] The event record decoding model first defines the filling order of the corresponding event roles for a specific event type, and the definition method is based on the non-empty ratio of event elements; this definition method helps to provide more information in the role classification process to better fill the event table;

[0069] Then, according to the defined filling order, the event record is successfully transformed into a linked list of event elements; each event element node can be an entity mention or an empty node, that is, there is no entity mention that plays the role of the current event;

[0070] In role classification, the process of determining whether an entity plays a specific role is called path extension, which means predicting whether each entity plays the current event role in turn. If it plays the role, it is determined that path extension is needed;

[0071] Finally, by sharing the same prefix path, these linked lists are merged to obtain the EDAG under the current event type, that is, an event record is composed of a complete path in EDAG, and EDAG is a directed acyclic graph;

[0072] Fusion of entity embeddings using Transformer encoding Context Current event path U i and global memory G, extracting the entity embedding matrix from the encoded output The calculation formula is as follows:

[0073]

[0074] Based on the EDAG event record decoding scheme, the path extension task is defined as Multiple binary classification problems, i.e. predicting whether the i-th entity is the next argument role in the current record and expanding the path accordingly;

[0075] The joint training form is adopted, so the final loss function of the event extraction model in this chapter is as follows:

[0076] L total =λ1L er +λ2L ed +λ3L record

[0077] Among them, λ1, λ2, and λ3 are all hyperparameters, and L er is the loss value of the event element entity recognition model, L ed is the loss value of the event detection model, L record is the loss value of the event record decoding model.

[0078] In one embodiment, in step one, vocabulary information is integrated to build a vocabulary enhancement model, where the vocabulary information includes feature information of the vocabulary itself and position feature information corresponding to the vocabulary. The vocabulary enhancement model is improved based on the BERT model, and a vocabulary information injection adapter is built between the first layer Transfomer and the second layer Transfomer of the BERT model. The BERT model includes twelve layers of Transfomer.

[0079] In one embodiment, in step 2, an information fusion graph network is used to complete chapter-level text encoding, wherein the graph network adopts a graph convolutional network, which mainly includes entity nodes, sentence nodes, event type nodes, and role nodes.

[0080] In one embodiment, in step three, event detection is performed on event type nodes after long text information is fused using an information fusion graph network, and event detection is defined as a binary classification task for each event type embedding representation.

[0081] In one embodiment, in step 4, event role classification mainly adopts a directed acyclic graph solution, and the roles included in the defined event type are classified into two categories according to the roles played by the entities.

[0082] In one embodiment, the vocabulary enhancement model uses the Adam optimizer for parameter optimization. The Adam optimizer has the characteristic of adaptive learning rate. The Adam optimizer adjusts the learning step size in a clever way so that the frequently changing parameters are updated with a smaller step size, while the update step size of the sparse parameters is relatively large.

[0083] The initial learning rate of the vocabulary enhancement model is set to 0.00001, the batch size during training is set to 8, and the maximum number of iterations is 100. In step 2, the number of layers of the graph network is 3, and the final node embedding representation of the paragraph text perception is obtained by transforming the embedding representation of all layers;

[0084] A 4-layer Transformer is used for event detection. The hidden word vector dimension of Transformer is set to 768, the dimension size of the feedforward layer is set to 1024, and the learning rate is set to 5e-5.

[0085] A joint training method is used, and the sum of the loss values ​​of entity recognition, event detection, and role classification in the vocabulary enhancement model is used as the total loss value for training, and the weights are set to 0.05, 0.95, and 0.95 respectively;

[0086] Using the planned sampling method, from the 10th round to the 20th round, the proportion of predicted entity results is linearly increased from 0% to 100%, and the model with the best effect on the validation set is selected to output the test set.

[0087] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A chapter-level event extraction method based on information fusion graph network, characterized in that: The following steps are involved: Step 1: Integrate Chinese vocabulary features and use masked conditional random fields for decoding to complete event element entity recognition of chapter-level text; Step 2: Use the information fusion graph network to perform text encoding on the text sentences and event element entities; Step 3: Use the encoded event type node to perform event detection and obtain the time type; Step 4: Classify the event role of the event element entity according to the time type.

2. According to claim 1, a chapter-level event extraction method based on information fusion graph network is characterized in that: In step one, a vocabulary enhancement model is constructed by integrating vocabulary information, wherein the vocabulary information includes feature information of the vocabulary itself and position feature information corresponding to the vocabulary. The vocabulary enhancement model is improved based on the BERT model, and a vocabulary information injection adapter is constructed between the first layer Transfomer and the second layer Transfomer of the BERT model. The BERT model includes twelve layers of Transfomer.

3. The chapter-level event extraction method based on information fusion graph network according to claim 2 is characterized by: In step 1, label prediction is performed using masked conditional random fields.

4. The chapter-level event extraction method based on information fusion graph network according to claim 3 is characterized by: In step 2, an information fusion graph network is used to complete chapter-level text encoding, wherein the graph network adopts a graph convolutional network, and the graph convolutional network mainly includes entity nodes, sentence nodes, event type nodes and role nodes.

5. The chapter-level event extraction method based on information fusion graph network according to claim 4 is characterized by: In step three, event detection is performed on the event type nodes after the information fusion graph network fuses the long text information, and event detection is defined as a binary classification task for each event type embedding representation.

6. The method for extracting chapter-level events based on information fusion graph network according to claim 5 is characterized in that: In step 4, the event role classification mainly adopts a directed acyclic graph solution to classify the roles played by the entities into two categories, based on the roles included in the defined event types.

7. The method for extracting chapter-level events based on information fusion graph network according to claim 6 is characterized by: In step one, the vocabulary enhancement model uses an Adam optimizer to perform parameter optimization, and the Adam optimizer has the characteristic of adaptive learning rate.

8. The method for extracting chapter-level events based on information fusion graph network according to claim 7 is characterized in that: The initial learning rate of the vocabulary enhancement model is set to 0.00001, the batch size during training is set to 8, and the maximum number of iterations is 100.

9. The method for extracting chapter-level events based on information fusion graph network according to claim 8 is characterized in that: In step 2, the graph network has three layers, and the final node embedding representation of the chapter text is obtained by transforming the embedding representations of all layers.

10. The chapter-level event extraction method based on information fusion graph network according to claim 9 is characterized in that: In steps 3 and 4, a 4-layer Transformer is used for event detection. The hidden word vector dimension of Transformer is set to 768, the dimension size of the feedforward layer is set to 1024, and the learning rate is set to 5e-5. A joint training method is used, and the sum of the loss values ​​of entity recognition, event detection, and role classification in the vocabulary enhancement model is used as the total loss value for training, and the weights are set to 0.05, 0.95, and 0.95 respectively; Using the planned sampling method, from the 10th round to the 20th round, the proportion of predicted entity results is linearly increased from 0% to 100%, and the model with the best effect on the validation set is selected to output the test set.

Images