An event extraction method using a graph convolutional neural network and word fusion

Abstract

The application provides an event extraction method using a graph convolutional neural network and word fusion, which sequentially comprises four parts of obtaining a word feature vector, obtaining a word feature vector, word vector fusion and sequence label prediction. The application uses a graph convolutional neural network to extract a word feature vector in view of the problem that an existing event extraction method based on word fusion cannot fully utilize syntactic structure information when obtaining a word vector. However, the current graph convolutional neural network only considers whether there is a dependency relationship between nodes when extracting a word feature vector, ignores the importance of different types of dependency relationships and nodes themselves, and cannot well extract dependency relationship features. Therefore, the application improves the graph convolutional neural network, and uses values in an adjacency matrix to express the importance of different nodes and dependency relationships between the nodes, so as to extract features as part of the input of the graph convolutional neural network, and effectively improve the accuracy of event extraction.

Description

Technical Field

[0001] This invention belongs to the field of event extraction, and particularly relates to an event extraction method that utilizes graph convolutional neural networks and word fusion. Background Technology

[0002] Event extraction technology can extract structured event information from messy, unstructured information, quickly obtaining key event information. Unlike English, which has natural separators, in Chinese event extraction, "characters" and "words" are different dimensional units, expressing different semantic information. Currently, Chinese event extraction is divided into word-based event extraction, character-based event extraction, and character-word fusion-based event extraction. The information obtained by using character-based or word-based methods alone is incomplete, losing semantic information from characters or words. Character-word fusion-based methods can enhance the expression of semantic information, including richer sequence representations. Representative event extraction methods include NuggetProposal Networks (NPNS) and Trigger-Aware Lattice Neural Networks (TLNN). The NPNS method uses a dynamic multi-pooling convolutional neural network to obtain feature representations of characters and words separately, then uses a feature fusion method to fuse character and word vectors, and uses a classifier to obtain classification labels to complete the event extraction task. To enhance the understanding of polysemous characters or words, the TLNN method integrates an external language knowledge base at the embedding layer and uses a hybrid feature approach to fuse word vectors to complete the event extraction task. While current methods employ different approaches to enrich feature information representation when acquiring word vectors, none fully utilize the syntactic structure information of sentences. This leads to the easy loss of information from distant and dependent words, affecting the performance of subsequent tasks. Summary of the Invention

[0003] The technical problem that this invention aims to solve is that existing event extraction methods based on word fusion cannot fully utilize syntactic structure information when obtaining word vectors, which affects the further improvement of event extraction accuracy.

[0004] This invention solves the above-mentioned technical problems through the following technical means: an event extraction method utilizing graph convolutional neural networks and word fusion, the method comprising the following steps:

[0005] Step 1: Obtaining the character feature vector

[0006] The BERT model is used to obtain the character representation vectors in the sentence, and then the vectors are fed into the BiLSTM model to capture contextual information and obtain semantically rich character feature vectors.

[0007] Step 2: Obtaining Word Feature Vectors

[0008] To address the issue that current word-based event extraction methods do not adequately consider syntactic structure information when acquiring word vectors, this patent employs Word2Vec and graph convolutional neural networks (GCNNs) to obtain word feature vectors. GCNNs extract features from words in a sentence, incorporating syntactic structure information. This patent uses the text processing tool LTP to segment the event sentence text and generate a syntactic dependency graph. For the generated syntactic dependency graph, an adjacency matrix A is used to represent the dependencies between words. Current GCNNs only consider whether dependencies exist between nodes, neglecting the varying importance of different types of dependencies, thus leading to inconsistencies in the graph's structure. Convolutional neural networks (CNNs) cannot effectively represent the importance of dependencies. Therefore, this patent improves upon existing graph CNNs by using statistical methods to determine the importance of different dependencies and utilizing the values ​​in the adjacency matrix A to express the importance of different types of dependencies. This value is then used as part of the input to the graph CNN for feature extraction. Similarly, previous graph CNNs did not consider the importance of the nodes themselves. This patent uses the part-of-speech tag of a node in the syntactic dependency graph as an indicator of the importance of its self-looping behavior and calculates the importance of each part-of-speech tag. The dependency importance adjacency matrix is ​​obtained by adding the adjacency matrix A to the self-looping matrix. Word2Vec is used to encode the segmented words to generate word representation vectors, which are then compared with the dependency importance adjacency matrix. The inputs are fed into a graph convolutional neural network to obtain word feature vectors.

[0009] Step 3: Word Vector Fusion

[0010] The character feature vector obtained in step one and the word feature vector obtained in step two are added together and fused to obtain the feature vector after character-word fusion.

[0011] Step 4: Label Sequence Prediction

[0012] This patent treats the event extraction method as a sequence labeling task. It inputs the feature vector after word fusion into a conditional random field, constrains the label sequence, outputs the label sequence with the highest score, and completes the event extraction task based on the label.

[0013] Furthermore, as described in step one, for a given event text sentence, the word representation vector obtained through the BERT pre-trained model is X = (x1, x2, ... x...). n ), where x iLet n be the representation vector of a single character in the text sentence, and n be the number of characters in the input sentence. The obtained character vector representation is input into the BiLSTM network to capture the sequence information of the context and obtain the character feature vector containing deeper semantic information. The calculation formula is Equation (1):

[0014]

[0015] in, Let i be the feature vector of the i-th character captured from front to back. The feature vector of the i-th character captured from back to front is obtained by concatenating the two vectors together to obtain the character feature vector containing contextual semantic information.

[0016] Furthermore, as described in step two, this patent uses statistical methods to determine the importance of different dependencies in a syntactic dependency graph. Specifically, this patent defines a dependency as important if one end node of a dependency in a syntactic dependency graph contains both a trigger word and an argument. The importance of a certain type of dependency is calculated using formula (2).

[0017]

[0018] Where dr is the type of dependency, C dr I represents the number of such important dependencies in the dataset. dr To indicate the importance of this type of dependency, ∑ i∈DR C i This represents the number of important dependencies in the dataset.

[0019] Furthermore, as described in step two, this patent uses the part-of-speech tag of a node in the syntactic dependency graph as an indicator of the importance of the node's self-loop, and calculates the importance of each part-of-speech tag; specifically, assuming that the trigger words and arguments in the text have the same importance, the importance of a certain part-of-speech tag is calculated using the following formula (3):

[0020]

[0021] Where st indicates part of speech, C st I represents the number of times this part-of-speech tag is a trigger word or argument in the dataset. st To indicate the importance of this part of speech, ∑ i∈ST C j This represents the total number of trigger words and arguments in the dataset.

[0022] Furthermore, as described in step two, the adjacency matrix A is added to the self-loop matrix to obtain the dependent importance adjacency matrix. The calculation formula is shown in equation (4):

[0023]

[0024] where α and β are the coefficients of the adjacency matrix A and the self-loop matrix Diag, which are used to balance the relationship between the two importance degrees.

[0025] Further, as described in step 2, the following formula (5) is used to generate a word feature vector containing syntactic structure information and dependency relationship importance information:

[0026]

[0027] The output vector of formula (5) is H (l) , where, is the dependency importance adjacency matrix, W (l) is the parameter of the graph convolutional neural network of this layer, σ is the activation function, H (l-1) is the feature vector of the previous hidden layer, is the degree matrix, which represents the degree of each node in the graph; is to perform normalization operation on to prevent the value of the output feature vector from being too large.

[0028] Further, as described in step 3, in an event sentence, a word is composed of one or more characters, so that the total number of words in a sentence is always less than or equal to the total number of characters. If the characters and words are added or spliced in a one-to-one correspondence manner, there will be a problem of interactive error between character vectors and word vectors; therefore, the present invention determines how many character vectors each word vector is fused with through a tag, for example, the word vector "mushroom" is fused with the character vector "mu" and the character vector "gu" respectively, and the process is formalized as:

[0029]

[0030] where, represents the addition of the word vector and the character vector; is the corresponding character feature vector obtained by the characters in the sentence through BERT and BiLSTM; λ j represents the number of characters present in each word; is the corresponding word feature vector obtained by the words in the sentence through Word2Vec and the graph convolutional neural network.

[0031] Further, as described in step 4, for the vector Y={Y1, Y2, ...Y obtained after the word and character fusion obtained in the third step n}, there is a corresponding output sequence Q={y1, y2, ...y n}, Y i represents the feature vector after word and character fusion, y iLet S represent the label corresponding to each character, n represent the number of characters in the sentence, and let S denote the scoring function of the conditional random field, representing Y. i To tag y i The emission fraction, T, is the transformation function of the random field, representing the label y. i-1 To tag y i The transition score is the final score function, which is composed of S and T. The higher the score, the more accurate the predicted sequence. Finally, the Viterbi algorithm is used to decode the label sequence with the highest score. The specific loss function calculation process is shown in formulas (7), (8) and (9):

[0032]

[0033]

[0034]

[0035] In the formula above, and Let M represent the trainable parameter values, C represent all possible label sequences, Q′ represent a random sequence of labels, and in the loss expression, M represents the total number of sentences, Q′ ... i The representative model represents sentence S i Predicted sequence labels.

[0036] The beneficial effects of the present invention will be further explained below:

[0037] This invention determines the importance of different dependency relationships and integrates the importance of dependency relationships into a graph convolutional neural network to model the syntactic structure of sentences. This approach can better extract the relationships between syntactic structures and effectively alleviate the problem of losing distant related information in long text sentences, thus effectively improving the accuracy of event extraction. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 The attached figure is an example of a syntactic dependency graph generated by a word segmentation tool in an event extraction method that utilizes graph convolutional neural networks and word fusion provided by this invention.

[0040] Figure 2The attached figure is a complete event extraction flowchart of an event extraction method using graph convolutional neural networks and word fusion provided by the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] An event extraction method utilizing graph convolutional neural networks and word fusion includes the following steps:

[0043] S101. For an event text sentence, label each word in the sentence according to the BIO labeling rules to obtain a correct label sequence, which is used to compare with the prediction results of the model later.

[0044] S102. Define the maximum character length in the event text sentence as 128, and encode the characters using the one-hot method. If the length is less than 128, pad the text sentence with 0; if it exceeds 128, truncate it. Input the encoding result into the BERT model to generate a 128*768-dimensional character encoding vector, and use formula (2) to capture sequence information in the forward and backward directions in the BiLSTM model to obtain a 128*500-dimensional character encoding vector containing contextual semantic features.

[0045] S103. Define the maximum word length in the text sentence as 64. If the length is less than 64, pad the text sentence with 0. If it exceeds 64, truncate it. Use Stanford CoreNLP to segment the event text sentence, generate a syntactic analysis graph, and input the segmentation results into Word2Vec to obtain a 64*200 dimensional word encoding vector.

[0046] S104. A 64*64-dimensional dependency importance adjacency matrix generated based on the above syntactic analysis graph and the importance of each dependency relation and part of speech. And The word encoding vector is input into the convolutional neural network to generate a 64*130 dimensional word feature vector containing syntactic structure information;

[0047] S105. The word encoding vector containing the above contextual semantic features is reduced in dimensionality using a fully connected layer to obtain a 200*130 dimensional word encoding vector.

[0048] S106. Use formula (6) to fuse the above word vectors to obtain a 264*130 dimension feature vector;

[0049] S107. The fused feature vector is fed into the CRF layer, and the label is constrained according to the loss function of formula (9). Finally, the label sequence prediction result is output.

[0050] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An event extraction method utilizing graph convolutional neural networks and word fusion, characterized in that... This includes the following steps: To obtain character feature vectors, the BERT model is used to obtain character representation vectors in the sentence. Then, the character representation vectors are input into the BiLSTM model to capture contextual information and obtain semantically rich character feature vectors. To obtain word feature vectors, the text processing tool LTP was used to segment the event sentence text and generate a syntactic dependency graph. For the generated syntactic dependency graph, an adjacency matrix A was used to represent the dependencies between words in the graph. Then, Word2Vec and graph convolutional neural networks were used to obtain word feature vectors. In the process of using graph convolutional neural networks to extract features from words in the sentence, the importance of different dependencies and the importance of the nodes themselves were considered, thus extracting word feature vectors that integrate the syntactic structure information of the sentence more accurately. The dependencies between words are determined statistically to determine the importance of different dependencies, and the values ​​in the adjacency matrix are used to express the importance of different types of dependencies. These values ​​are then used as part of the input to a graph convolutional neural network to extract features. The specific method involves defining the two ends of a dependency relation in the syntactic dependency graph. If one end node contains both a trigger word and an argument, then the dependency relation is considered an important dependency relation. The importance of a certain type of dependency relation is then calculated using the following formula: in dr For the type of dependency, C dr This represents the number of such important dependencies in the dataset. I dr To indicate the importance of this type of dependency, ∑ i∈DR C i This represents the number of important dependencies in the dataset. The importance of a node is determined by using the part-of-speech tag (POS) of the node in the syntactic dependency graph as an indicator of the importance of its self-loop. The importance of each POS is calculated, and the adjacency matrix is ​​added to the self-loop matrix to obtain a dependency importance adjacency matrix. Word2Vec is used to encode the segmented words to generate word representation vectors, which are then input together with the dependency importance adjacency matrix into a graph convolutional neural network to obtain word feature vectors. Specifically, the importance of each POS is calculated, assuming that trigger words and arguments in the text have the same importance. The importance of a particular POS is calculated using the following formula: in st Indicates part of speech, C st This represents the number of words or arguments in the dataset whose part of speech is a trigger word or argument. I st To indicate the importance of this part of speech, ∑ j∈ST C j This represents the total number of trigger words and arguments in the dataset. Word-character vector fusion involves adding and fusing the obtained character feature vectors and word feature vectors to obtain the fused feature vector. Label sequence prediction treats the event extraction method as a sequence labeling task. It inputs the feature vector after word fusion into a conditional random field to constrain the label sequence and outputs the label sequence with the highest score. The event extraction task is completed based on the labels.

2. The event extraction method using graph convolutional neural networks and word fusion according to claim 1, characterized in that, The acquisition of word feature vectors specifically includes: In the process of word fusion, Word2Vec and graph convolutional neural networks are used to obtain word feature vectors that incorporate the syntactic structure information of the sentence.

3. The event extraction method using graph convolutional neural networks and word fusion according to claim 1, characterized in that, The balance between word dependencies and the importance of the words themselves includes: Adding the adjacency matrix A to the self-loop matrix yields the dependent importance adjacency matrix. Â The calculation formula is shown in the figure: in α and β Let adjacency matrix A and self-circulating matrix A be the adjacency matrix A and self-circulating matrix A. Diag A coefficient is used to balance the relationship between two levels of importance.

4. The event extraction method using graph convolutional neural networks and word fusion according to claim 1, characterized in that, Word vector fusion, specifically including: The acquired character feature vectors and word feature vectors are fused to obtain a fused feature vector, which is then used for sequence label prediction to complete the event extraction task. Specifically, a label is used to determine how many character vectors each word vector is fused with, and the process is formalized into the following formula: Where ⊕ represents the sum of word vectors; Y i b The characters in the sentence are used to obtain corresponding word feature vectors through BERT and BiLSTM. λ j This indicates the number of characters present in each word. Y j z The word feature vectors in a sentence are obtained through Word2Vec and graph convolutional neural networks.

Images