An entity relation extraction method based on multi-head attention

Abstract

This invention discloses an entity relation extraction method based on multi-head attention. The steps are as follows: 1) Obtain the contextual representation of the input sentence using Bi-LSTM; 2) Obtain the global feature vector and features of different subspaces of the sentence using a multi-head attention mechanism; 3) Generate corresponding hidden forests for the different subspaces using the matrix tree theorem; 4) Encode the global feature vector and hidden forests separately using GCN; 5) Obtain the final representations of entity and sentence vectors through pooling; 6) Fuse the global feature vector and the global feature vector calculated by convolution of the hidden forest; 7) Finally, output the results through a fully connected layer in a classifier to identify the relationship types between entities. This invention was tested on three datasets: Semeval2010task8, CPR, and PGR. The results show that this method performs excellently in entity relation extraction tasks.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to an entity relation extraction method, which is an entity relation extraction method based on multi-head attention. Background Technology

[0002] In recent years, with the continuous development of research in various fields, the amount of literature in each field has also increased significantly. Too much literature can make it difficult for researchers to easily track the information they need, thus affecting their research. Natural Language Processing (NLP) technology plays a crucial role in various fields. Among them, relation extraction plays a vital role. Because literature from different professional fields often contains many different technical terms, resulting in sparser data, NLP can capture the structural information between important entities in the text of a specific professional field.

[0003] Early work has demonstrated the importance of dependency structure for relation extraction, utilizing Graph-LSTM or graph neural networks to encode 1-best dependency trees to obtain deeper syntactic relations. In the biomedical field, recent works have used dependency forests as external features, allowing models to automatically learn optimal syntactic structures from the forest. These edges are provided by a dependency parser trained on the news domain, or by merging K-best trees by combining identical dependency edges. Recently, full dependency forests have been used to establish a connection between the parser and the relation extraction model. However, forest-based models generate significant noise during the generation of full dependency forests, leading to the loss of key information and affecting the model's extraction performance. How to fully utilize dependencies in textual information to improve the performance of entity relation extraction models and create practical application value is currently a major research problem. Summary of the Invention

[0004] The purpose of this invention is to provide an entity relation extraction method based on multi-head attention, which has superior entity relation extraction performance, strong robustness, and strong scalability, effectively improving the accuracy of information extraction.

[0005] To achieve more accurate extraction of entity relationships, the solution of this invention is:

[0006] A method for entity relation extraction based on multi-head attention, characterized by the following steps:

[0007] Step 1: Construct training corpus. Train the corpus based on GloVe English word vectors obtained from the general domain to obtain word vector representations.

[0008] Step 2: Use Bi-LSTM to encode word vector representations to obtain the contextual representation of the sentence;

[0009] Step 3: Obtain the global feature vector of the sentence and the feature representations of different subspaces through a multi-head attention mechanism;

[0010] Step 4: Generate the corresponding hidden forests for different subspaces using the matrix tree theorem;

[0011] In step 4, the method for generating a hidden forest using the matrix tree theorem is as follows:

[0012] Step A41: Calculate the edge score and root score of each subspace using multi-head attention.

[0013] Step A42: Using the edge score and root score as input, calculate the marginal probability of each edge to generate the hidden forest;

[0014] Step 5: Input the hidden forest and the global feature vectors learned from the sentences into the graph convolutional neural network (GCN) for encoding, and then obtain the vector representations of entities and sentences through pooling operations respectively.

[0015] Step 6: Fuse the sentence vector from Step 5 and the global feature vector from Step 3, and then combine them with the entity vector into a fully connected layer to obtain the final vector representation;

[0016] Step 7: Obtain the probability distribution of the final vector representation from Step 6 on the relationship labels through a classifier, thereby identifying the relationship type between entities.

[0017] The specific content of step 2 above is as follows:

[0018] We use a bidirectional long short-term memory mechanism, Bi-LSTM, to obtain the context representation, and obtain the context representation h of the sentence according to the following formula:

[0019] f(t)=σ(W f h t-1 +U f x t +bf) (1)

[0020] i(t)=σ(W i h t-1 +U i x t +b i (2)

[0021] a(t) = tanh(W) a h t-1 +U a x t +b a (3)

[0022] o(t)=σ(W o h t-1+U o x t +b o (4)

[0023] c(t)=c(t-1)⊙f(t)+i(t)⊙a(t) (5)

[0024] h(t)=o(t)⊙tanh(c(t)) (6)

[0025] Where, x t The input represents time t, h t-1 W represents the hidden state at time t-1. f W i W o and W a These represent the forget gate, input gate, output gate, and h in the feature extraction process, respectively. t-1 The weights, coefficients, and U f U i U o and U a These represent the forget gate, input gate, output gate, and x in the feature extraction process, respectively. t The weighting coefficient, b f b i b o and b a Let represent the bias values ​​in the forget gate, input gate, output gate, and feature extraction process, respectively; tanh is the tangent hyperbolic function; and σ is the sigmoid activation function. We use tanh, σ, and σ to represent the bias values ​​in the forget gate, input gate, output gate, and feature extraction process, respectively. and This represents the outputs of the forward LSTM and the backward LSTM. For the i-th word, we combine the outputs from both directions to represent the result.

[0026]

[0027] The specific content of step 3 above is as follows:

[0028] A multi-head attention mechanism is employed to capture features of the input sentence from different perspectives. We set the number of heads in the multi-head attention mechanism to be equal to the number of different subspaces, with each head corresponding to a subspace structure representing one perspective. Each subspace is represented by an adjacency matrix. The weights between relational nodes in the adjacency matrix are calculated using the following formula:

[0029]

[0030] Where Q∈R n×d , K∈R n×d , V∈R n×d These represent the query, key, and value, respectively, and d represents the dimension of the hidden layer.

[0031] The specific content of step A41 above is as follows:

[0032] a) The edge score between the i-th node and the j-th node in the k-th hidden forest is expressed by the following formula:

[0033]

[0034] Where Q∈R n×d , K∈R n×d , V∈R n×d These represent the query, key, and value, respectively. Q ∈R d×d and W K ∈R d×d is the projection parameter, and d represents the dimension of the hidden layer.

[0035] b) We calculate the root fraction It represents the normalized probability that the i-th node is selected as the root node of the k-th potential forest:

[0036]

[0037] Among them, h i It is the hidden layer vector of the i-th node. It is h i The weights of the projection.

[0038] The specific content of step A42 above is as follows:

[0039] According to the matrix tree theorem, the marginal probability of the dependency edge from the i-th word to the j-th word in the k-th forest can be expressed as:

[0040]

[0041] Among them, T k Let θ represent the k-th hidden forest induced by the matrix tree theorem, h represent the context representation vector obtained in step 2, and θ represent the hidden forest induced by the matrix tree theorem. k ∈R m The weight vector represents the dependency relationship, and m∈R represents the number of dependencies in the k-th (k∈[1,N]) hidden forest.

[0042] We can obtain the corresponding hidden forest based on the marginal probability, where the k-th hidden forest is generated from the i-th node to the j-th node. It can be represented as:

[0043]

[0044]

[0045] in, Let A represent the k-th Laplace matrix of the i-th and j-th nodes, δ represent the Kronecker coefficients, and A' ... k ∈R n×n Let represent the weighted adjacency matrix of the k-th forest.

[0046] The specific content of step 5 above is as follows:

[0047] Given an adjacency matrix The convolution computation of the i-th node in the l-th layer of the k-th potential forest is derived from the representation of the previous layer. As input and output update representation It can be defined as:

[0048]

[0049] in, and Let be the weight matrix and bias vector of the k-th hidden forest in the l-th layer, respectively, and σ be the activation function Sigmoid.

[0050] The specific content of step 6 above is as follows:

[0051] The entity vectors h1 and h2, and the sentence vector h are obtained from the hidden forest in step 5 through convolution. g The global vector is convolved to obtain the sentence vector h. a Pool the four vectors to obtain h. p1 ,h p2 α pooling ,g pooling Then, the two pooled sentence vectors are fused together, as follows:

[0052]

[0053] The function β is defined as follows:

[0054]

[0055] Finally, α and h p1 h p2 The three vectors are concatenated, and the final vector representation is:

[0056] h out =[α;h p1 h p2 (17)

[0057] The specific content of step 7 above is as follows:

[0058] The probability distribution on the relation labels is obtained by using a Softmax classifier.

[0059] This invention has the following characteristics:

[0060] (1) In the relation extraction task of this invention, the global dependency relationship learned by multi-head attention is retained to make up for the important information lost due to the noise generated by the dependency forest; the accuracy of information extraction is effectively improved; and the absence of an external parser also reduces the error of the parsed syntax information.

[0061] (2) The method proposed in this invention has been tested and the results show that it performs well in relation extraction tasks on datasets. It is also easily extensible and can be applied to various fields such as biology, chemistry, medicine, and news. Attached Figure Description

[0062] Figure 1 This is a flowchart of the present invention;

[0063] Figure 2 These are comparison images of model performance under different numbers of forests;

[0064] Figure 3 The curves showing the changes in precision, recall, and F-score of this invention and other reputation evaluation methods with increasing epochs under different datasets are shown. Detailed Implementation

[0065] The technical achievements of the present invention will be described in detail below with reference to the accompanying drawings.

[0066] like Figure 1 As shown, a multi-head attention-based entity relation extraction method is proposed, with the following approach:

[0067] Step 1, input the sentence s = w1, w2, ..., w n And the corresponding dependency graph G=(V,E), where w n V represents the nth word in the sentence, and V and E represent the set of edges that connect words and the set of word nodes, respectively.

[0068] Step 2: Use the Bi-LSTM mechanism to obtain the contextual representation h of the sentence. The process is represented by the following formula:

[0069] f(t)=σ(W f h t-1 +U f x t +bf) (1)

[0070] i(t)=σ(W i h t-1 +U i x t +b i (2)

[0071] a(t) = tanh(W) a h t-1 +U a x t +b a (3)

[0072] o(t)=σ(W o h t-1 +U o x t +b o (4)

[0073] c(t)=c(t-1)⊙f(t)+i(t)⊙a(t) (5)

[0074] h(t)=o(t)⊙tanh(c(t)) (6)

[0075] Where, x t The input represents time t, h t-1 W represents the hidden state value at time t-1. f W i W o and W a These represent the forget gate, input gate, output gate, and h in the feature extraction process, respectively. t-1 The weights, coefficients, and U f U i U o and U a These represent the forget gate, input gate, output gate, and x in the feature extraction process, respectively. t The weighting coefficient, b f b i b o and b a Let represent the bias values ​​during the forget gate, input gate, output gate, and feature extraction process, respectively. tanh is the tangent hyperbolic function, and σ is the sigmoid activation function. We use bidirectional LSTM in the model to obtain contextual representations, and we use tanh, σ, and σ to represent the bias values ​​during feature extraction, respectively. and This represents the outputs of the forward LSTM and the backward LSTM. For the i-th word, we combine the outputs from both directions to represent the result.

[0076]

[0077] Step 3: Employ a multi-head attention mechanism to capture features of the input sentence from different perspectives. We set the number of heads in the multi-head attention mechanism to be equal to the number of different subspaces, with each head corresponding to a subspace structure for one perspective. Each subspace is represented by an adjacency matrix. The weights between relational nodes in the adjacency matrix are calculated using the following formula:

[0078]

[0079] where Q∈r n ×d, K∈r n ×d,V∈r n ×d represent the query, key, and value, respectively, and d represents the dimension of the hidden layer.

[0080] Step A41, the specific content is as follows:

[0081] a) The edge score between the i-th node and the j-th node in the k-th hidden forest is expressed by the following formula:

[0082]

[0083] Where Q∈R n×d , K∈R n×d , V∈R n×d These represent the query, key, and value, respectively. Q ∈R d×d and W K ∈R d×d is the projection parameter, and d represents the dimension of the hidden layer.

[0084] b) We calculate the root fraction It represents the normalized probability that the i-th node is selected as the root node of the k-th potential forest:

[0085]

[0086] Among them, h i It is the hidden layer vector of the i-th node. It is h i The weights of the projection.

[0087] Step A42, according to the matrix tree theorem, the marginal probability of the dependency edge from the i-th word to the j-th word in the k-th forest can be expressed as:

[0088]

[0089] Among them, T k Let θ represent the k-th hidden forest induced by the matrix tree theorem, h represent the context representation vector obtained in step 2, and θ represent the hidden forest induced by the matrix tree theorem. k ∈R m The weight vector represents the dependency relationship, and m∈R represents the number of dependencies in the k-th (k∈[1,N]) hidden forest.

[0090] We can obtain the corresponding hidden forest based on the marginal probability, where the k-th hidden forest is generated from the i-th node to the j-th node. It can be represented as:

[0091]

[0092]

[0093] in, Let A represent the k-th Laplace matrix of the i-th and j-th nodes, δ represent the Kronecker coefficients, and A' ... k ∈R n×n Let represent the weighted adjacency matrix of the k-th forest.

[0094] Step 5, given the adjacency matrix The convolution computation of the i-th node in the l-th layer of the k-th potential forest is derived from the representation of the previous layer. As input and output update representation It can be defined as:

[0095]

[0096] in, and Let be the weight matrix and bias vector of the k-th hidden forest in the l-th layer, respectively, and σ be the activation function Sigmoid.

[0097] Step 6: Classify and output the entity and sentence vectors through a softmax classifier using a fully connected layer.

[0098] In step 6, the method for connecting the entities and sentence vectors from step 4 is as follows:

[0099] Step A61, the input is the entity vectors h1, h2 and the global vector h obtained through the hidden forest in step 4. g The sentence global vector h obtained through the multi-head attention mechanism in step 4 a ;

[0100] Step A62: Max pool the entity vectors h1 and h2 to obtain h p1 h p2 , will h g and h a Perform average pooling operations separately to obtain g pooling and a pooling Calculate the final representation α of the global vector:

[0101]

[0102] The function β is defined as follows:

[0103]

[0104] Step A63, compare α and h p1 h p2 The three vectors are concatenated, and the final vector representation is:

[0105] h out =[α;h p1 h p2 (17)

[0106] Step 7: Obtain the probability distribution of the final vector representation from Step 6 on the relationship label using a softmax classifier, thereby identifying the relationship type between entities.

[0107] The effectiveness of this method is verified below using real data. The datasets used in the experiment are the Semeval2010task8 dataset from the news domain, and the BioCreative VI CPR dataset and PGR dataset from the medical domain. The Semeval dataset is located at http: / / semeval2.fbk.eu / semeval2.php, the CPR dataset at https: / / biocreative.bioinformatics.udel.edu / tasks / biocreative-vi / track-5 / , and the CPR dataset at http: / / bioasq.lip6.fr / tools / BioASQword2vec / .

[0108] The CPR dataset primarily contains relationships between compounds and proteins. It provides three entity types ("CHEMICAL", "GENE-Y", and "GENE-N"), five common relations (CPR:3", "CPR:4", "CPR:5", "CPR:6", and "CPR:9"), and a special "None" relation. It has 16,107 training instances, 10,030 development instances, and 14,269 test instances. The PGR dataset primarily contains relationships between diseases and genes. It provides binary relations ("yes" and "no"). It has 18,451 training instances and 219 test instances. To test the performance of our model in other domains, we selected the SemEval-2010 Task 8 dataset. It contains 10,717 instances and provides 9 relations (such as "Cause-Effect", "Component-Whole", "Entity-Destination", and "Member-Collection"). Table 1 compares the F1 scores of our invention with other models on the SemEval-2010 Task 8 dataset.

[0109] Table 1

[0110] Train Dev Test Relation type CPR 16107 10030 14269 5 PGR 11780 - 219 2 SemEval2010 8000 - 2718 9

[0111] The method of this invention and the model on two medical datasets, CPR and PGR, are shown in Tables 2 and 3. Figure 3 A comparison was made. These systems are divided into three types based on the syntactic structures they use. Table 2 compares the F1 scores of this invention with other models on the CPR dataset; Table 3 compares the F1 scores of this invention with other models on the PGR dataset.

[0112] Table 2

[0113]

[0114]

[0115] Table 3

[0116]

[0117] As shown in the table, these results demonstrate that the structure of the input sentence is beneficial for relation extraction tasks. Models using dependency forests as input produce better performance than models using 1-best trees, indicating that the propagation of erroneous information caused by the low parsing accuracy of out-of-domain parsers can be mitigated by constructing a weighted structure. Models with forest structures (including AGGCN, ForestFT-DDCNN, and LF-GCN) achieve higher performance compared to models that encode fixed dependency forests generated during data preprocessing (EdgewiseGRN and KBest-GRN). Compared to LF-GCN, the method of this invention effectively mitigates the noise problem that occurs during the generation of hidden forests by preserving the global dependency information learned through multi-head attention. Figure 3 The performance of LF-GCN with different numbers of forests and the method of this invention were compared. The method of this invention consistently outperforms LF-GCN with the same number of forests. The method of this invention achieved scores of 62.4 and 92.4 on the CPR and PGR datasets, respectively, outperforming all other forest generation methods. These results demonstrate that the induced latent structure can capture task-specific information for better relation extraction.

[0118] The method of this invention was compared with the state-of-the-art model on SemEval. Table 4 shows the comparison results from the news dataset SemEval2010Task8.

[0119] Table 4 shows the test results on the SemEval dataset.

[0120]

[0121] The method of this invention outperforms dependency tree models using TreeGRN and GCN, and is comparable to dependency forest models using AGGCN and ForestFT-DDCNN.

[0122] The present invention was tested on three datasets: Semeval2010task8, CPR, and PGR. The results show that the method performs well in entity relation extraction tasks.

[0123] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for entity relation extraction based on multi-head attention, characterized in that Includes the following steps: Step 1: Construct training corpus by training based on GloVe English word vectors obtained from the general domain to obtain word vector representations; Step 2: Use Bi-LSTM to encode word vector representations to obtain the contextual representation of the sentence; Step 3: Obtain the global feature vector of the sentence and the feature representations of different subspaces through a multi-head attention mechanism; Step 4: Generate the corresponding hidden forests for different subspaces using the matrix tree theorem; In step 4, the method for generating a hidden forest using the matrix tree theorem is as follows: Step A41: Calculate the edge score and root score of each subspace using multi-head attention. Step A42: Using the edge score and root score as input, calculate the marginal probability of each edge to generate the hidden forest; Step 5: Input the hidden forest and the global feature vectors learned from the sentences into the graph convolutional neural network (GCN) for encoding, and then obtain the vector representations of entities and sentences through pooling operations respectively. Step 6: Fuse the sentence vector from Step 5 and the global feature vector from Step 3, and then combine them with the entity vector into a fully connected layer to obtain the final vector representation; Step 7: The final vector representation from Step 6 is used by a classifier to obtain the probability distribution on the relationship labels, thereby identifying the type of relationship between entities; In step 5, given the adjacency matrix The convolution computation of the i-th node in the l-th layer of the k-th potential forest is derived from the representation of the previous layer. As input and output update representation Defined as: ; in, and Let be the weight matrix and bias vector of the k-th hidden forest in the l-th layer, respectively. It is the activation function Sigmoid; In step 6, the method for obtaining the final vector representation of the text is as follows: The entity vector is obtained from the hidden forest in step 5 through convolution calculation. , Sentence vectors The global vector is convolved to obtain the sentence vector. Pooling the four vectors respectively yields Then, the two pooled sentence vectors are fused together, as follows: ; function The definition is as follows: = ; Finally and , The three vectors are concatenated, and the final vector representation is: 。 2. The entity relation extraction method based on multi-head attention as described in claim 1, characterized in that: In step 2, a bidirectional long short-term memory (Bi-LSTM) mechanism is used to obtain the context representation. The context representation h of the sentence is obtained according to the following formula: (1); (2); (3); (4); (5); (6); in, The input represents time t. This represents the hidden layer state at time t − 1. , , and These represent the forget gate, input gate, output gate, and [other gates] in the feature extraction process, respectively. The weights, coefficients , , and These represent the forget gate, input gate, output gate, and [other gates] in the feature extraction process, respectively. The weighting coefficients, , , and Let represent the bias values ​​for the forget gate, input gate, output gate, and feature extraction process, respectively, and tanh is the tangent hyperbolic function. It is the activation function Sigmoid; respectively using and This represents the output of the forward LSTM and the backward LSTM. For the i-th word, the outputs from both directions are combined to represent: = [ ; ] (7) 。 3. The entity relation extraction method based on multi-head attention as described in claim 1, characterized in that: The specific content of step 3 is as follows: A multi-head attention mechanism is employed to capture features of the input sentence from different perspectives. The number of heads in the multi-head attention mechanism is set to be equal to the number of different subspaces, with each head corresponding to a subspace structure for one perspective. Each subspace is represented by an adjacency matrix. The weights between relational nodes in the adjacency matrix are calculated using the following formula: (8) ; in , , These represent the query, key, and value, respectively, and d represents the dimension of the hidden layer.

4. The entity relation extraction method based on multi-head attention as described in claim 1, characterized in that: The specific content of step A41 is as follows: a) The edge score between the i-th node and the j-th node in the k-th hidden forest is expressed by the following formula: V (9); Among them, , , These represent the query, key, and value, respectively. and These are the projection parameters, where d represents the dimension of the hidden layer; b) Calculate the root fraction It represents the normalized probability that the i-th node is selected as the root node of the k-th potential forest: = (10); in, It is the hidden layer vector of the i-th node. yes The weights of the projection.

5. The entity relation extraction method based on multi-head attention as described in claim 1, characterized in that: The specific content of step A42 is as follows: According to the matrix tree theorem, the marginal probability of the dependency edge from the i-th word to the j-th word in the k-th forest is expressed as: (11); in, This represents the k-th hidden forest induced by the matrix tree theorem. This represents the context representation vector obtained in step 2. A weight vector representing the dependency relationship. This represents the number of dependencies in the k-th (k ∈ [1, N]) hidden forest. The corresponding hidden forest is obtained based on the marginal probability, where the k-th hidden forest is generated from the i-th node to the j-th node. Represented as: (12); (13); in, This represents the k-th Laplace matrix of the i-th node and the j-th node. Represents the Kronecker coefficient. Let represent the weighted adjacency matrix of the k-th forest.

6. The entity relation extraction method based on multi-head attention as described in claim 1, characterized in that: In step 7, the probability distribution on the relation labels is obtained using the Softmax classifier.

Images