A relation classification method based on pre-training model combined with syntactic subtree

Abstract

The present invention discloses a relation classification method based on a pre-trained model combined with a syntactic subtree. First, a word vector, a sentence representation vector, and an entity vector are constructed using a BERT pre-trained model. In order to combine syntactic information, a Spacy toolkit is used to perform dependency syntactic analysis on the sentence, and then the analysis result is pre-processed to obtain edges and edge categories. When combining syntactic information, a recurrent neural network RvNN is used to perform recursive calculations to obtain a representation vector for each subtree. The purpose of this step is to obtain topological information, semantic information, and edge category information of a syntactic dependency tree. The representation vector of each subtree is subjected to maximum pooling to obtain a representation vector of the tree. The entity vector, the sentence representation vector, and the tree representation vector are spliced, and then a relationship category prediction is performed. The present invention can better solve the problem of word vector and syntactic subtree information fusion, and can extract deeper and more informative word vectors, and has good robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of relationship classification, and in particular to a relationship classification method based on a pre-training model combined with a syntactic subtree. Background Art

[0002] Relation classification is the task of predicting the relationship category between two entities from unstructured text without human intervention.

[0003] The relation classification task is an important natural language processing task and is often used as an intermediate step in various natural language processing applications. In recent years, deep neural networks have been used for relation classification. The usual method uses dependency parsing or named entity recognition technology to extract features through deep neural networks to obtain semantic feature vectors.

[0004] Some experts proposed a method to apply recursive neural networks to relation classification, which assigns a matrix vector representation to each node in the syntactic analysis tree and calculates the complete sentence representation vector from bottom to top based on the grammatical structure of the parse tree. Some researchers proposed a CNN model that takes vocabulary embedding and position features as input. They concatenate the vocabulary features and the output vector of the CNN into a single vector, which is then fed into the softmax function to predict the relationship category. In 2015, experts proposed a factor-based component embedding model that builds sentence-level and substructure embeddings from vocabulary embeddings through dependency trees and named entities.

[0005] On the one hand, researchers handle relationship classification tasks by using a model called CR-CNN for sorting. The loss function of the model is based on pairwise sorting. In 2019, some researchers took advantage of the natural advantages of the BERT pre-trained model to convert text sentences into word vectors, and then concatenated the two entity vectors and input them into the softmax function for classification, making the model architecture very streamlined and greatly improving the accuracy. In 2020, some researchers used dependency syntactic analysis on the benchmark of the BERT pre-trained model. First, the predicates of the relationship were screened, and then the text sentences were input into the BERT pre-trained model to obtain the sentence vector. Then, the predicate vector and entity vector were concatenated and input into the classification function to predict the relationship category.

[0006] On the other hand, researchers believe that graph convolutional neural networks can better solve the problem of document-level relationship extraction (input a paragraph, there are multiple entities in the sentence, and determine the entity relationship). Before the graph neural network, the syntactic dependency tree is used to build the graph, and then the graph convolutional neural network is used for feature extraction, which better solves the problem of dependency loss between entities and achieves better results. Summary of the invention

[0007] After comprehensively considering the above problems, the present invention proposes a relationship classification method based on a pre-trained model combined with a syntactic subtree to address the problems existing in the prior art.

[0008] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0009] Step (1) segment the sentences S in the data set and construct an input sequence; pass the input sequence through the BERT pre-training model to obtain the input word vector of each word, the sentence representation vector of the whole sentence, and the entity vectors of the two relationship categories that need to be predicted.

[0010] Step (2) Use the Spacy tool to perform dependency syntactic analysis on the sentence S in the dataset to obtain the edges between words in the sentence, the edge categories, and the syntactic dependency tree, and use the mapping function to convert the edge categories into edge category ids.

[0011] Step (3) takes the two entities in the syntactic dependency tree as leaf nodes and calculates the nearest common ancestor node of the two leaf nodes; obtains a subtree with the nearest common ancestor node as the root and the two entity nodes as leaves, and inputs the word vector of the subtree part in the input word vector, the category id of the edge, and the edge into the recursive neural network RvNN for calculation to obtain a representation vector for each subtree.

[0012] Step (4) performs maximum pooling on all subtree representation vectors to obtain the tree representation vector of the sentence.

[0013] Step (5) concatenates the sentence representation vector, the sentence tree representation vector, and the two entity vectors whose relationship categories need to be predicted and inputs them into the softmax classification function for classification to predict the relationship category.

[0014] The beneficial effects of the present invention are as follows:

[0015] The present invention uses the BERT model to construct input word vectors. Compared with word vectors constructed by other methods, it contains richer information, extracts more features, and is more effective for subsequent feature fusion. Secondly, the recursive neural network is used to calculate the subtree representation vector, so that the model can learn the semantic information of the subtree, the topological information of the tree, and the category information of the edge. After that, the sentence representation vector, the tree representation vector, and the two entity vectors that need to predict the relationship category are spliced ​​to perform information fusion. Finally, the relationship category is classified and predicted to improve accuracy and robustness.

[0016] The present invention is experimented on Chinese and English data sets. The present invention can better solve the problem of word vector and syntactic subtree information fusion, and can extract deeper and more informative word vectors with good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart of the overall implementation scheme of the present invention

[0018] Figure 2 Overall architecture diagram of the model of the present invention

[0019] Figure 3 Dependency syntactic analysis result diagram of the present invention

[0020] Figure 4 Recursive neural network diagram of the present invention

[0021] Specific implementation method

[0022] The present invention is further described below in conjunction with the accompanying drawings.

[0023] The present invention generates word vectors using the BERT pre-training model, enhances the information carried by the word vectors, and simultaneously uses the Spacy dependency syntax analysis tool to obtain edges and edge categories. The nearest common ancestor node of the two entities is calculated, and a tree with the nearest common group as the root and the two entity nodes as leaves is obtained. The word vectors, edges, and category ids of the edges in the input word vector are input into the neural network, so that the model learns syntactic information, so that the subtree vector has the semantic information of the syntax analysis tree, the topological information of the tree, and the category information of the edges. Finally, the neural network can obtain a more comprehensive word vector when encoding, and effectively improve the accuracy of relationship classification.

[0024] Reference Figure 1 is the overall flow chart of the present invention, a relation classification method based on pre-training model combined with syntactic subtree, referring to Figure 2 The overall model diagram includes the following steps:

[0025] Step (1) segment the sentences S in the data set and construct an input sequence; pass the input sequence through the BERT pre-training model to obtain the input word vector of each word, the sentence representation vector of the whole sentence, and the entity vectors of the two relationship categories that need to be predicted.

[0026] Step (2) Use the Spacy tool to perform dependency syntactic analysis on the sentence S in the dataset to obtain the edges between words in the sentence, the edge categories, and the syntactic dependency tree, and use the mapping function to convert the edge categories into edge category ids.

[0027] Step (3) takes the two entities in the syntactic dependency tree as leaf nodes and calculates the nearest common ancestor node of the two leaf nodes; obtains a subtree with the nearest common ancestor node as the root and the two entity nodes as leaves, and inputs the word vector of the subtree part in the input word vector, the category id of the edge, and the edge into the recursive neural network RvNN for calculation to obtain a representation vector for each subtree.

[0028] Step (4) performs maximum pooling on all subtree representation vectors to obtain the tree representation vector of the sentence.

[0029] Step (5) concatenates the sentence representation vector, the sentence tree representation vector, and the two entity vectors whose relationship categories need to be predicted and inputs them into the softmax classification function for classification to predict the relationship category.

[0030] Furthermore, the specific implementation process of step (1) is as follows:

[0031] 1-1 First, use the Spacy toolkit to segment the sentence S and construct the input sequence Se = {w1, w2, ..., wn}, where n represents the length of the sequence, and w1, w2, ..., wn is each word of the sentence;

[0032] 1-2 Assume that the two entities are e1 and e2. After the input sequence is constructed, it will be input into the BERT pre-trained model to obtain the input word vector H∈R for each word. d×n , the sentence representation vector SV of the whole sentence, and two entity vectors E1 and E2 that need to predict the relationship category. Where d is the dimension of the word vector, n is the sequence length, the dimensions of SV, E1, and E2 are 1×768, and the process is expressed as follows:

[0033] SV, H=BERT(Se)=SV, {h1, h2,..., hn} (1)

[0034] Among them, h1, h2, …, hn are the input word vectors after encoding by the BERT pre-trained model, SV is the sentence representation vector, Se is the input sequence in step 1-1, and BERT is the pre-trained model. The entity vectors corresponding to entities e1 and e2 are E1 and E2.

[0035] Furthermore, the specific implementation process of step (2) is as follows:

[0036] 2-1 Input sentence S into the Spacy syntactic analysis tool:

[0037] I=nlp(S) (2)

[0038] In formula (2), S is the input sentence, nlp is the Spacy dependency parsing toolkit, and I is the output. Figure 3 For the result graph of dependency syntactic analysis, the output I is preprocessed to obtain the edge Edge and the edge category EdgeType on the syntactic dependency tree. The edge category EdgeType is converted to the edge category id, i.e. EdgeTypeId, using a mapping function.

[0039] Furthermore, the specific implementation process of step (3) is as follows:

[0040] 3-1 Take the two entities as leaf nodes and calculate their nearest common ancestor node. Obtain the subtree subTree with the nearest common ancestor node as the root and the two entities as leaf nodes in I, and take out the corresponding vector subTreeV of the subtree subTree in H. The process is expressed as follows:

[0041] Root = lca(e1, e2) (3)

[0042] subTree=acquireSubTree(I, Root, e1, e2) (4)

[0043] subTreeV=acquire(H, subTree) (5)

[0044] Where, e1 and e2 are two entities, lca is a function for calculating the nearest common ancestor node, Root is the nearest common ancestor node with e1 and e2 as leaf nodes. I is the output result of formula (2) in step 2-1, acquireSubTree is a function for obtaining a subtree, subTree is a subtree with Root as the root and e1 and e2 as leaf nodes. H is the output vector of formula (1) in step 1-1, and subTreeV is the corresponding vector of the subtree with Root as the root and e1 and e2 as leaf nodes.

[0045] 3-2 Input the subtree corresponding vector subTreeV, the edge category id EdgeTypeId, and the edge Edge into the recursive neural network RvNN, and recursively calculate the representation vector of each subtree of the current subtree. The formula is as follows:

[0046] TreeV=RvNN(subTreeV, EdgeTypeId, Edge) (6)

[0047] Wherein, Edge is the edge obtained in step 2-1, EdgeTypeId is the category id of the edge, H is the output vector of formula (1) in step 1-2, RvNN is a recursive neural network, and TreeV is the subtree representation vector of the output of the recursive neural network.

[0048] For the representation vector of each subtree in the current subtree, the representation vector of each subtree in the current subtree is multiplied by the category id vector of the edge and then summed. The specific calculation formula is as follows:

[0049] ei=map(EdgeTypeId[i]) (7)

[0050] x i =subTreeV[i]+∑ p e p *xp (8)

[0051] TreeV = [x1, x2, ... xk] (9)

[0052] Where EdgeTypeId[i] is the category ID of the ith edge, map is the edge category ID mapping function, ei is the vector of the category ID mapping of the ith edge. subTreeV[i] is the vector corresponding to the i-th subtree, p is the adjacent subtree node of the i-th node, e p is the vector of the category ID mapping of the pth edge, x p To calculate the characterization vector of the pth subtree, the above x1, x2, ... xk are combined to form the characterization vector TreeV of each subtree in the subtree, where k is the size of the subtree with Root as the root node. Figure 4 Model diagram of a recurrent neural network.

[0053] Furthermore, the specific implementation process of step (4) is as follows:

[0054] 4-1 Input the representation vector TreeV into the maximum pooling function to obtain the tree representation vector of the sentence. The formula is as follows:

[0055] SentenceTreeV=Maxpool(TreeV) (10)

[0056] Among them, TreeV is the representation vector of the output of formula (6) in step 3-2, Maxpool is the maximum pooling function, and SentenceTreeV is the tree representation vector of the sentence.

[0057] Furthermore, the specific implementation process of step (5) is as follows:

[0058] 5-1 Concatenate the sentence representation vector SV, the sentence tree representation vector SentenceTreeV, and the two entity vectors E1 and E2 whose relationship categories need to be predicted, and then input them into the softmax function to obtain the category probability distribution. The maximum value is the predicted relationship category. The formula is as follows:

[0059] V=Cat(SV,SentenceTreeV,E1,E2) (11)

[0060] probability = softmax(V) (12)

[0061] predictType=argmax(probility) (13)

[0062] Among them, E1 and E2 are two entity vectors whose relationship categories need to be predicted, SentenceTreeV is the output vector of formula (11) in step 4-1, SV is the output vector of formula (1) in step 1-2, E1 and E2 are entity vectors. Cat is the concatenation function. V is the concatenated output vector. Softmax is the normalized exponential function, probility is the predicted probability distribution of all relationship categories. Argmax is the maximum subscript function to obtain the category probability distribution. PredictType is the category of the predicted relationship.

[0063] 5-2 The loss function is the cross entropy loss function, and the formula is as follows:

[0064]

[0065] Where M is the number of relationship categories, q is the qth sample, N is the total number of samples, and y pc is a sign function. If the true category of sample p is equal to c, it takes 1, otherwise it takes 0. log is the logarithmic function, p qc is the probability that sample q belongs to category c.

Claims

1. A relation classification method based on a pre-trained model combined with a syntactic subtree, characterized in that The following steps are involved: Step (1) segment the sentences S in the dataset and construct an input sequence; Pass the input sequence through the BERT pre-training model to obtain the input word vector of each word, the sentence representation vector of the whole sentence, and the entity vectors of the two relationship categories that need to be predicted; Step (2) Use the Spacy tool to perform dependency syntactic analysis on the sentence S in the dataset, obtain the edges between words in the sentence, the edge categories, and the syntactic dependency tree, and use the mapping function to convert the edge categories into edge category ids; Step (3) takes the two entities in the syntactic dependency tree as leaf nodes and calculates the nearest common ancestor node of the two leaf nodes; obtains a subtree with the nearest common ancestor node as the root and the two entity nodes as leaves, and inputs the word vector of the subtree part in the input word vector, the category id of the edge, and the edge into the recursive neural network RvNN for calculation to obtain a representation vector for each subtree; Step (4) performs maximum pooling on all subtree representation vectors to obtain the tree representation vector of the sentence; Step (5) concatenates the sentence representation vector, the sentence tree representation vector, and the two entity vectors whose relationship categories need to be predicted and inputs them into the softmax classification function for classification to predict the relationship category.

2. According to claim 1, a relationship classification method based on a pre-trained model combined with a syntactic subtree is characterized in that The specific implementation process of step (1) is as follows: 1-1 First, use the Spacy toolkit to segment the sentence S and construct the input sequence Se = {w1, w2, ..., wn}, where n represents the length of the sequence, and w1, w2, ..., wn is each word of the sentence; 1-2 Assume that the two entities are e1 and e2. After the input sequence is constructed, it will be input into the BERT pre-trained model to obtain the input word vector H∈R for each word. d×n , the sentence representation vector SV of the whole sentence, and two entity vectors E1 and E2 whose relationship categories need to be predicted; where d is the dimension of the word vector, n is the sequence length, and the dimension of SV, E1, and E2 is 1×768. The process is expressed as follows: SV,H=BERT(Se)=SV,{h1,h2,…,hn} (1) Among them, h1, h2, …, hn are the input word vectors after encoding by the BERT pre-trained model, SV is the sentence representation vector, Se is the input sequence in step 1-1, and BERT is the pre-trained model; the entity vectors corresponding to entities e1 and e2 are E1 and E2.

3. A relation classification method based on a pre-trained model combined with a syntactic subtree according to claim 2, characterized in that The specific implementation process of step (2) is as follows: 2-1 Input sentence S into the Spacy syntactic analysis tool: I=nlp(S) (2) In formula (2), S is the input sentence, nlp is the loaded Spacy dependency parsing toolkit, and I is the output; the output I is preprocessed to obtain the edge Edge and the edge category EdgeType on the syntactic dependency tree; the mapping function is used to convert the edge category EdgeType into the edge category id, namely EdgeTypeId.

4. A relation classification method based on a pre-trained model combined with a syntactic subtree according to claim 3, characterized in that The specific implementation process of step (3) is as follows: 3-1 Take the two entities as leaf nodes and calculate their nearest common ancestor node. Obtain the subtree subTree with the nearest common ancestor node as the root and the two entities as leaf nodes in I, and take out the corresponding vector subTreeV of the subtree subTree in H. The process is expressed as follows: Root=lca(e1,e2) (3) subTree=acquireSubTree(I,Root,e1,e2) (4) subTreeV=acquire(H,subTree) (5) Where, e1 and e2 are two entities, lca is the function for calculating the nearest common ancestor node, Root is the nearest common ancestor node with e1 and e2 as leaf nodes; I is the output result of formula (2) in step 2-1, acquireSubTree is the function for obtaining a subtree, subTree is the subtree with Root as the root and e1 and e2 as leaf nodes; H is the output vector of formula (1) in step 1-2, subTreeV is the corresponding vector of the subtree with Root as the root and e1 and e2 as leaf nodes; 3-2 Input the subtree corresponding vector subTreeV, the edge category idEdgeTypeId, and the edge Edge into the recursive neural network RvNN, and recursively calculate the representation vector of each subtree of the current subtree. The formula is as follows: TreeV=RvNN(subTreeV,EdgeTypeId,Edge) (6) Wherein, Edge is the edge obtained in step 2-1, EdgeTypeId is the category id of the edge, H is the output vector of formula (1) in step 1-2; RvNN is a recursive neural network, and TreeV is the subtree representation vector of the output of the recursive neural network; For the representation vector of each subtree in the current subtree, the representation vector of each subtree in the current subtree is multiplied by the category id vector of the edge and then summed. The specific calculation formula is as follows: e i =map(EdgeTypeId[i]) (7) x i =subTreeV[i]+∑ p e p *x p (8) TreeV=[x1,x2,…xk] (9) Where EdgeTypeId[i] is the category ID of the ith edge, map is the edge category ID mapping function, and e i is the vector mapped to the category ID of the i-th edge; subTreeV[i] is the vector corresponding to the i-th subtree, p is the adjacent subtree node of the i-th node, e p is the vector of the category ID mapping of the pth edge, x p To calculate the characterization vector of the pth subtree currently, the above x1, x2, …xk are combined to form the characterization vector TreeV of each subtree in the subtree, where k is the size of the subtree with Root as the root node.

5. A method for relation classification based on a pre-trained model combined with a syntactic subtree according to claim 4, characterized in that The specific implementation process of step (4) is as follows: 4-1 Input the representation vector TreeV into the maximum pooling function to obtain the tree representation vector of the sentence. The formula is as follows: SentenceTreeV=Maxpool(TreeV) (10) Among them, TreeV is the representation vector of the output of formula (6) in step 3-2, Maxpool is the maximum pooling function, and SentenceTreeV is the tree representation vector of the sentence.

6. A method for relation classification based on a pre-trained model combined with a syntactic subtree according to claim 5, characterized in that The specific implementation process of step (5) is as follows: 5-1 Concatenate the sentence representation vector SV, the sentence tree representation vector SentenceTreeV, and the two entity vectors E1 and E2 whose relationship categories need to be predicted, and then input them into the softmax function to obtain the category probability distribution. The maximum value is the predicted relationship category. The formula is as follows: V=Cat(SV,SentenceTreeV,E1,E2) (11) probability = softmax(V) (12) predictType=argmax(probility) (13) Among them, E1, E2 are two entity vectors whose relationship categories need to be predicted, SentenceTreeV is the output vector of formula (11) in step 4-1, SV is the output vector of formula (1) in step 1-2, E1, E2 are entity vectors; Cat is the concatenation function; V is the concatenated output vector; softmax is the normalized exponential function, probility is the predicted probability distribution of all relationship categories; argmax is the maximum subscript function to obtain the category probability distribution; predictType is the category of the predicted relationship; 5-2 The loss function is the cross entropy loss function, and the formula is as follows: Where M is the number of relationship categories, q is the qth sample, N is the total number of samples, and y qc is a sign function. If the true category of sample q is equal to c, it takes 1, otherwise it takes 0. log is the logarithmic function, p qc is the probability that sample q belongs to category c.

Images