A relation extraction method based on syntactic dependency and fusion information

Abstract

The invention discloses a method for extracting relations based on syntactic dependency and fusion information. First, a BERT pre-training model is used to construct a word vector. Secondly, the long short-term memory network has the problem of not considering the lexical position information. The invention adds a position vector to the word vector before the LSTM network. In order to combine the syntactic information, the Spacy toolkit is used to perform dependency syntactic analysis on the sentence, and then the analysis result is preprocessed to obtain the syntactic information. When combining the syntactic information, the word vector is fused with the syntactic information using a key-value memory network to obtain a word vector containing the syntactic information. Then, the LSTM network is used to extract features. Finally, the LSTM network is used for decoding, predicting relations and predicting entities. The invention conducts experiments on Chinese and English data sets. The invention can better solve the problem of word vector and syntactic information fusion, and can extract deeper and more informative word vectors, and has good robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of relation extraction, and in particular to a relation extraction method based on syntactic dependency and fusion information. Background Art

[0002] Relation extraction is the task of extracting triplets (relation category, head entity, tail entity) from unstructured text without human intervention.

[0003] There are two solutions to the relationship extraction task: 1 pipeline extraction, 2 joint learning extraction. One pipeline extraction method is to first extract two entities and then classify the relationship between the entity pairs. For example, the BIO (Begin, Inside, Outside) sequence labeling method is used for entity prediction, and then the entity vectors are input into the classifier for relationship classification. Although the pipeline model has achieved relatively good results, the overall accuracy will decrease. Joint learning can implicitly establish the correlation between tasks, so many corresponding methods have been proposed. For example, in 2019, a joint model CopyRE based on a copy mechanism was proposed to convert the joint extraction task into a triple generation task. This method can not only extract triples, but also solve the problem of entity coverage. However, during the operation, since the vector is input into the recurrent neural network (RNN), and the RNN model has poor encoding effect on complex or long sentences, the previous information will be lost when encoding in the following text, which makes the prediction inaccurate when predicting the relationship or entity in the following text, which has a serious impact.

[0004] Later, the long short-term memory network (LSTM) proposed by experts can accurately solve the problem of long-distance dependency. The performance of the long short-term memory network in long sentences is significantly better than that of ordinary neural networks and recurrent neural networks. Due to the wide application of attention mechanism in deep learning, some researchers proposed to combine attention mechanism with relation extraction. By adding attention mechanism after neural network extraction, the influence of each word on the result is obtained by analyzing the sentence, and the corresponding weight vector is constructed, that is, the weight of key words is large and the weight of irrelevant words is small. The introduction of attention mechanism improves the accuracy of relation extraction task, but the problem that comes with it is that when there are many entities in the sentence, the weight of all entities will be large, resulting in the model being difficult to distinguish the entities corresponding to the relationship when predicting the corresponding entity under the condition that the relationship is determined. Some researchers use graph convolutional neural network (GCN) for relation extraction. First, the sentence is analyzed using dependency syntactic analysis tools to obtain the syntactic analysis tree, and then the sentence vector and syntactic analysis tree are input into the graph convolutional neural network, so that the extracted feature vector information is more sufficient (including vocabulary information, position information, and syntactic information). However, due to the complexity of the graph structure, the amount of calculation continues to increase, and graph convolutional neural networks can only be used in specific fields, so the application of graph convolutional neural networks is not wide. Other researchers have introduced a variety of strategies, such as multi-task learning, such as running BIO sequence labeling tasks while performing entity prediction to increase the accuracy of entity prediction. Summary of the invention

[0005] The purpose of the present invention is to propose a relation extraction method based on syntactic dependency and fusion information in view of the problems existing in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem comprises the following steps:

[0007] Step (1) Segment the sentence S to construct an input sequence; use the input sequence as the input of the BERT pre-trained model, and obtain the basic word vector of each word after BERT model preprocessing;

[0008] Step (2) Use the Spacy dependency syntactic analysis tool to process the sentence S in the data set, obtain the dependency relationship and edge information between words in the sentence, and use the conversion function to convert the dependency relationship into a relationship vector;

[0009] Step (3) For each word in the sentence, the position vector is calculated using the method of calculating the position vector of the transformer model; then the position vector is added to each word, and the basic word vector and the position vector information are fused to obtain a fused vector;

[0010] Step (4) inputs the fusion vector, relationship vector and edge information into the key-value memory neural network to generate a syntactic information vector for each word, which is then concatenated with the fusion vector in step (3);

[0011] Step (5) inputs the concatenated vector into the LSTM network for encoding, and outputs the encoded feature vector;

[0012] Step (6) uses LSTM network decoding; during the decoding process, the relationship category is first predicted, and then the two entities of the relationship category are predicted.

[0013] Furthermore, the specific implementation process of step (1) is as follows:

[0014] 1-1 First, use the Spacy toolkit to perform word segmentation to construct the input sequence S = {w1, w2, ..., wn}, where n represents the sequence length, and then construct the sentence vector X. The formula is as follows:

[0015] X=(w1,w2,…,wn) (1)

[0016] Among them, w1, w2, …, wn constitute the sentence vector as the input vector;

[0017] After the sentence vector is constructed, it will be input into the BERT pre-training model to obtain a more informative basic word vector, denoted as H∈R d×n , where d is the dimension of the output basic word vector, n is the sequence length, and the process is expressed as follows:

[0018] H=BERT(X)={h1,h2,…,hn} (2)

[0019] Among them, h1, h2, …, hn are the vectors after encoding by the BERT pre-training model, X is the sentence vector output by formula (1) in step 1-1, and h1, h2, …, hn are merged into the basic word vector H.

[0020] Furthermore, the specific implementation process of step (2) is as follows:

[0021] 2-1 Input the sentence into the Spacy syntactic analysis tool:

[0022] I=nlp(sentence) (3)

[0023] In formula (3), sentence is the input sentence, nlp is the loaded Spacy dependency parsing toolkit, and I is the output. The output I is preprocessed, including word subscript and dependency tree processing, so as to obtain the dependency relationship Rela and edge information EdgeInfo of the nodes on the dependency tree, and the dependency relationship Rela is converted into a relation vector relation using the Embedding function of pytorch.

[0024] Furthermore, the specific implementation process of step (3) is as follows:

[0025] 3-1 Perform position encoding on the basic word vector H, and then fuse the basic word vector and the position vector to obtain the fused position vector Pos:

[0026] Pos=PositionEncoding(H) (4)

[0027] Where H is the basic word vector output by equation (2) in step 1-2, and PositionEncoding is the encoding function;

[0028] Then the position vector Pos and the basic word vector H are concatenated to form the fusion vector XP as shown in formula (5);

[0029] XP=Cat(H,Pos) (5)

[0030] Among them, Cat is the splicing function, and XP is the fusion vector output after splicing.

[0031] Furthermore, the specific implementation process of step (4) is as follows:

[0032] 4-1 Input the concatenated output fusion vector XP, relation vector relation, and edge information EdgeInfo into the key-value memory network to obtain the syntactic vector corresponding to each word. The formula is as follows:

[0033] syntax=KVMN(XP,relation,EdgeInfo) (6)

[0034] Where XP is the fusion vector of formula (5) in step 3-1, relation is the relation vector in step 2-1, EdgeInfo is the edge information in step 2-1, KVMN is the key-value memory network, and syntax is the syntax vector;

[0035] 4-2 Concatenate the syntactic vector syntax and the fusion vector XP to obtain the final input vector InputV.

[0036] Furthermore, the specific implementation process of step (5) is as follows:

[0037] 5-1 Input the input vector InputV into the LSTM network for encoding, and output the encoded feature vector:

[0038] output,h idden=LSTM(InputV) (7)

[0039] Among them, InputV is the input vector concatenated in step 4-1, LSTM is a long short-term memory network, output is the output vector of the last layer of the LSTM network, and hidden is the hidden layer output vector.

[0040] Furthermore, the specific implementation process of step (6) is as follows:

[0041] 6-1 maps the output vector output to another vector V through an LSTM gating unit;

[0042] 6-2. The V vector is mapped to all relationship categories through the fully connected layer network to obtain the probability distribution of each relationship category, and the relationship category with the highest probability in the probability distribution is selected for prediction; after the current relationship category prediction is completed, the sequence labeling algorithm is used to predict the entity subscript with the highest probability corresponding to the current relationship category, thereby obtaining two entities;

[0043] 6-3. Use the V vector of this step as the output vector of the next step, and repeat steps 6-1 and 6-2 until all relationship predictions are completed. The implementation formula is as follows:

[0044] V = lstm_cell (output) (8)

[0045] Where output is the output vector of formula (7) in step 5-1, lstm_cell is the LSTM gating unit, and V is another vector obtained by mapping;

[0046] The relationship category prediction formula is as follows:

[0047] relationType=do_predict(V) (9)

[0048] Among them, do_predict is the relationship prediction function, relationType is the predicted relationship;

[0049] The entity prediction formula is as follows:

[0050] entity=NER(V) (10)

[0051] Among them, NER is the named entity recognition model, entity is the obtained entity output; repeat the entity prediction once to obtain two entities of the current relationship category.

[0052] 6-4 During training, the standard gradient descent algorithm is used to update the network parameters, using the cross entropy loss function and L2 regularization, and the calculation formula is as follows:

[0053]

[0054]

[0055]

[0056] The overall loss consists of two parts, as shown in formula (11), first the weighted entity loss, then the relation loss; For the overall loss, For physical losses, is the relationship loss, λ is the loss weight of the label; for entity loss The calculation formula is (13), p(tag * |S) is each tag category tag of each word in sentence S * The probability distribution of ; taking the negative logarithm of the above probability is the entity loss; for the relationship loss The calculation formula is (12): Represents each relation predicted by S in the sentence The probability distribution of , and then take the negative logarithm sum, where is the predicted value, y is the true value, and t is the prediction to the tth step.

[0057] The beneficial effects of the present invention are as follows:

[0058] The present invention uses the BERT model to construct basic word vectors. Compared with other word vectors, the extracted features are more sufficient and more effective for subsequent calculations. Secondly, before inputting into the neural network, adding position information to the word vector can solve the problem that the neural network does not consider position information. Finally, the syntactic information and semantic information are combined in the model to improve the information richness of the feature vector. This model is run on Chinese and English data sets. The results show that the present invention can solve the problem of feature extraction very well, and can combine syntactic dependencies to better extract features, has good robustness, and can effectively improve the accuracy of relationship extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Flow chart of the overall implementation scheme of the present invention

[0060] Figure 2 Overall architecture diagram of the model of the present invention

[0061] Figure 3 Word vector construction diagram of the present invention

[0062] Figure 4 Dependency syntactic analysis result diagram of the present invention

[0063] Figure 5 The present invention integrates the location information map

[0064] Figure 6 Key-value memory network model diagram of the present invention DETAILED DESCRIPTION

[0065] The present invention is further described below in conjunction with the accompanying drawings.

[0066] The present invention uses the BERT model to generate word vectors, combines position information, enhances the information carried by the word vectors, and uses dependency syntactic analysis to obtain syntactic information. According to the obtained syntactic information, it is integrated into the neural network so that the word vector can not only obtain contextual information, but also obtain syntactic information. The neural network can obtain a more comprehensive word vector when encoding. The word vector contains not only semantic information, but also position information and syntactic information, which can effectively improve the accuracy of relationship extraction.

[0067] Reference Figure 1 is the overall flow chart of the present invention, a relation extraction method based on syntactic dependency and fusion information, referring to Figure 2 The overall model diagram includes the following steps:

[0068] Step (1) Segment the sentence S to construct an input sequence; use the input sequence as the input of the BERT pre-trained model, and obtain the basic word vector of each word after BERT model preprocessing;

[0069] Step (2) Use the Spacy dependency syntactic analysis tool to process the sentence S in the data set, obtain the dependency relationship and edge information between words in the sentence, and use the conversion function to convert the dependency relationship into a relationship vector;

[0070] Step (3) For each word in the sentence, the position vector is calculated using the method of calculating the position vector of the transformer model; then the position vector is added to each word, and the basic word vector and the position vector information are fused to obtain a fused vector;

[0071] Step (4) inputs the fusion vector, relationship vector and edge information into the key-value memory neural network to generate a syntactic information vector for each word, which is then concatenated with the fusion vector in step (3);

[0072] Step (5) inputs the concatenated vector into the LSTM network for encoding, and outputs the encoded feature vector;

[0073] Step (6) uses LSTM network decoding; during the decoding process, the relationship category is first predicted, and then the two entities of the relationship category are predicted.

[0074] Furthermore, the specific implementation process of step (1) is as follows:

[0075] 1-1 First, use the Spacy toolkit to perform word segmentation to construct the input sequence S = {w1, w2, ..., wn}, where n represents the sequence length, and then construct the sentence vector X. The formula is as follows:

[0076] X=(w1,w2,…,wn) (1)

[0077] Among them, w1, w2, ..., wn constitute the sentence vector as the input vector;

[0078] After the sentence vector is constructed, it will be input into the BERT pre-training model to obtain a more informative basic word vector, denoted as H∈R d×n , where d is the dimension of the output basic word vector, n is the sequence length, and the process is expressed as follows:

[0079] H=BERT(X)={h1, h2,…,hn} (2)

[0080] Among them, h1, h2, ..., hn are the vectors after encoding by the BERT pre-training model, X is the sentence vector output by formula (1) in step 1-1, and h1, h2, ..., hn are combined into the basic word vector H. Figure 3 .

[0081] Further, such as Figure 4 As shown, the specific implementation process of step (2) is as follows:

[0082] 2-1 Input the sentence into the Spacy syntactic analysis tool:

[0083] I=nlp(sentence) (3)

[0084] In formula (3), sentence is the input sentence, nlp is the loaded Spacy dependency parsing toolkit, and I is the output. The output I is preprocessed, including word subscript and dependency tree processing, so as to obtain the dependency relationship Rela and edge information EdgeInfo of the nodes on the dependency tree, and the dependency relationship Rela is converted into a relation vector relation using the Embedding function of pytorch.

[0085] Furthermore, the specific implementation process of step (3) is as follows:

[0086] 3-1 Perform position encoding on the basic word vector H, and then fuse the basic word vector and the position vector to obtain the fused position vector Pos:

[0087] Pos=PositionEncoding(H) (4)

[0088] Where H is the basic word vector output by equation (2) in step 1-2, and PositionEncoding is the encoding function;

[0089] Then the position vector Pos and the basic word vector H are concatenated to form the fusion vector XP as shown in formula (5);

[0090] XP=Cat(H,Pos) (5)

[0091] Among them, Cat is the splicing function, and XP is the fusion vector output after splicing. Figure 5 .

[0092] Furthermore, the specific implementation process of step (4) is as follows:

[0093] 4-1 Input the concatenated output fusion vector XP, relation vector relation, and edge information EdgeInfo into the key-value memory network to obtain the syntactic vector corresponding to each word. The formula is as follows:

[0094] syntax=KVMN(XP,relation,EdgeInfo) (6)

[0095] Where XP is the fusion vector of formula (5) in step 3-1, relation is the relation vector in step 2-1, EdgeInfo is the edge information in step 2-1, KVMN is the key-value memory network, and syntax is the syntax vector;

[0096] 4-2 The syntax vector syntax is concatenated with the fusion vector XP to obtain the final input vector InputV. Figure 6 .

[0097] Furthermore, the specific implementation process of step (5) is as follows:

[0098] 5-1 Input the input vector InputV into the LSTM network for encoding, and output the encoded feature vector: output,hidden=LSTM(InputV) (7)

[0099] Among them, InputV is the input vector concatenated in step 4-1, LSTM is a long short-term memory network, output is the output vector of the last layer of the LSTM network, and hidden is the hidden layer output vector.

[0100] Furthermore, the specific implementation process of step (6) is as follows:

[0101] 6-1 maps the output vector output to another vector V through an LSTM gating unit;

[0102] 6-2. The V vector is mapped to all relationship categories through the fully connected layer network to obtain the probability distribution of each relationship category, and the relationship category with the highest probability in the probability distribution is selected for prediction; after the current relationship category prediction is completed, the sequence labeling algorithm is used to predict the entity subscript with the highest probability corresponding to the current relationship category, thereby obtaining two entities;

[0103] 6-3. Use the V vector of this step as the output vector of the next step, and repeat steps 6-1 and 6-2 until all relationship predictions are completed. The implementation formula is as follows:

[0104] V = lstm_cell (output) (8)

[0105] Where output is the output vector of formula (7) in step 5-1, lstm_cell is the LSTM gating unit, and V is another vector obtained by mapping;

[0106] The relationship category prediction formula is as follows:

[0107] relationType=do_predict(V) (9)

[0108] Among them, do_predict is the relationship prediction function, relationType is the predicted relationship;

[0109] The entity prediction formula is as follows:

[0110] entity=NER(V) (10)

[0111] Among them, NER is the named entity recognition model, entity is the obtained entity output; repeat the entity prediction once to obtain two entities of the current relationship category.

[0112] 6-4 During training, the standard gradient descent algorithm is used to update the network parameters, using the cross entropy loss function and L2 regularization, and the calculation formula is as follows:

[0113]

[0114]

[0115]

[0116] The overall loss consists of two parts, as shown in formula (11), first the weighted entity loss, then the relation loss; For the overall loss, For physical losses, is the relationship loss, λ is the loss weight of the label; for entity loss The calculation formula is (13), p(tag * |S) is each tag category tag of each word in sentence S * The probability distribution of ; taking the negative logarithm of the above probability is the entity loss; for the relationship loss The calculation formula is (12): Represents each relation predicted by S in the sentence The probability distribution of , and then take the negative logarithm sum, where is the predicted value, y is the true value, and t is the prediction to the tth step.

Claims

1. A relation extraction method based on syntactic dependency and fusion information, characterized in that The following steps are involved: Step (1) Segment the sentence S to construct an input sequence; use the input sequence as the input of the BERT pre-trained model, and obtain the basic word vector of each word after BERT model preprocessing; Step (2) Use the Spacy dependency syntactic analysis tool to process the sentence S in the data set, obtain the dependency relationship and edge information between words in the sentence, and use the conversion function to convert the dependency relationship into a relationship vector; Step (3) For each word in the sentence, the position vector is calculated using the method of calculating the position vector of the transformer model; then the position vector is added to each word, and the basic word vector and the position vector information are fused to obtain a fused vector; Step (4) inputs the fusion vector, relationship vector and edge information into the key-value memory neural network to generate a syntactic information vector for each word, which is then concatenated with the fusion vector in step (3); Step (5) inputs the concatenated vector into the LSTM network for encoding, and outputs the encoded feature vector; Step (6) uses LSTM network decoding; during the decoding process, the relationship category is first predicted, and then the two entities of the relationship category are predicted.

2. According to claim 1, a method for extracting relations based on syntactic dependency and fusion information is characterized in that The specific implementation process of step (1) is as follows: 1-1 First, use the Spacy toolkit to perform word segmentation to construct the input sequence S = {w1, w2, ..., wn}, where n represents the sequence length, and then construct the sentence vector X. The formula is as follows: X=(w1,w2,…,wn) (1) Among them, w1, w2, …, wn constitute the sentence vector as the input vector; After the sentence vector is constructed, it will be input into the BERT pre-training model to obtain a more informative basic word vector, denoted as H∈R d×n , where d is the dimension of the output basic word vector, n is the sequence length, and the process is expressed as follows: H=BERT(X)={h1,h2,…,hn} (2) Among them, h1, h2, …, hn are the vectors after encoding by the BERT pre-training model, X is the sentence vector output by formula (1) in step 1-1, and h1, h2, …, hn are merged into the basic word vector H.

3. A method for extracting relations based on syntactic dependency and fusion information according to claim 2, characterized in that The specific implementation process of step (2) is as follows: 2-1 Input the sentence into the Spacy syntactic analysis tool: I=nlp(sentence) (3) In formula (3), sentence is the input sentence, nlp is the loaded Spacy dependency parsing toolkit, and I is the output. The output I is preprocessed, including word subscript and dependency tree processing, so as to obtain the dependency relationship Rela and edge information EdgeInfo of the nodes on the dependency tree, and the dependency relationship Rela is converted into a relation vector relation using the Embedding function of pytorch.

4. A method for extracting relations based on syntactic dependency and fusion information according to claim 3, characterized in that The specific implementation process of step (3) is as follows: 3-1 Perform position encoding on the basic word vector H, and then fuse the basic word vector and the position vector to obtain the fused position vector Pos: Pos=PositionEncoding(H) (4) Where H is the basic word vector output by equation (2) in step 1-2, and PositionEncoding is the encoding function; Then the position vector Pos and the basic word vector H are concatenated to form the fusion vector XP as shown in formula (5); XP=Cat(H,Pos) (5) Among them, Cat is the splicing function, and XP is the fusion vector output after splicing.

5. A method for extracting relations based on syntactic dependency and fusion information according to claim 4, characterized in that The specific implementation process of step (4) is as follows: 4-1 Input the concatenated output fusion vector XP, relation vector relation, and edge information EdgeInfo into the key-value memory network to obtain the syntactic vector corresponding to each word. The formula is as follows: syntax=KVMN(XP,relation,EdgeInfo) (6) Where XP is the fusion vector of formula (5) in step 3-1, relation is the relation vector in step 2-1, EdgeInfo is the edge information in step 2-1, KVMN is the key-value memory network, and syntax is the syntax vector; 4-2 Concatenate the syntactic vector syntax and the fusion vector XP to obtain the final input vector InputV.

6. A method for extracting relations based on syntactic dependency and fusion information according to claim 5, characterized in that The specific implementation process of step (5) is as follows: 5-1 Input the input vector InputV into the LSTM network for encoding, and output the encoded feature vector: output,hidden=LSTM(InputV) (7) Among them, InputV is the concatenated input vector in step 4-1, LSTM is the long short-term memory network, output is the output vector of the last layer of the LSTM network, and hidden is the hidden layer output vector.

7. The method for extracting relations based on syntactic dependency and fusion information according to claim 6, characterized in that The specific implementation process of step (6) is as follows: 6-1 maps the output vector output to another vector V through an LSTM gating unit; 6-2. The V vector is mapped to all relationship categories through the fully connected layer network to obtain the probability distribution of each relationship category, and the relationship category with the highest probability in the probability distribution is selected for prediction; after the current relationship category prediction is completed, the sequence labeling algorithm is used to predict the entity subscript with the highest probability corresponding to the current relationship category, thereby obtaining two entities; 6-3. Use the V vector of this step as the output vector of the next step, and repeat steps 6-1 and 6-2 until all relationship predictions are completed. The implementation formula is as follows: V = lstm_cell (output) (8) Where output is the output vector of formula (7) in step 5-1, lstm_cell is the LSTM gating unit, and V is another vector obtained by mapping; The relationship category prediction formula is as follows: relationType=do_predict(V) (9) Among them, do_predict is the relationship prediction function, relationType is the predicted relationship; The entity prediction formula is as follows: entity=NER(V) (10) Among them, NER is the named entity recognition model, and entity is the obtained entity output; repeat the entity prediction once to obtain two entities of the current relationship category; 6-4 During training, the standard gradient descent algorithm is used to update the network parameters, using the cross entropy loss function and L2 regularization, and the calculation formula is as follows: The overall loss consists of two parts, as shown in formula (11), first the weighted entity loss, then the relation loss; For the overall loss, For physical losses, is the relationship loss, λ is the loss weight of the label; for entity loss The calculation formula is (13), p(tag * |S) is the probability distribution of each tag category tag* of each word in sentence S; taking the negative logarithm of the above probability is the entity loss; for the relationship loss The calculation formula is (12): Represents each relation predicted by S in the sentence The probability distribution of , and then take the negative logarithm sum, where is the predicted value, y is the true value, and t is the prediction to the tth step.

Images