A triple extraction method integrating dependency syntax and pointer generation network

Abstract

The present invention discloses a triple extraction method integrating dependency syntax and pointer generation network. First, the present invention uses Baidu NLP tool to parse sentences, and generalizes some special nouns using the part-of-speech results obtained after sentence parsing. In addition, in order to make full use of the dependency structure information of the sentence, the present invention quotes the KVMN model to fully learn the dependency arcs between words and the information of their dependency relationship categories, and then uses Bi-LSTM to learn the context content, and uses the attention mechanism to calculate the attention distribution and sentence representation vector. Finally, the present invention establishes an extended vocabulary to represent the total set of all words appearing in the original vocabulary and source documents, calculates the probability of each word, and performs word generation. Experimental results show that the present invention can solve the problem of insufficient vocabulary well, and emphasizes the information of dependency structure, has strong applicability, and effectively improves the accuracy of the triple extraction method.

Description

Technical Field

[0001] The present invention relates to the field of information extraction, and in particular to a triple extraction method integrating dependency syntax and pointer generation network. Background Art

[0002] Triple extraction is a classic information extraction task that aims to extract structured information from unstructured text to help people get the information they need faster. An SPO triple consists of three parts: subject, predicate, and object. There may be multiple triples in a text, and they may share one or two components.

[0003] At present, the mainstream triple extraction methods include pipeline methods, joint extraction based on feature learning, and sequence-to-sequence methods (Seq2Seq). The pipeline method divides the entire triple extraction into two steps. It first extracts all entity information from the text through named entity recognition, and then performs classification tasks on each entity pair to obtain relational predicates. This method often encounters the problem of error propagation. The wrong entities identified during the entity recognition process cannot be corrected later. In order to solve this problem, a feature-based joint learning method was proposed. It replaced the artificially constructed features with learned representations and achieved considerable success in triple extraction tasks. However, as the text content becomes more and more complex, the text contains more and more overlapping triples, and its performance gradually becomes ordinary. The problem of overlapping triples directly challenges the sequence labeling scheme and brings great difficulties to the classification of relational predicates.

[0004] Therefore, people applied the sequence-to-sequence method (Seq2Seq) to the triple extraction problem. The Seq2Seq model receives unstructured text as input and decodes triples as a sequence output. This concise approach is also consistent with the human annotation process, that is, the annotator first reads the sentence, understands the meaning of the sentence, and then points out the triples in turn. In addition, there are methods that try to use syntactic information, such as adding a GCN graph network to the Seq2Seq model to use the dependency arc information between words to improve the triple extraction task. Although this type of sequence generation method has achieved satisfactory results, this type of method still has some defects, and the Seq2Seq model cannot avoid the problem of insufficient vocabulary. In addition, in the current methods that use dependency analysis, most of them model different dependencies equally, thereby ignoring the dependency relationship categories between words, and the noise generated will damage the performance of the model to a certain extent. Summary of the invention

[0005] After comprehensively considering the above problems, the present invention proposes a triple extraction method that integrates dependency syntax and pointer generation network to address the problems existing in the prior art.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] Step (1) Use the DDParser tool to parse the input sentence to obtain word segmentation results, part of speech, and syntactic dependency tree information;

[0008] Step (2) generalize the special words in the word segmentation according to the part of speech and word segmentation results, and use the pre-trained model Bert for word embedding, that is, the generalized word segmentation is used as the input of the pre-trained model Bert, and the hidden vector I is output;

[0009] Step (3) Input the syntactic dependency tree obtained in step (1) and the hidden vector I obtained in step (2) into KVMN to learn and obtain the hidden vector II containing dependency structure information, concatenate the hidden vector II and the hidden vector I, and input the concatenated hidden vector into the bidirectional LSTM to capture the context information, thereby obtaining the hidden vector III;

[0010] Step (4) input the hidden vector III into the attention mechanism and calculate the attention distribution and sentence representation vector;

[0011] Step (5) uses the sentence representation vector, the current state of the decoder, and the last generated vocabulary to calculate the probability p of generating a word from the vocabulary or selecting a word from the input sequence;

[0012] Step (6) uses the sentence representation vector and the current state of the decoder to calculate the probability distribution of all words in the vocabulary, uses an extended vocabulary to represent the total set of all words that appear in the vocabulary and the source document, and uses the probability distribution of all words in the vocabulary, the probability p in step (5) and the attention distribution in step (4) to calculate the probability distribution of all words in the extended vocabulary. The word with the highest probability is taken as the next generated word, and it is determined whether the generated word is a terminator. If it is not a terminator, steps (4) to (6) are repeated until the generation is terminated.

[0013] The beneficial effects of the present invention are as follows:

[0014] Based on the characteristics of the triple extraction task, the present invention uses a pointer generation network to allow the entire model to pay more attention to the words in the input sequence at the decoding layer, so that the probability of generating words appearing in the input sequence is higher. First, the present invention uses Baidu NLP tools to parse sentences, and generalizes some special nouns using the part-of-speech results obtained after sentence parsing, so that some special nouns of the same type (time, place, organization, etc.) express the same meaning, weakening the influence of useless word meanings on the entire task, and further highlighting the importance of sentence structure information. In addition, in order to make full use of the dependency structure information of the sentence, the present invention quotes the KVMN model to fully learn the dependency arcs between words and their dependency relationship categories, and then uses Bi-LSTM to learn the context content, and uses the attention mechanism to calculate the attention distribution and sentence representation vector. Finally, the present invention establishes an extended vocabulary to represent the total set of all words appearing in the original vocabulary and source documents, calculates the probability of each word, and generates words. The experimental results show that the present invention can solve the problem of insufficient vocabulary well, and emphasizes the information of dependency structure, has strong applicability, and effectively improves the accuracy of the triple extraction method. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flow chart of the overall implementation scheme of the present invention

[0016] Figure 2 The overall structure diagram of the model of the present invention

[0017] Figure 3 The word segmentation part-of-speech result diagram of the present invention

[0018] Figure 4 Word embedding construction diagram of the present invention

[0019] Figure 5 Dependency structure information fusion graph of the present invention DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings.

[0021] The present invention still uses the encoder-decoder model structure, and regards triple extraction as a sequence generation task. The encoding end uses Bert for word embedding, uses the KVMN model to incorporate syntactic dependency structure information, and uses bidirectional LSTM to learn context information. The decoding end uses a single-layer LSTM structure for decoding. The present invention connects each triple as a target sequence, and the triples are separated by the symbol ";". The three components in the same triple are separated by "," and " <start>"and" <end>" The two labels serve as the start and end symbols of the entire generated sequence.

[0022] like Figure 1 and 2 As shown, a triple extraction method integrating dependency syntax and pointer generation network includes the following steps:

[0023] Step (1) Use the DDParser tool to parse the input sentence to obtain word segmentation results, part of speech, and syntactic dependency tree information;

[0024] Step (2) generalize the special words in the word segmentation according to the part of speech and word segmentation results, and use the pre-trained model Bert for word embedding, that is, the generalized word segmentation is used as the input of the pre-trained model Bert, and the hidden vector I is output;

[0025] Step (3) Input the syntactic dependency tree obtained in step (1) and the hidden vector I obtained in step (2) into KVMN to learn and obtain the hidden vector II containing dependency structure information, concatenate the hidden vector II and the hidden vector I, and input the concatenated hidden vector into the bidirectional LSTM to capture the context information, thereby obtaining the hidden vector III;

[0026] Step (4) input the hidden vector III into the attention mechanism and calculate the attention distribution and sentence representation vector;

[0027] Step (5) uses the sentence representation vector, the current state of the decoder, and the last generated vocabulary to calculate the probability p of generating a word from the vocabulary or selecting a word from the input sequence;

[0028] Step (6) uses the sentence representation vector and the current state of the decoder to calculate the probability distribution of all words in the vocabulary, uses an extended vocabulary to represent the total set of all words that appear in the vocabulary and the source document, and uses the probability distribution of all words in the vocabulary, the probability p in step (5) and the attention distribution in step (4) to calculate the probability distribution of all words in the extended vocabulary. The word with the highest probability is taken as the next generated word, and it is determined whether the generated word is a terminator. If it is not a terminator, steps (4) to (6) are repeated until the generation is terminated.

[0029] Furthermore, the specific implementation process of step (1) is as follows:

[0030] Use DDParser to parse the text sentence and get the result:

[0031] X=(x1,x2,…,x n )(1)

[0032] T(X)=(t1,t2,…,t n )(2)

[0033] D(X)=Dependency_Parser(X) (3)

[0034] Where X represents the sequence after word segmentation. In formula (1), x1, x2, …, x n Indicates the word segmentation result. In formula (2), t1, t2, …, t n Corresponding to x1,x2,…,x in formula (1) n The part-of-speech tagging result of , D(X) is the syntactic dependency tree information.

[0035] Furthermore, the specific implementation process of step (2) is as follows:

[0036] 2-1 Perform general processing on the original sequence X according to the part-of-speech tagging result T(X), Figure 3 is a set of part-of-speech and proper noun category labels. Specifically, the sequence X is generalized into the specified label according to the set rules. The rules are as follows: 1. Replace the words with part-of-speech tagging results of "LOC", "f", and "s" with " <loc>" tag to indicate the meaning of the place; 2. Replace the word with the part-of-speech tag result of "TIME" with " <time>" tag, indicating the meaning of time; 3. Replace the word with the part-of-speech tag result of "PER" with " <per>" tag, indicating the meaning of a person's name; 4. Replace the word with the part-of-speech tag result of "ORG" with " <org>" label, indicating the meaning of the organization; 5. Replace the words with the part-of-speech tagging results of "m" and "q" with " <num>" label, indicating the meaning of the number; 6. Replace the words with the part-of-speech tagging results of "nw" and "nz" with " <pn>" label, indicating the meaning of other proper nouns. Generalization is performed to obtain the generalized sequence X':

[0037] X′=(x′1,x′2,…,x′ n ) (4)

[0038] Among them, x′1,x′2,…,x′ n Indicates the generalized vocabulary;

[0039] 2-2 As Figure 4 As shown in the figure, for the generalized sequence X′, Bert's embedding structure is used for word embedding. Bert first sends the sequence X′ to the Token Embedding layer to convert each word into a vector form, and then sends the sequence X′ to the Position Embedding layer to obtain the sequential features of each word. Finally, the two results are concatenated and input into the Bert model to obtain the final word embedding, and the output hidden vector I is obtained;

[0040] The word embedding process can be expressed as follows:

[0041] H = BERT(X′) = {h1,h2,…,h n } (5)

[0042] Among them, H is the set of hidden vectors I of the output, h1,h2,…,h n is the hidden vector I.

[0043] Furthermore, the specific implementation process of step (3) is as follows:

[0044] 3-1 Process the dependency analysis tree D(X):

[0045] For each word x in the sequence X i Construct a key-value pair, using the word connected to the word in the dependency tree D(X) as the key, and the dependency syntactic relationship type between the two words as the value, and finally get the key sequence K i ={k i,1 ,k i,2 ,…,k i,q }, the sequence of values ​​V i = {v i,1 ,v i,2 ,…,v i,q }, where q represents the same i The number of directly associated nodes. Figure 5 As shown in the figure, taking the word "create" as an example, analyzing the dependency tree can get the key sequence "XXX", "1999", "XXX Co., Ltd." and the corresponding value sequence "SBV", "ADV", "VOB". In KVMN, using x i The hidden vector h obtained after BERT i The word embedding of the key in each key-value pair The inner product calculates the weight of the corresponding dependency. The detailed calculation process is as follows:

[0046]

[0047] Next, the model adds weights p i,j Applied to the corresponding value embedding The weighted average of the value embedding is obtained as the hidden vector II of the entire KVMN output. The calculation process is as follows:

[0048]

[0049] 3-2 KVNM outputs the vector o with syntactic structure information i and the hidden layer vector h output in BERT i Concatenate them, and finally input these concatenated vectors one by one into the bidirectional LSTM to capture context information and generate a series of encoded hidden states. The process can be expressed as follows:

[0050] H BiLSTM ={h′1,h′2,…,h′ n } (8)

[0051] Among them, H BiLSTM represents the set of hidden vectors III, h′1,h′2,…,h′ n represents the hidden vector III.

[0052] Furthermore, the specific implementation process of step (4) is as follows:

[0053] 4-1 Calculate attention distribution:

[0054] Considering the attention distribution as the probability distribution over the source word sequence X, telling the decoder where to generate the next word, the process can be expressed as the following formula:

[0055]

[0056] a t =softmax(e t ) (10)

[0057] where v T , W h , W s and b are learnable parameters, s t is the current decoder state, a t represents the attention distribution, e t Represents the sequence X and the encoder state s t The set of alignment scores, Represents the i-th word in sequence X and the encoder state s t The alignment score of

[0058] 4-2 The attention distribution is used to generate the weighted sum of the encoder hidden state to generate the sentence vector. The specific process is as follows:

[0059]

[0060] in, represents the attention weight.

[0061] Furthermore, the specific implementation process of step (5) is as follows:

[0062] Calculate the generation probability and use the generation probability to select the generation probability p of sampling a generated word from the vocabulary or the probability 1-p of copying a word from the input sequence in the attention distribution. This process is recorded as follows:

[0063]

[0064] in and b pt are all learnable parameters, σ is the sigmoid function, s t is the current state of the decoder, x t is the decoder input.

[0065] Furthermore, the specific implementation process of step (6) is as follows:

[0066] 6-1 Representing sentences as vectors As a fixed-size representation of the content read from the source word sequence X, the sentence representation vector With the decoder state s t Connect them and then send them to two linear layers to generate the vocabulary distribution. The process is as follows:

[0067]

[0068] Where V′, V, b1, b2 are all learnable parameters, and the final generated P voc is the probability distribution of all words in the vocabulary;

[0069] 6-2 uses an extended vocabulary to represent the union of all words in the vocabulary and the source text, and finally obtains the following probability distribution on the extended vocabulary:

[0070]

[0071] Take the word with the highest probability as the current generated word and determine whether the generated word is a terminator" <end>", if not, repeat steps (4) to (6) until termination.

[0072] During training, the loss function at each step is calculated using the negative log-likelihood of the target word:

[0073]

[0074] The loss function of the entire text sequence is:

[0075]

[0076] Among them, loss t Represents the loss value of the tth step, and T represents the number of steps from the beginning to the end to generate the terminal symbol.< / end> < / pn> < / num> < / org> < / per> < / time> < / loc> < / end> < / start>

Claims

1. A triple extraction method integrating dependency syntax and pointer generation network, characterized in that The steps include: Step (1) Use the DDParser tool to parse the input sentence to obtain word segmentation results, part of speech, and syntactic dependency tree information; Step (2) generalize the special words in the word segmentation according to the part of speech and word segmentation results, and use the pre-trained model Bert for word embedding, that is, the generalized word segmentation is used as the input of the pre-trained model Bert, and the hidden vector I is output; Step (3) Input the syntactic dependency tree obtained in step (1) and the hidden vector I obtained in step (2) into KVMN to learn and obtain the hidden vector II containing dependency structure information, concatenate the hidden vector II and the hidden vector I, and input the concatenated hidden vector into the bidirectional LSTM to capture the context information, thereby obtaining the hidden vector III; Step (4) input the hidden vector III into the attention mechanism and calculate the attention distribution and sentence representation vector; Step (5) uses the sentence representation vector, the current state of the decoder, and the last generated word to calculate the probability p of generating a word from the vocabulary or selecting a word from the input sequence; Step (6) uses the sentence representation vector and the current state of the decoder to calculate the probability distribution of all words in the vocabulary, uses an extended vocabulary to represent the total set of all words that appear in the vocabulary and the source document, uses the probability distribution of all words in the vocabulary, the probability p in step (5) and the attention distribution in step (4) to calculate the probability distribution of all words in the extended vocabulary, takes the word with the highest probability as the next generated word, and determines whether the generated word is a terminator. If it is not a terminator, repeats steps (4) to (6) until the generation is terminated; The specific implementation process of step (4) is as follows: 4-1 Calculate attention distribution: The attention distribution is regarded as the probability distribution over the source word sequence X, telling the decoder where to generate the next word. The process is expressed as the following formula: the t =softmax(e t ) (10) where v T , W h , W s and b are learnable parameters, s t is the current decoder state, a t represents the attention distribution, e t Represents the sequence X and the encoder state s t The set of alignment scores, Represents the i-th word in sequence X and the encoder state s t The alignment score of 4-2 Use the attention distribution to generate the weighted sum of the encoder hidden state to generate the sentence representation vector The specific process is as follows: in, represents the attention weight; The specific implementation process of step (5) is as follows: Calculate the generation probability and use the generation probability to select the generation probability p of sampling a generated word from the vocabulary or the probability 1-p of copying a word from the input sequence in the attention distribution. This process is recorded as follows: in and b pt are all learnable parameters, σ is the sigmoid function, s t is the current state of the decoder, x t is the decoder input; The specific implementation process of step (6) is as follows: 6-1 Representing sentences as vectors As a fixed-size representation of the content read from the source word sequence X, the sentence representation vector With the decoder state s t Connect them and then send them to two linear layers to generate the vocabulary distribution. The process is as follows: Where V', V, b1, b2 are all learnable parameters, and the final generated P voc is the probability distribution of all words in the vocabulary; 6-2 uses an extended vocabulary to represent the union of all words in the vocabulary and the source text, and finally obtains the following probability distribution P(w) on the extended vocabulary: Take the word with the highest probability as the current generated word and determine whether the generated word is a terminator" <end> ", if not, repeat steps (4) to (6) until termination;< / end> During training, the loss function at each step is calculated using the negative log-likelihood of the target word: The loss function of the entire text sequence is: Among them, loss t Represents the loss value of the tth step, and T represents the number of steps from the beginning to the end to generate the terminal symbol.

2. According to the triple extraction method integrating dependency syntax and pointer generation network according to claim 1, the specific implementation process of step (1) is as follows: Use DDParser to parse the text sentence and get the result: X=(x1,x2,…,x n ) (1) T(X)=(t1,t2,…,t n ) (2) D(X)=Dependency_Parser(X) (3) in, X represents the sequence after word segmentation. In formula (1), x1, x2, …, x n Indicates the word segmentation result. In formula (2), t1, t2, …, t n Corresponding to x1,x2,…,x in formula (1) n The part-of-speech tagging result of , D(X) is the syntactic dependency tree information.

3. A triple extraction method integrating dependency syntax and pointer generation network according to claim 2, characterized in that The specific implementation process of step (2) is as follows: 2-1 According to the part-of-speech tagging result T(X), the sequence X is generalized into the specified label according to the set rules. The rules are as follows:

1. Replace the words with the part-of-speech tagging results of "LOC", "f", and "s" with " <loc>" tag to indicate the meaning of the place; 2. Replace the word with the part-of-speech tag result of "TIME" with " <time>" tag, indicating the meaning of time; 3. Replace the word with the part-of-speech tag result of "PER" with " <per>" tag, indicating the meaning of a person's name; 4. Replace the word with the part-of-speech tag result of "ORG" with " <org>" label, indicating the meaning of the organization; 5. Replace the words with the part-of-speech tagging results of "m" and "q" with " <num>" label, indicating the meaning of the number; 6. Replace the words with the part-of-speech tagging results of "nw" and "nz" with " <pn> " label, indicating the meaning of other proper nouns; generalization is performed to obtain the generalized sequence X':< / pn> < / num> < / org> < / per> < / time> < / loc> X'=(x'1,x'2,…,x' n ) (4) where x'1, x'2, ..., x' n Indicates the generalized vocabulary; 2-2 For the generalized sequence X', use Bert's embedding structure to do word embedding processing; Bert first sends the sequence X' to the Token Embedding layer to convert each word into a vector form, and at the same time sends the sequence X' to the PositionEmbedding layer to obtain the sequential features of each word. Finally, the vector and the corresponding sequential features are concatenated and input into the Bert model to obtain the final word embedding, and the output hidden vector I is obtained; The word embedding process is expressed as follows: H=BERT(X')={h1,h2,…,h n } (5) Among them, H is the set of hidden vectors I of the output, h1,h2,…,h n is the hidden vector I.

4. A triple extraction method integrating dependency syntax and pointer generation network according to claim 3, characterized in that The specific implementation process of step (3) is as follows: 3-1 Process the dependency analysis tree D(X): For each word x in the sequence X i Construct a key-value pair and add the key value in the dependency analysis tree D(X) to the word x i The connected words are used as keys, and the dependency syntactic relationship type between the two words is used as the value, and finally the key sequence K is obtained. i ={k i,1 ,k i,2 ,…,k i,q }, the sequence of values ​​V i = {v i,1 ,v i,2 ,…,v i,q }, where q represents the same i The number of directly associated nodes; in KVMN, use x i The hidden vector h obtained after BERT i The word embedding of the key in each key-value pair The inner product calculates the weight p of the corresponding dependency i,j , the detailed calculation process is as follows: Next, the model adds weights p i,j Applied to the corresponding value embedding The weighted average of the value embedding is obtained as the hidden vector II of the entire KVMN output. The calculation process is as follows: 3-2 The hidden vector o with syntactic structure information output by KVNM i and the hidden vector h output in BERT i Concatenate them, and finally input these concatenated vectors one by one into the bidirectional LSTM to capture context information, generating a series of encoded hidden states. The process is expressed as follows: H BiLSTM ={h'1,h'2,…,h' n } (8) Among them, H BiLSTM represents the set of hidden vectors III, h'1,h'2,…,h' n represents the hidden vector III.

Images