An open domain corpus relation joint extraction method

Abstract

An open domain corpus relationship joint extraction method comprises the following steps: S1, extracting the feature vector of characters in the corpus; S2, performing feature fusion in the graph attention network; S3, extracting the relationship phrase in the corpus; S4, extracting the entity pair phrase in the corpus; S5, according to the relationship phrase extracted in step S3 and the corresponding entity pair phrase extracted in step S4, the three tuples are formed, and the confidence of the three tuples is determined, if the confidence is greater than or equal to the set confidence threshold, then the three tuples are taken as the open domain relationship three tuples of the input corpus. Through the above scheme, the problems of redundant relationship three tuple sequence, overlapping relationship three tuple, and low relationship three tuple extraction accuracy in open domain relationship extraction are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to an open domain corpus relation joint extraction method. BACKGROUND

[0002] With the rapid development of Internet technology, information technology produces a large amount of unstructured data, and information extraction technology has been widely used to extract structured and useful data information from unstructured data. Unlike traditional information extraction technology, open domain relation extraction does not need to define relation types, and in addition, it is not limited by specific domain data sets, and can extract relation triples from unstructured data, and is better applied to information extraction in data conversion, scene switching and the like. With the in-depth research, the open domain relation extraction technology based on deep learning has gradually become the mainstream trend.

[0003] At present, the open domain relation extraction technology based on deep learning mainly adopts the method of sequence labeling, extraction or generation, and these methods divide the process of extracting relation triples into two steps, first extracting the relation phrase in the corpus, and then extracting the possible entity pair phrase according to the extracted relation phrase result. The traditional open domain relation extraction method mainly has three problems: (1) the extraction of entity pair phrases and the extraction of relation phrases are divided into two sub-tasks subjectively, ignoring the internal semantic connection and the dependency relationship on the syntax structure between the two, which is easy to cause error propagation, resulting in low accuracy of open domain relation extraction result; (2) ignoring the problem of entity phrase overlap in which the entity phrase exists in multiple relation triples, resulting in insufficient semantics of open domain relation extraction result, affecting the extraction accuracy; (3) not fully considering the dependency relationship between the words in the corpus, and part of the relation triple sequence is redundant, resulting in error accumulation, affecting the accuracy and efficiency of open domain relation extraction.

[0004] In view of the problems existing in the above-mentioned traditional open domain relation extraction technology, the present application innovatively proposes an open domain relation joint extraction method based on a multi-link graph attention network, adopts an entity pair phrase and relation phrase joint learning mode, and realizes joint extraction of open domain relation triples. In addition, the graph attention network is used to fuse the dependency relationship between the characters in the corpus, and through the multi-link decoding triple mechanism, the problems of long and redundant relation triples are solved. SUMMARY

[0005] The present application provides an open domain corpus relation joint extraction method to solve the problems of redundant relation triple sequence, overlapping relation triples and low relation triple extraction accuracy in open domain relation extraction.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical scheme:

[0007] A method for joint extraction of relations from open-domain corpora includes the following steps:

[0008] S1. Extract feature vectors of characters in the corpus: Input the corpus into the BERT pre-trained language model to encode the corpus and obtain the feature vectors of characters in the corpus;

[0009] S2. Feature fusion in graph attention network: Based on graph attention network, the features contained in the feature vector of characters are fused and the dependencies between characters are learned;

[0010] S3. Extracting relational phrases from the corpus: Extracting relational phrases from the corpus by designing a relational phrase sequence labeling model, where the meaning of the relational phrase is the predicate part of the sentence in the corpus;

[0011] S4. Extract entity pairs from the corpus: Since the meaning of a relational phrase is the predicate part of a sentence in the corpus, each relational phrase has a corresponding entity pair, which includes a first entity phrase and a last entity phrase. The meaning of the first entity phrase is the subject of the sentence in the corpus, and the meaning of the last entity phrase is the object of the sentence in the corpus. Based on the character dependencies in step S2 and the entity pairs predicted by the graph attention network for each relational phrase, the entity pairs are extracted.

[0012] S5. Based on the relational phrases extracted in step S3 and the corresponding entity pair phrases extracted in step S4, form them into triples and determine the confidence level of the triples. If the confidence level is greater than or equal to the set confidence level threshold, then the triples are used as open domain relational triples of the input corpus.

[0013] To optimize the above technical solution, the specific measures also include:

[0014] Furthermore, the specific calculation formula for the learned dependencies between characters in step S2 is as follows:

[0015]

[0016] In the formula, This represents the hidden state vector of the i-th character at layer t, which reflects the dependencies between characters; Let M represent the hidden state vector of the i-th character in layer (t-1), σ represent the sigmoid activation function, n represent the length of the input corpus, and M represent the hidden state vector of the i-th character in layer (t-1). ij W is represented as the dependency weights in a graph attention network. t and b t These are the parameter matrix and bias vector of the graph attention network, respectively.

[0017] Furthermore, the specific content of step S3 is as follows:

[0018] S3.1 Design a relational phrase sequence labeling model to calculate the probability that a certain character is the start or end position of a relational phrase. The specific calculation formula of this model is as follows:

[0019]

[0020]

[0021] In the formula, and Let x represent the probabilities that the i-th character in the input corpus sequence is the start and end position of a relational phrase, respectively. i W represents the encoded sequence of the i-th word. start W represents the weight of the starting position of the relational phrase. end b represents the weight of the ending position of the relational phrase. start b represents the deviation from the starting position of the relational phrase. end σ represents the deviation in calculating the end position of the relational phrase, and σ represents the sigmoid activation function.

[0022] S3.2, the calculation in step S3.1 and If the probability value is greater than the probability threshold, the position is set to 1; otherwise, it is set to 0. This determines the position of the relational phrase and enables the extraction of relational phrases from the corpus.

[0023] Furthermore, the specific content of step S4 is as follows: Step S4 involves extracting the entity pairs corresponding to each relation phrase based on the character dependencies identified in step S2 and by predicting them using a graph attention network.

[0024] S4.1. Let the index of a certain relation phrase extracted in step S3 in the relation phrase set be λ, and let it be represented as a vector h through relation embedding. λ ;

[0025] S4.2, Transfer vector h λ and the hidden state vector obtained in step S2 The features are input together into the decoder and processed by a graph attention network to obtain the decoder output that fuses the features of the convolutional layers.

[0026] S4.3. Input the output of step S4.2 into the prediction layer of the graph attention network to predict the first and last entity phrases in the entity pair and extract them.

[0027] Furthermore, the specific calculation formula for "predicting and extracting the first and last entity phrases in the entity pair phrase" in step S4.3 is as follows:

[0028] The formula for calculating the position of the extracted first entity phrase is as follows:

[0029]

[0030]

[0031] In the formula, This represents the probability value of the starting position of the first entity phrase extracted. This represents the probability value of the end position of the first entity phrase extracted. This indicates the weight of the starting position of the first entity phrase. This indicates the weight of the end position of the first entity phrase. This indicates the deviation of the starting position of the extracted first entity phrase. h represents the deviation of the end position of the extracted first entity phrase. G This represents the contextual features processed by the graph attention network;

[0032] The formula for calculating the position of the extracted tail entity phrase is as follows:

[0033]

[0034]

[0035] In the formula, This represents the probability value of the starting position of the extracted tail entity phrase. This represents the probability value of the end position of the extracted tail entity phrase. This indicates the weight of the starting position of the extracted tail entity phrase. This indicates the weight of the end position of the extracted tail entity phrase. This indicates the deviation of the starting position of the extracted tail entity phrase. h represents the deviation of the end position of the extracted tail entity phrase. G This represents the contextual features processed by the graph attention network.

[0036] Furthermore, the confidence threshold in step S5 is 0.8.

[0037] The beneficial effects of this invention are as follows: This invention employs a joint extraction method of relational phrases and entity pair phrases to establish the inherent deep semantic dependency relationship between entity pair phrases and relational phrases. Furthermore, by fusing character-level feature connections in the corpus through a multi-link graph attention network, it better addresses issues such as overlapping entity phrases and redundant relation triple sequences, thereby improving the semantic reliability and simplicity of the open-domain relation extraction results. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall framework of the present invention.

[0039] Figure 2 This is a flowchart illustrating the overall extraction scheme of the present invention. Detailed Implementation

[0040] The invention will now be described in further detail with reference to the accompanying drawings.

[0041] refer to Figure 1 The overall technical solution of this application includes:

[0042] Step 1. Corpus Feature Extraction. The corpus is input into an existing BERT pre-trained language model to encode the corpus, extract character-level features, and obtain the feature vector of the corpus.

[0043] h0 = SW s +W p (1)

[0044] hα=Trans(h α-1 ),α∈[1,N] (2)

[0045] Where S represents the one-hot vector matrix of words in the input sentence, and W s Representative word embedding matrix, W p represents the position embedding matrix, where p represents the position index in the input sentence; hα represents the hidden state vector, i.e., the context representation of the input sentence at layer α; and N represents the number of Transformer blocks.

[0046] In this context, formula (1) represents the feature vector extracted from the corpus; formula (2) represents the hidden state vector; and h0 represents the obtained corpus feature vector. Formulas (1) and (2) are specific manifestations of BERT's extraction of corpus feature vectors. BERT is a general model.

[0047] Step 2. Feature Fusion using Graph Attention Network. Character-level features from the corpus are fused using a graph attention network to learn the dependencies between characters in the corpus.

[0048]

[0049] Among them, M ij W represents the dependency weights of the input graph attention network. t and b t These are the parameter matrix and bias vector of the graph convolutional neural network, respectively. When t = 0, This represents the initial hidden layer vector of the i-th node before the graph convolution operation.

[0050] in, The vector represents the hidden state of the i-th character at layer t, reflecting the character dependencies between corpora; n represents the length of the input corpus. M represents the hidden state vector of the i-th character in layer t-1. ij This represents the dependency weights of the graph attention network, which are the dependency weight matrix between characters in the corpus.

[0051] Step 3. Relational phrase extraction. Using a relational phrase sequence labeling model, extract the relational phrases that may exist in the corpus.

[0052]

[0053]

[0054] in, ... i Let W represent the encoded sequence of the i-th word, W represent the weight, b represent the bias, and σ represent the sigmoid activation function.

[0055] Step 4. Entity Pair Phrase Extraction. Through multi-link decoding, the start and end positions of the entity pair phrases corresponding to each relational phrase are predicted, and then the entity pair phrases corresponding to each relational phrase are decoded (relational phrases refer to phrases in the corpus that can represent entity pair relations, such as verbs, verb phrases, etc.; they can also be understood as the predicate in a simple sentence; entity pair phrases refer to noun phrases in the corpus that have a relationship of dependence; they can also be understood as the subject and object in a sentence).

[0056] Suppose that a relation phrase was extracted in the previous stage, and its index in the relation set is λ, which is represented as a vector h through relation embedding. λ The latent vectors in (3) are input into the decoder, and the decoder output, which fuses the features of the convolutional layer, is obtained through the graph attention mechanism. Finally, it is input into the prediction layer for the first and last positions of the first entity phrase. The formula for extracting the start and end positions of the first entity phrase is as follows:

[0057] o s ,h s =BERT decoder (h λ ,h t (6)

[0058] o,A s =Attention(o s ,h G (7)

[0059]

[0060]

[0061] Among them, o s ,h s h represents the decoded graph attention network feature output and latent vector output. G Let W represent the contextual features passed through the graph attention network, where W represents the weights and b represents the biases. and represents the probability value of the starting position of the extracted first entity phrase, and o represents the decoder output of the convolutional layer features; A s This represents the parameters of the attention mechanism.

[0062] The formula for extracting the start and end positions of the last entity phrase is as follows:

[0063]

[0064]

[0065] in, and These represent the probability values ​​of the start and end positions of the extracted tail entity phrases.

[0066] Step 5. Select relation triples. Based on the confidence of the extracted relation triples, filter the candidate relation triple set of the corpus, and select relation triples with a confidence greater than the threshold of 0.8 as the open domain relation triple extraction results of the corpus.

[0067] refer to Figure 2 The following is a specific example:

[0068] Given the input corpus s = {Dr. Lü ab graduated from CDE University of Technology and is currently a senior researcher in the department}

[0069] Step 1. Encode the corpus information and extract the feature information of the input corpus. The feature vector of the corpus is extracted using the BERT pre-trained model as [0.50451,0.68607,-0.59517,-0.022801,0.60046,-0.13498,-0.08813,0.47377,-0.61798,-0.31012,-0.076666,1.493,-0.034189,-0.98173,0.68229,0.81722,-0.51874,-0.31503,-0.55809,0.66421,0.1961,-0.13495,-0.11476,-0.30344].

[0070] Step 2. Graph attention network feature fusion. The feature vector obtained by encoding the input corpus by BERT is passed through a graph convolutional neural network to obtain the character fusion feature of the corpus as [0.41177, -2.223, -1.0756, -1.0783, -0.34354, 0.33505, 1.9927, -0.04234, -0.64319, 0.71125, 0.49159, 0.16754, 0.34344, -0.25663, -0.8523, 0.1661, 0.40102, 1.1685, -1.0137, -0.21585, -0.15155, 0.78321, -0.91241, -1.6106, -0.64426, -0.51042];

[0071] Step 3. Relationship phrase extraction. Extract the possible relationship phrases in the input corpus, and the result is {graduated from, current position, holds the position of};

[0072] Step 4. Entity pair phrase extraction. Through multi-link decoding, extract the entity pair phrases corresponding to the relationship phrase result set respectively, and obtain the candidate relationship triple set of the input corpus as {(Lv ab, graduated from, cde Industrial University), (Lv ab, current position, Senior Researcher of the department), (Lv ab, holds the position of, Senior Researcher of the department), (Lv ab, holds the position of, Researcher), (Lv ab, graduated from, department)};

[0073] Step 5. Select relationship triples. Screen the candidate relationship triple set of the input corpus according to the confidence. For the candidate relationship triple set obtained from the input corpus (Lv ab, graduated from, cde Industrial University), (Lv ab, current position, Senior Researcher of the department), (Lv ab, holds the position of, Senior Researcher of the department), (Lv ab, holds the position of, Researcher), (Lv ab, graduated from, department), the corresponding confidences are 0.82, 0.86, 0.74, 0.48, 0.33 respectively. Select the relationship triples (Lv ab, graduated from, cde Industrial University), (Lv ab, current position, Senior Researcher of the department) with a confidence greater than the threshold of 0.8 as the domain relationship triples of the input corpus.

[0074] It should be noted that the terms such as "up", "down", "left", "right", "front", "back", etc. cited in the invention are only for the convenience of clear description, rather than to limit the scope of implementation of the present invention. The change or adjustment of their relative relationship, without substantial change in the technical content, should also be regarded as the scope of implementation of the present invention.

[0075] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for joint extraction of relations from open-domain corpora, characterized in that, Includes the following steps: S1. Extract feature vectors of characters in the corpus: Input the corpus into the BERT pre-trained language model to encode the corpus and obtain the feature vectors of characters in the corpus; h0=SW s +W p (1) hα=Trans(h α-1 ),α∈[1,N](2) Where S represents the one-hot vector matrix of words in the input sentence, and W s Representative word embedding matrix, W p represents the position embedding matrix, where p represents the position index in the input sentence; hα represents the hidden state vector, i.e., the context representation of the input sentence at layer α; and N represents the number of Transformer blocks. Wherein, formula (1) represents the feature vector extracted from the corpus; formula (2) represents the hidden state vector; h0 represents the obtained corpus feature vector; formula (1) and formula (2) are the specific manifestations of BERT extracting corpus feature vectors; BERT is a general model. S2. Feature fusion in graph attention network: Based on graph attention network, the features contained in the feature vector of characters are fused and the dependencies between characters are learned; S3. Extracting relational phrases from the corpus: Extracting relational phrases from the corpus by designing a relational phrase sequence labeling model, where the meaning of the relational phrase is the predicate part of the sentence in the corpus; S4. Extract entity pairs from the corpus: Since the meaning of a relational phrase is the predicate part of a sentence in the corpus, each relational phrase has a corresponding entity pair, which includes a first entity phrase and a last entity phrase. The meaning of the first entity phrase is the subject of the sentence in the corpus, and the meaning of the last entity phrase is the object of the sentence in the corpus. Based on the character dependencies in step S2 and the entity pairs predicted by the graph attention network for each relational phrase, the entity pairs are extracted. S5. Based on the relational phrases extracted in step S3 and the corresponding entity pair phrases extracted in step S4, form them into triples and determine the confidence level of the triples. If the confidence level is greater than or equal to the set confidence level threshold, then the triples are used as open domain relational triples of the input corpus.

2. The method for joint extraction of relations from open-domain corpora according to claim 1, characterized in that, The specific formula for calculating the dependency relationship between the learned characters in step S2 is as follows: In the formula, This represents the hidden state vector of the i-th character at layer t, which reflects the dependencies between characters; Let M represent the hidden state vector of the i-th character in layer (t-1), σ represent the sigmoid activation function, n represent the length of the input corpus, and M represent the hidden state vector of the i-th character in layer (t-1). ij W is represented as the dependency weights in a graph attention network. t and b t These are the parameter matrix and bias vector of the graph attention network, respectively.

3. The method for joint extraction of relations from open-domain corpora according to claim 2, characterized in that, The specific content of step S3 is as follows: S3.1 Design a relational phrase sequence labeling model to calculate the probability that a certain character is the start or end position of a relational phrase. The specific calculation formula of this model is as follows: In the formula, and Let x represent the probabilities that the i-th character in the input corpus sequence is the start and end position of a relational phrase, respectively. i W represents the encoded sequence of the i-th word. start W represents the weight of the starting position of the relational phrase. end b represents the weight of the ending position of the relational phrase. start b represents the deviation from the starting position of the relational phrase. end σ represents the deviation in calculating the end position of the relational phrase, and σ represents the sigmoid activation function. S3.2, the calculation in step S3.1 and If the probability value is greater than the probability threshold, the position is set to 1; otherwise, it is set to 0. This determines the position of the relational phrase and enables the extraction of relational phrases from the corpus.

4. The method for joint extraction of relations from an open-domain corpus according to claim 3, characterized in that, The specific content of step S4 is as follows: Step S4 involves extracting the entity pairs corresponding to each relation phrase based on the character dependencies in step S2 and predicting them using a graph attention network. S4.

1. Let the index of a certain relation phrase extracted in step S3 in the relation phrase set be λ, and let it be represented as a vector h through relation embedding. λ ; S4.2, Transfer vector h λ and the hidden state vector obtained in step S2 The features are input together into the decoder and processed by a graph attention network to obtain the decoder output that fuses the features of the convolutional layers. S4.

3. Input the output of step S4.2 into the prediction layer of the graph attention network to predict the first and last entity phrases in the entity pair and extract them.

5. The method for joint extraction of relations from open-domain corpora according to claim 4, characterized in that, The specific calculation formula for "predicting and extracting the first and last entity phrases in the entity pair phrase" in step S4.3 is as follows: The formula for calculating the position of the extracted first entity phrase is as follows: In the formula, This represents the probability value of the starting position of the first entity phrase extracted. This represents the probability value of the end position of the first entity phrase extracted. This indicates the weight of the starting position of the first entity phrase. This indicates the weight of the end position of the first entity phrase. This indicates the deviation of the starting position of the extracted first entity phrase. h represents the deviation of the end position of the extracted first entity phrase. G This represents the contextual features processed by the graph attention network; The formula for calculating the position of the extracted tail entity phrase is as follows: In the formula, This represents the probability value of the starting position of the extracted tail entity phrase. This represents the probability value of the end position of the extracted tail entity phrase. This indicates the weight of the starting position of the extracted tail entity phrase. This indicates the weight of the end position of the extracted tail entity phrase. This indicates the deviation of the starting position of the extracted tail entity phrase. h represents the deviation of the end position of the extracted tail entity phrase. G This represents the contextual features processed by the graph attention network.

6. The method for joint extraction of relations from an open-domain corpus according to claim 1, characterized in that, The confidence threshold mentioned in step S5 is 0.8.

Images