Entity relation extraction method and device based on new relation attention mechanism

Abstract

The present invention provides an entity relationship extraction method based on a novel relationship attention mechanism, comprising the following steps: extracting a semantic vector of a sentence using a PCNN model; extracting a feature vector of the sentence package using a NATT attention mechanism according to the semantic vector of each sentence in the sentence package; inputting the feature vector of the sentence package into a sofmax classifier to obtain the confidence of the relationship between the semantic vector of each sentence in the sentence package and the relationship vector. The present invention provides the beneficial effect of improving the accuracy of the relationship vector representation.

Description

Technical Field

[0001] The present invention relates to the field of information extraction technology, and in particular to an entity relationship extraction method based on a novel relationship attention mechanism. Background Art

[0002] In recent years, the explosive growth of Internet information has brought great difficulties to people in obtaining the information they need quickly and accurately. Information extraction technology can conduct in-depth analysis of texts, thereby providing users with more accurate information services. Relation extraction is one of the main tasks of information extraction, which refers to the semantic relationship classification of entity pairs contained in texts. It plays an important role in the fields of intelligent question answering, knowledge base construction, etc.

[0003] At present, the most commonly used method for entity relationship extraction is the supervised method. Although its extraction effect is obvious and its accuracy is high, this method requires a large amount of manually annotated corpus, which is time-consuming and labor-intensive, and limits large-scale entity relationship extraction. In response to the problem of corpus, Mintz et al. tried to use the FreeBase knowledge base to replace the manually annotated corpus. They used Freebase to align with free text to obtain a large amount of annotated training corpus. Mintz called this supervision method remote supervision. Its idea is based on an assumption: if there is a certain relationship between two entities in the knowledge base, then the sentences containing these two entities more or less express this relationship. Obviously, this assumption is not rigorous and introduces a lot of noisy annotated data. How to reduce noise has become a problem that we need to solve further.

[0004] The prior art proposes a segmented PCNN network to extract the semantic vector of a sentence, query the relationship vector corresponding to the label from the relationship matrix, and then use it to calculate the similarity with the sentence. The higher the similarity, the higher the weight, and vice versa. The PCNN network structure is divided into an input layer, a convolution layer, and a pooling layer.

[0005] There is also a method based on deep learning. This method uses the deep learning model PCNN network to extract the semantic vector of each sentence, and then uses the entity background knowledge in the Freebase knowledge base to train the entity vector. Finally, the relationship vector is obtained by subtracting the entities, and the weight of each sentence corresponding to this relationship is calculated. If the sentence expresses this relationship, the higher the weight, otherwise, the lower it is.

[0006] Both of the above methods use the relationship vector and PCNN network to extract the relationship. The relationship vector obtained by them indicates that the relationship semantics is inaccurate. For example, Snape looked at Lily affectionately. ---- Lovers. The subtraction of the two entities cannot get the relationship of lovers. This is because lovers are embodied by "affectionate". Summary of the invention

[0007] In view of this, the present invention provides an entity relationship extraction method based on a novel relationship attention mechanism, comprising the following:

[0008] S101: Acquire labeled data; the labeled data is specifically labeled sentence packages;

[0009] S102: extracting a semantic vector of each sentence in the labeled sentence bag using a PCNN model; the PCNN model includes an input layer, a convolution layer, and a pooling layer;

[0010] S103: extracting a feature vector of the labeled sentence bag using a NATT attention mechanism according to the semantic vector of each sentence in the labeled sentence bag;

[0011] S104: Input the feature vector of the labeled sentence bag into the sofmax classifier to obtain the confidence of the relationship between the semantic vector and the relationship vector of each sentence in the labeled sentence bag; the relationship vector is specifically an initialization vector randomly generated according to the original vector of the labeled sentence bag.

[0012] Furthermore, in step S101, labeled data is obtained, specifically in the following manner: a labeled data set is obtained by automatically aligning a remote knowledge base using a remote supervision method.

[0013] Furthermore, in step S102, the semantic vector of each sentence in the labeled sentence bag is extracted using the PCNN model, and the specific steps are as follows:

[0014] S201: Any sentence s in the labeled sentence bag includes n words, that is, s = (w1, w2, ..., w n ) Among them, w i is the i-th word in sentence s; sentence s is mapped to the word vector table, and its word w i is converted into word vector e i , that is, s=(w1,w2,...,w n ) is converted into s = (e1, e2, ..., e n );

[0015] S202: For word w in sentence s i , calculate the relative distances to the head entity and the tail entity in sentence s, and then map these two relative distances into words w i Position vector d i ;

[0016] S203: word w i The word vector and position vector of the word w are concatenated to get the word w iThe feature vector of each word in sentence s is used as the input of the input layer of the PCNN model;

[0017] S204: Use the convolution layer of the PCNN model to perform a convolution operation on the feature vector of each word in the sentence s to obtain a result vector c of each word, as shown in formula (1):

[0018]

[0019] In formula (1), c i is the result vector of the i-th word after the convolution operation; W i is the i-th weight matrix, b is the preset bias value;

[0020] S205: Split the result vector c of each word into three parts, specifically c i ={c i,1 ,c i,2 ,c i,3}, use the pooling layer of the PCNN model to perform maximum pooling on each part and obtain the maximum pooling vector p i ={p i,1 ,p i,2 ,p i,3}; where p i,j =max(c i,j ), j = 1, 2, 3;

[0021] S206: Obtain the semantic vector S = tanh(p) of sentence s according to the maximum pooling vector, where p = (p1, p2, ..., p n ).

[0022] Furthermore, in step S103, according to the semantic vector of each sentence in the labeled sentence bag, the NATT attention mechanism is used to extract the feature vector of the labeled sentence bag, specifically:

[0023] S301: Use the score function to calculate the attention weight of each sentence semantic vector and relationship vector, as shown in formula (2):

[0024]

[0025] In formula (2), i represents the number of the sentence in the labeled sentence bag, m is the total number of sentences in the labeled sentence bag, and j represents the score sequence number of each sentence in the score function; a i is the sentence s in the labeled sentence bag i The attention weight between the relation vector and w i The expression of is shown in formula (3):

[0026]

[0027] In formula (3), b a is the preset offset; is the intermediate matrix; r c is the relationship vector; S i is the i-th sentence s in the labeled sentence bag i The semantic vector S i ; r c The expression is r c =(e1-e2)M r ; e1, e2 are two entity vectors, M r is the transformation matrix;

[0028] S302: Calculate the feature vector of the labeled sentence bag according to formula (2) and (3), as shown in formula (4):

[0029]

[0030] Furthermore, the confidence of the relationship between the semantic vector and the relation vector of each sentence in the labeled sentence bag is specifically shown in formula (5):

[0031]

[0032] In formula (5), p(r i |B;θ) means that the current package B belongs to relation r i The probability of; θ represents all parameters in the process of relation extraction; B represents a labeled sentence bag; n r represents the total number of relation labels; j represents the number of each output in the output set o; o i represents the ith output in the output set o of the classifier softmax; the calculation formula of the output set o of the classifier softmax is shown in formula (6):

[0033] o=M s b+b s (6)

[0034] In formula (6), M s represents the intermediate matrix, b s Indicates the offset value.

[0035] In step S104, before the softmax classifier is used, it is trained using a plurality of labeled sentence bags, and the training objective function is specifically as shown in (7):

[0036] min L=L D +λL e (7)

[0037] In formula (7), L is the training objective function, L e , L D As shown in formula (8):

[0038]

[0039]

[0040] In formula (8), N represents the total number of labeled sentence bags, i represents the number of labeled sentence bags, and r is the original vector obtained by subtracting the two entity vectors in the sentence bag.

[0041] A storage device storing instructions and data for implementing an entity relationship extraction method based on a novel relationship attention mechanism.

[0042] An entity relationship extraction device based on a novel relationship attention mechanism comprises: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement an entity relationship extraction method based on a novel relationship attention mechanism.

[0043] The beneficial effect provided by the present invention is that the accuracy of relationship vector representation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the entity relationship extraction method based on the novel relationship attention mechanism of the present invention;

[0045] Figure 2 It is a schematic diagram of the experimental results of the present invention.

[0046] Figure 3 This is a schematic diagram of the hardware device working in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0048] Please refer to Figure 1 , an embodiment of the present invention provides an entity relationship extraction method based on a novel relationship attention mechanism, comprising the following:

[0049] S101: Acquire labeled data; the labeled data is specifically labeled sentence packages;

[0050] S102: extracting a semantic vector of each sentence in the labeled sentence bag using a PCNN model; the PCNN model includes an input layer, a convolution layer, and a pooling layer;

[0051] S103: extracting a feature vector of the labeled sentence bag using a NATT attention mechanism according to the semantic vector of each sentence in the labeled sentence bag;

[0052] S104: Input the feature vector of the labeled sentence bag into the sofmax classifier to obtain the confidence of the relationship between the semantic vector and the relationship vector of each sentence in the labeled sentence bag; the relationship vector is specifically an initialization vector randomly generated according to the original vector of the labeled sentence bag.

[0053] In step S101, labeled data is obtained, specifically by: using a remote supervision method to automatically align a remote knowledge base to obtain a labeled data set; using the Freebase knowledge base and free text alignment to obtain a large amount of annotated training corpus, the idea is based on an assumption: if there is a certain relationship between two entities in the knowledge base, then sentences containing these two entities more or less express this relationship;

[0054] In step S102, the semantic vector of each sentence in the labeled sentence bag is extracted using the PCNN model, and the specific steps are as follows:

[0055] S201: Any sentence s in the labeled sentence bag includes n words, that is, s = (w1, w2, ..., w n ) Among them, w i is the i-th word in sentence s; sentence s is mapped to the word vector table, and its word w i is converted into word vector e i , that is, s=(w1,w2,...,w n ) is converted into s = (e1, e2, ..., e n );

[0056] S202: For word w in sentence s i , calculate the relative distances to the head entity and the tail entity in sentence s, and then map these two relative distances into words w i Position vector d i ;

[0057] S203: word w i The word vector and position vector of the word w are concatenated to get the word w i The feature vector of each word in sentence s is used as the input of the input layer of the PCNN model;

[0058] S204: Use the convolution layer of the PCNN model to perform a convolution operation on the feature vector of each word in the sentence s to obtain a result vector c of each word, as shown in formula (1):

[0059]

[0060] In formula (1), c i is the result vector of the i-th word after the convolution operation; W i is the i-th weight matrix, b is the preset bias value;

[0061] S205: Split the result vector c of each word into three parts, specifically c i ={c i,1 ,c i,2 ,c i,3}, use the pooling layer of the PCNN model to perform maximum pooling on each part and obtain the maximum pooling vector p i ={p i,1 ,p i,2 ,p i,3}; where p i,j =max(c i,j ), j = 1, 2, 3;

[0062] S206: Obtain the semantic vector S = tanh(p) of sentence s according to the maximum pooling vector, where p = (p1, p2, ..., p n ).

[0063] In step S103, according to the semantic vector of each sentence in the labeled sentence bag, the NATT attention mechanism is used to extract the feature vector of the labeled sentence bag, specifically:

[0064] S301: Use the score function to calculate the attention weight of each sentence semantic vector and relationship vector, as shown in formula (2):

[0065]

[0066] In formula (2), i represents the number of the sentence in the labeled sentence bag, m is the total number of sentences in the labeled sentence bag, and j represents the score sequence number of each sentence in the score function; a i is the sentence s in the labeled sentence bag i The attention weight between the relation vector and w i The expression of is shown in formula (3):

[0067]

[0068] In formula (3), b a is the preset offset; is the intermediate matrix; r c is the relationship vector; S i is the i-th sentence s in the labeled sentence bag iThe semantic vector S i ; r c The expression is r c =(e1-e2)M r ; e1, e2 are two entity vectors, M r is the transformation matrix;

[0069] S302: Calculate the feature vector of the labeled sentence bag according to formula (2) and (3), as shown in formula (4):

[0070]

[0071] The confidence of the relationship between the semantic vector and the relation vector of each sentence in the labeled sentence bag is specifically shown in formula (5):

[0072]

[0073] In formula (5), p(r i |B;θ) means that the current package B belongs to relation r i The probability of; θ represents all parameters in the process of relation extraction; B represents a labeled sentence bag; n r represents the total number of relation labels; j represents the number of each output in the output set o; o i represents the ith output in the output set o of the classifier softmax; the calculation formula of the output set o of the classifier softmax is shown in formula (6):

[0074] o=M s b+b s (6)

[0075] In formula (6), M s represents the intermediate matrix, b s Indicates the offset value.

[0076] In step S104, before the softmax classifier is used, it is trained using a plurality of labeled sentence bags, and the training objective function is specifically as shown in (7):

[0077] min L=L D +λL e (7)

[0078] In formula (7), L is the training objective function, L e , L D As shown in formula (8):

[0079]

[0080]

[0081] In formula (8), N represents the total number of labeled sentence bags, i represents the number of labeled sentence bags, and r is the original vector obtained by subtracting the two entity vectors in the sentence bag.

[0082] The dataset used is the document-filtered version of the NYT10 dataset. The original NYT10 dataset was released by Riedel et al. and used by many studies on remote supervised relation extraction. The dataset is generated by aligning the New York Times corpus with the relations in Freebase. Sentences obtained from the news corpus from 2005 to 2006 are used as the training set, and the test set is the data generated by aligning the sentences in Freebase and NYT in 2007. The dataset contains 53 types of relations, including the special relation type "NA", which means there is no relationship between two entities. The training and test data contain 570,088 and 172,448 sentences respectively.

[0083] , using held-out and manual-out evaluations. And compare the accuracy and recall of all baselines in the held-out evaluation to compare the effectiveness of the proposed model

[0084] In the experiments of this paper, we use word2vec

[12] to pre-train word embeddings. We use sentence embedding vector dimensions in the range of {50, 100, 200, 300}, position embedding in the range of {5, 10, 20}, feature maps in the range of {100, 200, 230}, and batch_size in the range of {100, 200, 500}. It has been proven that the best parameter configuration is as follows: dw = 50, d P =5,batch_size=100,lr=0.001.

[0085] The embodiment of the present invention uses held-out and manual-out to evaluate the method proposed in this paper. The accuracy and recall of all baselines are compared in the held-out evaluation to compare the effect of the model proposed in this paper.

[0086] In order to verify the effect of the attention mechanism proposed in this paper in the experiment, we mainly use APCNN, PCNN+ATT, and PCNN+MIL models as comparison models of the experiment.

[0087] Manual evaluation: Table 1 is a TopN comparison table of each model, which shows the accuracy of the first N sentences based on the probability ranking of the test output. The comparison of Top100, Top200 and Top500 listed in Table 1 shows that: (1) The accuracy of the PCNN+MIL algorithm is the lowest, indicating that noise has a serious impact on the model. (2) The PCNN+ATT algorithm is better than the PCNN+MIL algorithm in all aspects, indicating that the selective attention model makes more use of sentence information and reduces the impact of noise; (3) The APCNN algorithm is better than the PCNN+ATT algorithm, indicating that the information of two entities can reflect certain characteristics of the relationship. (4) The PCNN+NATT attention mechanism can effectively give negative instances a smaller weight to reduce noise.

[0088] Table 1 Comparison of TopN models

[0089]

[0090] Held-out Assessment: Figure 2 The precision / recall curves of each model are shown. It can be seen from the figure that PCNN+NATT achieves better performance than other baseline models (APCNN, PCNN+ATT, PCNN+MIL). We conclude that the NATT attention mechanism can effectively give smaller weights to negative instances to reduce noise.

[0091] An entity relationship extraction device 401 based on a novel relationship attention mechanism: The entity relationship extraction method device 401 based on a novel relationship attention mechanism implements the entity relationship extraction method based on a novel relationship attention mechanism.

[0092] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the entity relationship extraction method based on the novel relationship attention mechanism.

[0093] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the entity relationship extraction method based on the novel relationship attention mechanism.

[0094] The beneficial effect of the implementation of the present invention is that the accuracy of the relationship vector representation is improved.

[0095] In the absence of conflict, the above embodiments and features of the embodiments of the present invention may be combined with each other.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An entity relationship extraction method based on a novel relationship attention mechanism, characterized by: S101: Acquire labeled data; the labeled data is specifically labeled sentence packages; S102: extracting a semantic vector of each sentence in the labeled sentence bag using a PCNN model; the PCNN model includes an input layer, a convolution layer, and a pooling layer; S103: extracting a feature vector of the labeled sentence bag using a NATT attention mechanism according to the semantic vector of each sentence in the labeled sentence bag; S104: Inputting the feature vector of the labeled sentence bag into the sofmax classifier to obtain the confidence of the relationship between the semantic vector and the relationship vector of each sentence in the labeled sentence bag; the relationship vector is specifically an initialization vector randomly generated according to the original vector of the labeled sentence bag; In step S102, the semantic vector of each sentence in the labeled sentence bag is extracted using the PCNN model, and the specific steps are as follows: S201: Any sentence s in the labeled sentence bag includes n words, that is, s = (w1, w2, ..., w n ) Among them, w i is the i-th word in sentence s; sentence s is mapped to the word vector table, and its word w i is converted into word vector e i , that is, s=(w1,w2,...,w n ) is converted into s = (e1, e2, ..., e n ); S202: For word w in sentence s i , calculate the relative distances to the head entity and the tail entity in sentence s, and then map these two relative distances into words w i Position vector d i ; S203: word w i The word vector and position vector of the word w are concatenated to get the word w i The feature vector of each word in sentence s is used as the input of the input layer of the PCNN model; S204: Use the convolution layer of the PCNN model to perform a convolution operation on the feature vector of each word in the sentence s to obtain a result vector c of each word, as shown in formula (1): In formula (1), c i is the result vector of the i-th word after the convolution operation; W i is the i-th weight matrix, b is the preset bias value; S205: Split the result vector c of each word into three parts, specifically c i ={c i,1 ,c i,2 ,c i,3 }, use the pooling layer of the PCNN model to perform maximum pooling on each part and obtain the maximum pooling vector p i ={p i,1 ,p i,2 ,p i,3 }; where pi,j = max(ci,j), j = 1, 2, 3; S206: Obtain the semantic vector S = tanh(p) of sentence s according to the maximum pooling vector, where p = (p1, p2, ..., p n ); In step S103, according to the semantic vector of each sentence in the labeled sentence bag, the NATT attention mechanism is used to extract the feature vector of the labeled sentence bag, specifically: S301: Use the score function to calculate the attention weight of each sentence semantic vector and relationship vector, as shown in formula (2): In formula (2), i represents the number of the sentence in the labeled sentence bag, m is the total number of sentences in the labeled sentence bag, and j represents the score sequence number of each sentence in the score function; a i is the sentence s in the labeled sentence bag i The attention weight between the relation vector and w i The expression of is shown in formula (3): In formula (3), b a is the preset offset; is the intermediate matrix; r c is the relationship vector; S i is the i-th sentence s in the labeled sentence bag i The semantic vector S i ; r c The expression is r c =(e1-e2)M r ; e1, e2 are two entity vectors, M r is the transformation matrix; S302: Calculate the feature vector of the labeled sentence bag according to formula (2) and (3), as shown in formula (4): The confidence of the relationship between the semantic vector and the relation vector of each sentence in the labeled sentence bag is specifically shown in formula (5): In formula (5), p(r i |B;θ) means that the current package B belongs to relation r i The probability of; θ represents all parameters in the process of relation extraction; B represents a labeled sentence bag; n r represents the total number of relation labels; j represents the number of each output in the output set o; o i represents the ith output in the output set o of the classifier softmax; the calculation formula of the output set o of the classifier softmax is shown in formula (6): o=M s b+b s (6) In formula (6), M s represents the intermediate matrix, b S Indicates the bias value; In step S104, before the softmax classifier is used, it is trained using a plurality of labeled sentence bags, and the training objective function is specifically as shown in (7): minL=L D +λL e (7) In formula (7), L is the training objective function, L e , L D As shown in formula (8): In formula (8), N represents the total number of labeled sentence bags, i represents the number of labeled sentence bags, and r is the original vector obtained by subtracting the two entity vectors in the sentence bag.

2. The entity relationship extraction method based on the novel relationship attention mechanism according to claim 1, characterized in that: In step S101, labeled data is obtained, specifically by using a remote supervision method to obtain a labeled data set by automatically aligning a remote knowledge base.

3. A storage device, characterized in that: The storage device stores instructions and data for implementing any one of the entity relationship extraction methods based on the novel relationship attention mechanism described in claims 1 to 2.

4. An entity relationship extraction device based on a novel relationship attention mechanism, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement any one of the entity relationship extraction methods based on the novel relationship attention mechanism described in claims 1 to 2.

Images