Remote supervision relationship extraction method and device based on label graph to suppress the influence of noise data

By constructing tag graphs and using graph convolution neural networks to capture tag dependencies, combining tag heuristic attention mechanism and PCNN networks, the problem of noise data impact caused by failure to fully utilize tag dependencies in the existing technology is solved, and significantly improved Accuracy and recall rate of remote supervision relationship extraction.

CN114461747BActive Publication Date: 2024-11-01INSTITUTE OF INFORMATION ENGINEERING CHINESE ACADEMY OF SCIENCES
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210049778.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-17
Publication Date
2024-11-01
Estimated Expiration
2042-01-17

AI Technical Summary

Technical Problem

The existing remote supervision relationship extraction method fails to fully utilize the dependence between labels, limiting the ability of attention mechanisms to suppress the influence of noisy sentences, resulting in a serious impact on model training.

Method used

Construct the label graph, use graph convolution neural network to capture the dependence between labels, and give low weight to noisy sentences and high weight to non-noisy sentences in the label heuristic attention mechanism, and generate sentences through graph convolution and PCNN networks. Packet represents that the cross entropy loss is calculated for model training based on long and short-term memory networks and word vectors.

Benefits of technology

It effectively reduces the impact of noisy data on model training, improves the accuracy and recall of relationship extraction, and significantly improves the model performance in the noisy data environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114461747B_ABST
    Figure CN114461747B_ABST
Patent Text Reader

Abstract

The present invention discloses a remote supervised relationship extraction method and device based on a label graph to suppress the influence of noise data, comprising constructing a label graph based on the original labels of entity pairs; generating a label representation matrix of the label graph; collecting sentences containing target entity pairs; and extracting a label graph based on the original labels of entity pairs. j , get the sentence bag; combine the position of the target entity pair in each sentence j with the vector representation of the target entity pair, and calculate each sentence s j The sentence indicates o j ; Calculate the label l for the sentence representation o j The weight α lj ; for the sentence j Perform weighted summation to obtain the sentence bag representation b of the sentence bag under the label l l ; Calculate the sentence bag representation b l The score g under label l l ; Splice the fraction g l , and based on the obtained basic label score vector g b , obtaining the original label of the target entity pair. The present invention better solves the problem of noise data affecting model training in remote supervision relationship extraction tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of entity relationship extraction in natural language processing, and is a remote supervision relationship extraction method and device based on a label graph to suppress the influence of noise data. Background Art

[0002] Entity relationship extraction is an important subtask in the field of natural language processing. It aims to discover the relationship between entities from unstructured text and is widely used in tasks such as question-answering systems, knowledge graph construction, and information retrieval. Remote supervision relationship extraction is a method of entity relationship extraction. This method uses an automatically annotated dataset (remote supervision dataset) to train the entity relationship extraction model. However, the noise data in the remote supervision dataset seriously affects the model training.

[0003] Multiple instance learning (MIL) can effectively solve the above noise problem. MIL combines all entities and The sentences are regarded as a sentence bag and marked as the corresponding relationship in the knowledge base (KB). Based on MIL, the traditional PCNN (Piece-wise Convolution Networks) method first performs a convolution operation on the sentence representation matrix; then, the convolution features are processed by maximum pooling in each dimension to obtain an effective sentence representation vector; finally, the attention mechanism is used to aggregate non-noise sentence information to obtain a sentence bag representation, and then the sentence bag representation is used to predict and Existing methods incorporate label knowledge into the attention mechanism, but do not fully utilize the dependencies between labels, which limits the ability of the attention mechanism to suppress the influence of noisy sentences. Summary of the invention

[0004] In order to solve the problem that existing methods do not fully utilize label knowledge, the present invention proposes a remote supervised relationship extraction method and device based on label graph to suppress the influence of noise data. Considering the theoretically unapproachable characteristics of noisy sentence representation and label representation in feature space, the present invention assigns low weight to noisy sentences and high weight to non-noise sentences in the label-inspired attention mechanism.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A remote supervision relationship extraction method based on a label graph to suppress the influence of noise data, the steps of which include:

[0007] Collecting original tags of entity pairs, taking the original tags as basic tags, and obtaining super tags of the original tags by using the hierarchical relationship of tags;

[0008] Based on the super tag and the basic tag, a tag graph is constructed, and a tag representation matrix of the tag graph is calculated, wherein a node in the tag graph is a tag l, and the tag l includes: the super tag or the basic tag; in the tag graph, if a tag l is a subtag of another tag l′, there is a connecting edge between the tag l and the tag l′;

[0009] Collect sentences s containing target entity pairs j , get the sentence bag, and combine the position of the target entity pair in each sentence j with the vector representation of the target entity pair to calculate each sentence s j The sentence indicates o j ;

[0010] Based on the label representation matrix, calculate the label l for the sentence representation o j The weight α lj After that, the sentence is expressed as o j Perform weighted summation to obtain the sentence bag representation b of the sentence bag under the label l l ;

[0011] Combined with the label representation matrix, the sentence bag representation b is calculated l The score g under label l l ;

[0012] Concatenate the fractions g l , and based on the obtained basic label score vector g b , obtain the original label of the target entity pair.

[0013] Furthermore, calculating the label representation matrix of the label graph includes:

[0014] 1) Using a graph convolutional neural network to capture the dependencies between the labels l, and obtaining a connection matrix A;

[0015] 2) Perform information propagation of a multi-layer graph convolutional neural network, wherein the information propagation rules include: v+1 =σ(AD -1 H v W v ), H v represents the feature representation matrix processed by the first v layers of the graph convolutional neural network, σ(.) represents the activation function, D is the degree matrix of the matrix connection matrix A, and W is the weight parameter matrix;

[0016] 3) Calculate each feature representation matrix H v The label representation matrix of the label graph is obtained by adding the results of .

[0017] Furthermore, the calculation of each sentence s jThe sentence indicates o j ,include:

[0018] 1) For each sentence s j , by concatenating the sentences s j Each word in w i Representation, get the matrix representation u j ;

[0019] 2) Use PCNN network to encode the matrix representation u j , we get the vector representation r j ;

[0020] 3) Input the vector representation of each entity in the target entity pair into the long short-term memory network to obtain the target entity pair representation r e ;

[0021] 4) According to the vector representation r j The entity pair with the label r e , get the sentence representation o j .

[0022] Further, the word w is obtained by the following steps i Representation:

[0023] 1) Through the word w i The vector representation of the position of the target entity in the discrete space is obtained, and the word w i The position vector of

[0024] 2) Calculate the word w i Word2vec word vector;

[0025] 3) Concatenate the position vector and the word2vec word vector to get the word w i Representation.

[0026] Furthermore, the matrix representation u is encoded using a PCNN network j ,include:

[0027] 1) According to the position of the target entity pair, the sentence s j Cut and correspond to the matrix representation u j On the above, we get matrix representation p1, matrix representation p2 and matrix representation p3;

[0028] 2) Perform maximum pooling processing on the matrix representation p1, the matrix representation p2, and the matrix representation p3 respectively to obtain the vector representation h1, the vector representation h2, and the vector representation h3;

[0029] 3) Represent the matrix u j Perform average pooling operation to obtain vector representation havg ;

[0030] 4) Concatenate vector representation h1, vector representation h2, vector representation h3 and vector representation h avg , get the matrix representation u j .

[0031] Further, the calculation of the label l represents o for the sentence j The weight α lj ,include:

[0032] 1) Through the attention mechanism, calculate the representation of label l for sentence s j Search results for lj ;

[0033] 2) According to the search result e lj , calculate the weight α lj .

[0034] Furthermore, a remote supervision relation extraction model is used to complete the label representation matrix and sentence bag representation b l With the fractional vector g l , wherein the cross entropy loss is used as the loss function when training the remote supervision relationship extraction model.

[0035] Furthermore, the cross entropy loss value is calculated by the following steps:

[0036] 1) Concatenate the scores g corresponding to the basic labels l , get the score vector of each basic label corresponding to the training data,

[0037] 2) Concatenate the scores g corresponding to the super tags l , get the score vector of the super label corresponding to the training data;

[0038] 3) Use the sigmoid activation function to process the score vector of the basic label and the score vector of the super label to obtain the sigmoid score vector;

[0039] 4) Calculate the cross entropy loss value for the sigmoid score vector p in is the basic tag set, For the super tag set, γ b is the weight of the basic label in the loss function, γ s is the weight of the super label in the loss function.

[0040] A storage medium stores a computer program, wherein the computer program is configured to execute the above method when running.

[0041] An electronic device comprises a memory and a processor, wherein the memory stores a program for executing the above method.

[0042] Compared with the existing technology, the present invention better solves the problem of noise data affecting model training in remote supervision relationship extraction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The framework structure diagram of the present invention.

[0044] Figure 2 Represents the PR curves of PCNN+ATT, PCEN+ATT, and PCEN+GATT on the NYT dataset.

[0045] Figure 3 T-SNE visualization results of label representation. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] Figure 1 The framework structure of the present invention is shown. First, in the label encoder, the present invention uses the hierarchical relationship between labels in the knowledge base to construct a label graph, mine the dependencies between labels, and learn rich label representations. Secondly, in the bag encoder, the present invention uses a PCNN network to generate sentence representations, and uses the target entity pair representation to enhance the sentence representation. Then, different weights are given to different sentences based on the label-inspired attention mechanism to generate an efficient sentence bag representation. Finally, the sentence bag representation is used to predict the target relationship.

[0048] In the present invention, the steps of remote supervision relationship extraction are mainly as follows:

[0049] 1. Build a label graph in the label encoder and use a graph convolutional neural network (GCN) to capture the dependencies between labels and generate effective label representations. The detailed steps are as follows:

[0050] (1) According to the label hierarchy, the original label is expanded into a basic label and a super label. For example, the label " / people / person / place_lived" can be expanded into a super label " / people", " / people / person" and a basic label " / people / person / place_lived". The label hierarchy is used to construct a label graph, where the nodes only contain labels. If a label is a sub-label of another label, there is a connecting edge between the two labels.

[0051] (2) Use two layers of GCN to realize the label graph. According to the different dependencies between labels, the connection matrix A of GCN is defined as:

[0052]

[0053] Among them, A∈R L×L , L represents the sum of the number of super tags and basic tags. C(l,l′) represents the number of times that tag l and tag l′ appear in the same sentence bag in the training set, and C(l) represents the number of times tag l appears in the training set. ll′ represents the possibility of label l being transferred to label l'. li′ Asymmetric, meaning that the dependencies between labels l and l′ are different:

[0054]

[0055] For example, the tag “ / people / deceased_person” (tag l) is a subtag of the tag “ / people” (tag l′), so A ll′ =1, otherwise it is not true, so A ll′ <1. The information propagation rule in GCN is defined as:

[0056] H v+1 =σ(AD -1 H v W v ) (3)

[0057] Among them, H v ∈R L×d H represents the feature representation matrix processed by the first v layers of GCN, and d is the dimension of the feature matrix. 0 =V, V represents the randomly initialized node feature matrix. σ(.) is the hyperbolic tangent activation function. D is the degree matrix of matrix A, D ll =∑ l′ A ll′ .W v ∈R L×Lis the learnable weight parameter matrix of the vth layer GCN. The final label representation matrix is ​​defined as:

[0058] T=H 1 +H 2 (4)

[0059] 2. Given a sentence bag = {s1, s2, ..., s n}, sentence s j ={w1, w2, ..., w m}(1≤j≤m), m is the sentence s j The number of words in the packet encoder. In the packet encoder, the PCNN network is used to generate sentence representation. The detailed steps are as follows:

[0060] (1) Generating word w i For word w i , splicing w i The position vector and the corresponding word2vec word vector are used as word w i The position vector represents the word w i A vector representation of the position relative to the target entity pair in discrete space.

[0061] (2) Generate sentence s i Concatenate the word representations in (1) to get the corresponding sentence matrix, and use a two-dimensional convolution operation to process the matrix representation to get the sentence matrix representation u j .

[0062] (3) Generate a vector representation of the sentence. Use the PCNN network to encode the sentence s j The matrix representation u j , and get its vector representation r j The specific steps are as follows: First, according to the target entity The sentence is divided into three parts, corresponding to the matrix representation, that is, u j Divided into p1, p2, The three-part matrix representation, where d c is the number of convolution kernels in (2), m1, m2, m3 are the lengths of each part. Then, the matrix p k Perform maximum pooling to obtain vector representation

[0063] h k =Max_pooling(p i ) (5) For the matrix u j Perform average pooling operation to obtain vector representation

[0064] h avg=Average_pooling(u j ) (6)

[0065] Finally, splice h k and h avg , get the vector representation r of sentence j j R d , d = 4d c :

[0066] r j =[h1;h2;h3;h avg ] (7)

[0067] 3. Generate vector representation r of target entity pair e , using r e Enhance the sentence representation r in step 2 j The detailed steps are as follows:

[0068] (1) Based on the given sentence bag and its corresponding target entity pair and First, use the word vector and Mapped to discrete space, we get vector representations q1 and q2, and then use a single-layer long short-term memory network (LSTM) to learn the target entity pair representation r e :

[0069] r e =LSTM([q1;q2]) (8)

[0070] Among them, r e is the vector of the last unit in the LSTM hidden state.

[0071] (2) Use r e Enhance the vector representation r of sentence j j , and get the final representation of the sentence o j :

[0072] o j =βr e +(1-β)r j (9)

[0073] Among them, β is an adjustable weight parameter.

[0074] 4. Use the label-inspired attention mechanism to calculate the label l for the sentence representation o j The weight α lj Then, the weighted sum of the n sentence representations is taken to obtain the representation b of the sentence package under the label l. l :

[0075]

[0076]

[0077] where l∈{1, 2, ..., L}, e lj In the attention mechanism, the label l represents the sentence s j Search results for lj The calculation formula is:

[0078]

[0079] Among them, t l It is the l-th row vector of the label representation matrix T in step 1, representing the representation of label l.

[0080] 5. Loss calculation and result prediction. Define the sentence bag to represent the score of b under label l as g l :

[0081]

[0082] By g l The score vector g corresponding to the basic label can be obtained b The score vector g corresponding to the super label s . Use sigmoid activation function to process g b and g s , get the sigmoid score vector p:

[0083] p = sigmoid([g b ; g s ]) (14)

[0084] Calculate the cross entropy loss value for vector p

[0085]

[0086]

[0087] in and Represent the basic tag set and super tag set respectively. γ b and γ s is an adjustable parameter, which represents the weight of the basic label and super label in the loss function. y represents the one-hot form of the target label. Since the initial sentence bag label belongs to the basic label set, in the score vector g b In the above example, the basic label corresponding to the maximum score value is taken as the final prediction result.

[0088]

[0089] Table 1 shows the experimental results of the present invention and related comparative methods on the public dataset - NYT dataset. b and γ s When the values ​​of are set to 2 and 0.5 respectively, the present invention can achieve the best experimental results. In Table 1, PCNN+ATT represents the traditional PCNN network-based method, which uses a label-inspired attention mechanism. PCEN indicates that compared with the traditional PCNN, the target entity pair information is also used to encode the sentence representation, and both PCNN and PCEN represent sentence encoders. (+ATT) represents that the label-inspired attention mechanism does not consider label dependencies; (+GATT) represents that in the present invention, the label-inspired attention mechanism is based on the label representation captured by GCN. "One", "Two" and "All" represent randomly selecting one, two and all sentences from a sentence bag to predict entity pair relationships. The P@N indicator represents the accuracy corresponding to the top N sentence bags with the highest prediction scores in the test set.

[0090] From Table 1, we can observe that compared with PCNN+ATT and PCEN+ATT, the present invention (PCEN+GATT) has obvious improvements in all P@N indicators; in P@100, the present invention has the highest improvement of 5.8% and 12% respectively compared with the two comparison methods. Figure 2 As shown in the experiment, the accuracy-recall rate (PR) curve is also compared. Figure 2 The present invention achieved the best experimental results. The following conclusions can be drawn from the analysis of the experimental results: the label-inspired attention mechanism based on effective label knowledge can give more appropriate weights to different sentences, thereby reducing the impact of noise information. Figure 3 The corresponding T-SNE visualization results of label representation in PCNN+ATT and PCEN+GATT methods are shown. Figure 3 It can be seen that in PCEN+GATT, the propagation of label knowledge in the label graph makes similar labels close to each other in the feature space, while the difference between opposite labels increases. Figure 3 shows that the label knowledge learned by the present invention is more representative than that of the PCEN+ATT method. In the feature space, if a sentence representation is not similar to a label with obvious features, the possibility that the sentence (noise sentence) belongs to the label is low, so the sentence is given a lower weight, which further reduces the impact of noise information.

[0091]

[0092] Table 1

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A remote supervised relationship extraction method based on a label graph to suppress the influence of noise data, the steps of which include: Collecting original tags of entity pairs, taking the original tags as basic tags, and obtaining super tags of the original tags by using the hierarchical relationship of tags; Based on the super tag and the basic tag, a tag graph is constructed, and a tag representation matrix of the tag graph is calculated, wherein a node in the tag graph is a tag l, and the tag l includes: the super tag or the basic tag; in the tag graph, if a tag l is a subtag of another tag l′, there is a connecting edge between the tag l and the tag l′; Collect sentences s containing target entity pairs j , get the sentence bag, and combine the position of the target entity pair in each sentence j with the vector representation of the target entity pair to calculate each sentence s j The sentence indicates o j ; Based on the label representation matrix, calculate the label l for the sentence representation o j The weight α lj After that, the sentence is expressed as o j Perform weighted summation to obtain the sentence bag representation b of the sentence bag under the label l l ; Combined with the label representation matrix, the sentence bag representation b is calculated l The score g under label l l ; Concatenate the fractions g l , and based on the obtained basic label score vector g b , obtain the original label of the target entity pair.

2. The method according to claim 1, characterized in that The step of calculating a label representation matrix of the label graph includes: 1) Using a graph convolutional neural network to capture the dependencies between the labels l, and obtaining a connection matrix A; 2) Perform information propagation of a multi-layer graph convolutional neural network, wherein the information propagation rules include: v+1 =σ(AD - 1 H v W v ), H v represents the feature representation matrix processed by the first v layers of the graph convolutional neural network, σ(.) represents the activation function, D is the degree matrix of the matrix connection matrix A, and W is the weight parameter matrix; 3) Calculate each feature representation matrix H v The label representation matrix of the label graph is obtained by adding the results of .

3. The method according to claim 1, characterized in that The calculation of each sentence s j The sentence indicates o j ,include: 1) For each sentence s j , by concatenating the sentences s j Each word in w i Representation, get the matrix representation u j ; 2) Use PCNN network to encode the matrix representation u j , we get the vector representation r j ; 3) Input the vector representation of each entity in the target entity pair into the long short-term memory network to obtain the target entity pair representation r e ; 4) According to the vector representation r j The entity pair with the label r e , get the sentence representation o j .

4. The method according to claim 3, characterized in that The word w is obtained by the following steps i Representation: 1) Through the word w i The vector representation of the position of the target entity in the discrete space is obtained, and the word w i The position vector of 2) Calculate the word w i Word2vec word vector; 3) Concatenate the position vector and the word2vec word vector to get the word w i Representation.

5. The method according to claim 3, characterized in that The matrix representation u is encoded using the PCNN network j ,include: 1) According to the position of the target entity pair, the sentence s j Cut and correspond to the matrix representation u j On the above, we get matrix representation p1, matrix representation p2 and matrix representation p3; 2) Perform maximum pooling processing on the matrix representation p1, the matrix representation p2, and the matrix representation p3 respectively to obtain the vector representation h1, the vector representation h2, and the vector representation h3; 3) Represent the matrix u j Perform average pooling operation to obtain vector representation h avg ; 4) Concatenate vector representation h1, vector representation h2, vector representation h3 and vector representation h avg , get the matrix representation u j .

6. The method according to claim 1, characterized in that The calculation of the label l represents o for the sentence j The weight α lj ,include: 1) Through the attention mechanism, calculate the representation of label l for sentence s j Search results for lj ; 2) According to the search result e lj , calculate the weight α lj .

7. The method according to claim 1, characterized in that Use a remote supervision relationship extraction model to complete the label representation matrix and sentence bag representation b l With the fractional vector g b , wherein the cross entropy loss is used as the loss function when training the remote supervision relationship extraction model.

8. The method according to claim 7, characterized in that The cross entropy loss value is calculated by the following steps: 1) Concatenate the scores g corresponding to the basic labels l , get the score vector of each basic label corresponding to the training data, 2) Concatenate the scores g corresponding to the super tags l , get the score vector of the super label corresponding to the training data; 3) Use the sigmoid activation function to process the score vector of the basic label and the score vector of the super label to obtain the sigmoid score vector; 4) Calculate the cross entropy loss value for the sigmoid score vector p in is the basic tag set, For the super tag set, γ b is the weight of the basic label in the loss function, γ s is the weight of the super label in the loss function.

9. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

10. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • User mining method and device based on demand word

    CN108763556A

  • Remote supervision relation extraction method combined with background knowledge

    CN109635124A

  • Remote supervision text entity relationship extraction method based on deep reinforcement learning

    CN113591478A

  • Domain text similarity calculation method and system based on label graph conversion

    CN113742494A

  • Method and apparatus for automatically generating inference questions and answers

    WO2021184311A1