Global pointer network entity relation extraction method integrating relation filtering and multi-task learning
Through a multi-task learning strategy combining global pointer network and relationship filtering, the problem of triple overlap and nesting in entity relationship extraction is solved, the loss function is optimized, and more efficient entity relationship extraction and relationship filtering is realized, and the model is improved. Overall performance.
Patent Information
- Application Number
- CN202410515138.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-04-26
AI Technical Summary
The existing entity relationship extraction model has problems of prediction errors and redundant relationship judgments when dealing with triple overlapping and nested entities, and the loss function optimization of multi-task learning is not effective enough, resulting in weight imbalance between tasks.
The global pointer network is used to combine relationship filtering and multi-task learning strategies, and through adaptive loss weight adjustment, the five-tuple annotation strategy and matrix marking positions are used to optimize the loss function to solve overlap and nesting problems, and reduce redundant relationship judgments.
It improves the accuracy and efficiency of entity relationship extraction, enhances the global concept of the model, balances the performance of multi-task learning, reduces redundant relationship judgment, and improves the overall performance of the model.
Smart Images

Figure CN118504560B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing, and in particular to a global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning. Background Art
[0002] With the rapid development and popularization of the Internet, the way of information exchange has undergone earth-shaking changes. How to obtain resources on the Internet in a more intelligent and fast way has become a research hotspot. Accurate extraction of triples consisting of head entities, relations and tail entities is a key step in the construction of knowledge graphs. Entity relationship extraction, as a subtask in the field of information extraction, has attracted widespread attention from researchers at home and abroad. At present, the entity relationship extraction methods based on deep learning are mainly divided into two methods: pipeline methods and joint extraction methods. The pipeline method divides entity relationship extraction into two subtasks: named entity recognition and relationship extraction, but it has many problems such as error accumulation, ignoring the correlation between the two tasks and information redundancy. In order to solve the above problems, researchers have proposed a joint extraction model for the simultaneous extraction of many entity relations. However, due to the complex relationship of sentences, there are triples in the same sentence that share the same subject, object or relationship, and there is also a phenomenon of nested entities containing another entity in one entity.
[0003] At present, there have been many studies on the problems of overlapping triples and nested entities. The decomposition strategy ETL-Span proposed by Yu et al. and the cascade binary tagging framework CasRel proposed by Wei et al. use novel annotation strategies to cleverly avoid the problem of triple overlap, but do not solve the problem of nested entities. The TPLinker model proposed by Wang et al. and the GPLinker model proposed by Su et al. use matrix annotation to solve the problems of overlapping triples and nested entities. However, most of the existing joint extraction models usually require entity annotation for each relationship in the predefined relationship set, there are a lot of redundant relationship judgments, and do not consider the multi-task learning problem in the entity relationship extraction process.
[0004] Based on the above methods, there are the following disadvantages:
[0005] (1) Using a conventional pointer network to mark the position, indicating the start and end positions of the entity, only focuses on the head and tail positions of the entity, and fails to pay attention to the global information of the entity. When predicting, it is predicted according to the entire entity, which easily leads to prediction errors and affects the accuracy.
[0006] (2) It is necessary to label the entities of each relationship in the predefined relationship set, and there are a large number of redundant relationship judgments.
[0007] (3) For multi-task learning, its loss function is optimized. However, in the above-mentioned entity relationship extraction model, the loss is simply accumulated, ignoring the differences in input and output between sub-tasks and the importance of different sub-tasks. This may cause multi-task learning to be dominated by a certain task, that is, the fitting effect on the dominant task is very good, while the effect on other tasks becomes worse. Summary of the invention
[0008] To solve the above problems, the present invention provides a global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning, the method comprising:
[0009] Step 1: Get the original text, map it to the input sequence of the pre-trained language module, mark the beginning and end of the input sequence, and use the pre-trained language module as an encoder to extract features from the sentences in the original text to obtain the output word vector;
[0010] Step 2: Perform linear transformation on the word vector obtained in step 1 to obtain the transformed sequence, construct a first scoring function based on the transformed sequence, add relative position information to the first scoring function to obtain a second scoring function, construct a loss function for the entity extraction task, construct a global pointer network based on the second scoring function, train the global pointer network, and use the global pointer network for entity extraction;
[0011] Step 3: Based on the word vector obtained in step 1, a pooling operation is performed to obtain a pooling result. The pooling result is passed through a linear layer and a non-activation function to obtain a relationship probability. A threshold is set to filter redundant relationships. A corresponding second loss function is constructed for the relationship filtering task to obtain a relationship filtering decoding module. The relationship filtering decoding module is trained. The relationship filtering decoding module is used to filter relationships.
[0012] Step 4: Construct an entity and relationship extraction module based on the global pointer network and the relationship filtering and decoding module, process the text to be processed based on the entity and relationship extraction module, identify the first and last positions of the subject entity and the object entity respectively, and obtain the recognition results between the subject entity and the object entity, and obtain the entity and relationship extraction results corresponding to the text to be processed based on the recognition results.
[0013] In view of the above problems, the present invention proposes a global pointer network entity relationship extraction method that integrates relationship filtering and multi-task learning. A global pointer network is used as a label decoder to solve the overlapping triple problem and the nested entity problem. It regards the entity start and end position and entity category as a whole for identification, uses a matrix to mark the position, and places the entity information and the head and tail position information of the two entities that imply the relationship into several different matrix modules, that is, three global pointers are used to identify the subject and object, the starting position identification of the subject and object corresponding to each pair of relationships, and the ending position identification of the subject and object corresponding to each pair of relationships. In addition, since the global pointer network needs to label each relationship in the predefined relationship set when extracting entity relationships, there are a large number of redundant relationship judgments. The present invention proposes relationship filtering, first predicting the relationship, filtering out some impossible redundant relationships, and then extracting entity relationship triples more quickly and accurately. Based on this, the model is divided into two sub-tasks: relationship filtering and entity relationship extraction. A multi-task learning strategy is adopted for them. Multiple tasks are learned simultaneously by sharing the model. The optimization of the loss function is one of the important strategies of multi-task learning. Therefore, in order not to damage the performance of the model in this application, the present invention adopts an adaptive loss weight adjustment strategy to balance the loss functions of multiple tasks, and dynamically adjusts the loss weight according to the learning speed, difficulty of learning and learning effect of different tasks.
[0014] Preferably, step 4 is an entity relationship extraction step. The head position and tail position of the subject entity and the object entity are identified through the global pointer network designed in step 2 and the relationship filtering in step 3. Step 4 specifically includes:
[0015] Consider the extraction of triples (s, r, o) as five-tuples (s h ,s t ,r,o h ,o t ) is extracted, so that the head and tail of the subject entity and the object entity are separated to adopt a new labeling strategy, and a five-tuple scoring function is constructed. Based on the five-tuple scoring function, the five-tuple entity and relationship extraction processing is performed on the processed text to obtain the entity and relationship extraction results. The five-tuple entities and relationships include: the subject entity head, the subject entity tail, the relationship between the subject entity and the object entity, the object entity head and the object entity tail.
[0016] Preferably, the quintuple is (s h ,s t ,r,o h ,o t ), the scoring function of the quintuple is:
[0017] S(s h ,s t ,r,oh ,o t )=S(s h ,s t )+S(o h ,o t )+S(s h ,o h |r)+S(s t ,o t |r);
[0018] Among them, S(s h ,s t ) and S(o h ,o t ) represent the first and last scores of the subject entity and the object entity respectively. If both are greater than zero, the subject entity and the object entity are judged; S(s h ,o h |r) indicates that the relationship matching is performed using the head features of the subject entity and the object entity as representations. If there is no nested entity phenomenon where one entity contains another entity, and when S(s h ,o h |r)>0, all relationships are determined; if there is a nested entity phenomenon where one entity contains another entity, then S(s t ,o t |r), S(s t ,o t |r) is the tail feature of the subject entity and the object entity.
[0019] Preferably, the step 4 further comprises constructing a loss function for the entity and relationship extraction module, and the loss function of the entity and relationship extraction module is:
[0020]
[0021] Among them, L GP_head Represents the loss value of the subject entity and object entity head based on a specific relationship, L GP_tail Represents the loss value of the tail of the subject entity and the object entity based on a specific relationship; M and N are the sets of positive and negative categories, Si represents the predicted i-th value corresponding to the i-th position in the label, e is a natural constant, A=M∪N, A is the set of positive and negative categories;
[0022] Optimize the loss function of the entity and relationship extraction module by:
[0023]
[0024] Among them, S k (t-1) represents the relative rate of decrease of the loss function, L k(t-1) represents the current loss function value, L k (t-2) represents the previous loss function value, w k (t) represents the weight value of each task, X represents the temperature coefficient used to adjust the task distribution, and K is the set value.
[0025] Preferably, step 1 specifically includes:
[0026] The Roberta-wwm pre-trained language module is used as an encoder to extract the features of the sentence, and the vector representation of each word is learned according to the contextual semantic information. The original text S = [w1, w2, ..., w n ] is mapped to the input sequence of the Roberta-wwm model, the sequence length is n, and [CLS] and [SEP] are marked at the beginning and end of the sequence respectively. A text S with a sequence length of n is input, and the word vector output after Roberta-wwm encoding is expressed as:
[0027] V=([CLS],w1,w2,...,w n ,[SEP])=(h1,h2,...,h n );
[0028] Among them, w i Represents the i-th character of the input sequence, h i Represents the embedding vector corresponding to each character, 1≤i≤n.
[0029] Preferably, step 2 specifically includes:
[0030] Based on the word vectors [h1,h2,...,h n ] is subjected to linear transformation to obtain the transformed sequence, and the sequence [q 1,α ,q 2,α ,...,q n,α ] and [k 1,α ,k 2,α ,...,k n,α ], sequence [q 1,α ,q 2,α ,...,q n,α ] and [k 1,α ,k 2,α ,...,k n,α ] is the vector sequence used to identify the αth type of entity, and the linear transformation method is:
[0031] q i,a =w q,α h i +b q,α ;
[0032] ki,a =w k,α h i +b k,α ;
[0033] Among them, 1≤i≤n, w q,α is the weight parameter of the current linear transformation, b q,α is the bias term, w k,α is the weight parameter of the current linear transformation, b k,α is the bias term;
[0034] The first scoring function for determining the entity is constructed based on the transformed sequence. The first scoring function is:
[0035]
[0036] Among them, s α (i,j) indicates that the continuous segment from i to j is a score of an entity of type α, and q i,α and k i,α The inner product of is used as the score of the entity fragment type α;
[0037] The relative position information is added to the first scoring function to obtain the second scoring function, which is:
[0038]
[0039] Where T is the transpose matrix operator, R i and R j is a transformation matrix used to increase relative position information and satisfy the relationship
[0040] Construct the loss function of the entity extraction task. The loss function of the entity extraction task is:
[0041]
[0042] Among them, L GP_entity Represents the loss value of entity extraction. M and N are the sets of positive and negative categories, Si represents the predicted i-th value corresponding to the i-th position in the label, e is a natural constant, A=M∪N, and A is the set of positive and negative categories.
[0043] Preferably, step 3 specifically includes:
[0044] Based on the word vectors [h1,h2,...,h n ] performs pooling operation to obtain the pooling result, 1≤i≤n, and the pooling operation is expressed as:
[0045]
[0046] in, is the pooling result of the i-th word vector, G is the predefined relationship set {r1,r2,...,r k}, d is the dimension of embedding, k is the size of the relation set, r i for a relationship;
[0047] The pooling result is passed through a linear layer and a deactivation function to obtain the relationship probability, which is calculated as follows:
[0048]
[0049] Among them, P r ∈G k represents the relationship probability of the text about the relationship, Sigmod() represents the nonlinear activation function, and its value range is (0,1), W r ∈G d×k represents the trainable weights, b r ∈G k represents the bias term, G is a predefined relationship set {r1,r2,...,r k};
[0050] Set the threshold μ to filter redundant relations. If the relation probability P r If it is greater than μ, it is considered as relationship filtering, and if it is less than μ, it is considered as redundant relationship;
[0051] Constructing a corresponding second loss function for the relationship filtering task, obtaining a relationship filtering decoding module, and training the relationship filtering decoding module, wherein the relationship filtering decoding module is used to screen and filter the relationship;
[0052] The second loss function is:
[0053]
[0054] Among them, L rel represents the loss of the relation filtering decoding module, K represents the total number of relations in the predefined relation set, and P i r Represents the probability corresponding to the i-th relationship, y i is the true label of relation r.
[0055] Preferably, the method further comprises step 5:
[0056] The entity and relationship extraction results obtained in step 4 are experimented on the Chinese dataset, and the precision P, recall R and F1 value are used to evaluate the results.
[0057] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:
[0058] 1. In order to solve the problem of triple overlap and nested entities, the present invention designs a global pointer network. As a label decoder, it regards the entity start and end positions and entity categories as a whole for identification, which can solve the problem of overlapping triples and nested entities. In addition, the model has a more global view, achieves consistency between training and prediction goals, and enhances the performance of the model.
[0059] 2. In view of redundant relations, the present invention designs relation filtering and decoding to perform relation screening, thereby reducing redundant relation judgments in entity relation extraction.
[0060] 3. For multi-task learning, the present invention designs an adaptive loss function weight adjustment. For multi-task learning, the optimization of the loss function is very important. The adjustment of the loss function weight can solve the weight imbalance problem caused by the differences between different tasks and improve the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation on the embodiments of the present invention;
[0062] Figure 1 It is a structural diagram of the entity and relationship extraction module in the present invention. DETAILED DESCRIPTION
[0063] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0064] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those within the scope of this description. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0065] Embodiment 1;
[0066] According to the sentence structure characteristics of Chinese datasets, there are many triple overlaps and nested entity problems. The present invention provides the following technical solution: a global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning, and experiments are conducted on Chinese datasets. The method includes the following steps:
[0067] Step 1: Use the Roberta-wwm pre-trained language module as an encoder to extract features from the sentence. Learn the vector representation of each word based on the contextual semantic information. n] is mapped to the input sequence of the Roberta-wwm model, the sequence length is n, and [CLS] and [SEP] are marked at the beginning and end of the sequence respectively. A text S with a sequence length of n is input, and the word vector representation output after Roberta-wwm encoding is shown in formula (1):
[0068] V=([CLS],w1,w2,...,w n ,[SEP])=(h1,h2,...,h n )(1);
[0069] Among them, w i Represents the i-th character of the input sequence, h i Represents the embedding vector corresponding to each character.
[0070] Step 2: Design a global pointer network based on the word vector obtained in step 1 to mark the entity. Use the dot product attention mechanism and relative position RoPE to design the scoring matrix. Introducing RoPE relative position can increase the sensitivity to the length and span of the entity, and determine the entity according to the scoring function. The specific steps are as follows:
[0071] Step 2.1: First, pass the matrix [h1,h2,...,h n ], after the following linear transformation:
[0072] q i,a =w q,α h i +b q,α (2);
[0073] k i,a =w k,α h i +b k,α (3);
[0074] Get the sequence [q 1,α ,q 2,α ,...,q n,α ] and [k 1,α ,k 2,α ,...,k n,α ], the two sequences obtained are the vector sequences used to identify the αth type of entity.
[0075] Step 2.2: Then define the scoring function to determine the entity, as shown in formula (4):
[0076]
[0077] s α (i,j) indicates that the continuous segment from i to j is a score of an entity of type α, and qi,α and k i,α The inner product of is used as the score of the entity fragment type α.
[0078] Step 2.3: Add relative position information, that is, add the transformation matrix R i , satisfying the relationship R i T R j =R j-i , and then applied to q, k, we get the scoring function injected with relative position, as shown in formula (5), and finally output s a (i,j)>0 means all entities.
[0079]
[0080] Step 2.4: Loss function design. For the entity extraction task, a sparse version of the multi-label cross entropy loss function is used, and its formula is expressed as:
[0081]
[0082] Among them, L GP_entity Represents the loss value of entity extraction. M and N are the sets of positive and negative categories, Si represents the predicted i-th value corresponding to the i-th position in the label, e is a natural constant, A=M∪N, and A is the set of positive and negative categories.
[0083] Step 3: Design a relationship filtering and decoding module based on the word vector obtained in step 1. Filter the relationship to enhance the judgment of entity relationships in the global pointer network. The specific steps are as follows:
[0084] Step 3.1: First, pass the matrix [h1,h2,...,h n ] as input, perform pooling operation:
[0085]
[0086] in, is the dimension of the embedding, is the pooling result of the i-th word vector, G is the predefined relationship set {r1,r2,...,r k}, d is the dimension of embedding, k is the size of the relation set, r i For a certain relationship.
[0087] Step 3.2: Then pass the pooling result through a linear layer and a deactivation function to get the relationship probability:
[0088]
[0089] Among them, P r ∈Gk represents the relationship probability of the text about the relationship, Sigmod() represents the nonlinear activation function, and its value range is (0,1), W r ∈G d×k represents the trainable weights, b r ∈G k Represents the bias term.
[0090] Step 3.3: Set a threshold μ to filter redundant relations and model the task of filtering redundant relations as a multi-label binary classification task. r If it is greater than μ, it is considered as relationship filtering, and if it is less than μ, it is considered as a redundant relationship.
[0091] Step 3.4: Loss function design. For the relationship filtering task, the cross entropy loss function is used, and the formula is as follows:
[0092]
[0093] Where L rel represents the loss of the relation filtering decoding module, K represents the total number of relations in the predefined relation set, and P i r Represents the probability corresponding to the i-th relationship.
[0094] Step 4: Entity and relationship extraction. The first and last positions of the subject entity and object entity are identified through the global pointer network designed in step 2 and the relationship filtering in step 3. The specific steps are as follows:
[0095] Step 4.1: Treat the extraction of triples (s, r, o) as five-tuples (s h ,s t ,r,o h ,o t ), that is, <subject head, subject tail, relationship between subject and object, object head, object tail>.
[0096] Step 4.2: Design a scoring function S(s h ,s t ,r,o h ,o t ), during training, the labeled quintuples are set to be greater than zero, and the rest are set to be less than zero. Then all quintuples are enumerated, and the part of quintuples greater than zero is output. Assuming the sentence length is m, there are n relations, and the total number of quintuples is formula (10):
[0097]
[0098] Step 4.3: Since the total number of quintuples in formula (10) is huge, it is difficult to list S(s h ,s t ,r,o h ,o t )>0, so we use the following decomposition strategy to convert S(s h ,s t ,r,o h ,o t ) The scoring function is decomposed to simplify the calculation:
[0099] S(s h ,s t ,r,o h ,o t )=S(s h ,s t )+S(o h ,o t )+S(s h ,o h |r)+S(s t ,o t |r)(11);
[0100] Where S(s h ,s t ) and S(o h ,o t ) represent the first and last scores of the subject entity and the object entity respectively. If both are greater than zero, the subject entity and the object entity can be determined. h ,o h |r) indicates that the relationship matching is performed using the header features of the subject and object entities as representations. If there are no nested entities, when S(s h ,o h |r)>0, then all relationships can be determined; if there are nested entities, then S(s t ,o t |r), i.e., the tail features of the subject and object entities.
[0101] Step 4.4: Loss function design. The entity relationship extraction task also uses the sparse multi-label cross entropy loss function. At this time, entity extraction, head extraction of the subject and object of a specific relationship, and tail extraction of the subject and object of a specific relationship are three different tasks, and multi-task learning is performed with the relationship filtering task. The dynamic weighted average method is used to allow the four tasks to learn at a similar speed. The relative decline rate is expressed by the change ratio of the loss function during training, reflecting the difficulty of learning the task. It is believed that the task with a faster loss decline is easier to learn, so the task with a slower learning speed has a higher weight to balance different tasks.
[0102] The loss function of the entity and relationship extraction module is:
[0103]
[0104] Among them, L GP_head Represents the loss value of the subject entity and object entity head based on a specific relationship, L GP_tail Represents the loss value of the tail of the subject entity and the object entity based on a specific relationship; M and N are the sets of positive and negative categories, Si represents the predicted i-th value corresponding to the i-th position in the label, e is a natural constant, A=M∪N, A is the set of positive and negative categories;
[0105] The loss function of the entity and relationship extraction module is optimized as follows:
[0106]
[0107] Among them, S k (t-1) represents the relative rate of decrease of the loss function, L k (t-1) represents the current loss function value, L k (t-2) represents the previous loss function value, w k (t) represents the weight value of each task, X represents the temperature coefficient used to adjust the task distribution, the larger the P is, the more evenly each task is distributed, and K is the set value, which represents a certain value in order to make the weight move within a range.
[0108] Step 5: Result evaluation. The entity relationship quintuple information is obtained through the above steps. Finally, experiments are conducted on the Chinese data set. The precision P, recall R and F1 value are used to evaluate the results. The higher the value, the more accurate the entity relationship quintuple information extraction is. The calculation formula is as follows:
[0109]
[0110] Among them, TP represents the number of correct predictions, FP represents the number of wrong predictions, and FN represents the number of incorrect predictions.
[0111] Embodiment 2:
[0112] Based on the first embodiment, the present invention is further described below in conjunction with the accompanying drawings and specific implementation methods.
[0113] The overall structure of the global pointer network entity relationship extraction model that integrates relationship filtering and multi-task learning proposed in this invention is as follows: Figure 1 As shown, it includes the following four modules: Roberta-wwm encoding module, entity extraction module, relationship filtering module and entity relationship prediction module.
[0114] 1.Roberta-wwm encoding module:
[0115] The Roberta-wwm pre-trained language module is used as an encoder to extract the features of the sentence. The vector representation of each word is learned based on the contextual semantic information. The original text S = [w1, w2, ..., w n ] is mapped to the input sequence of the Roberta-wwm model, the sequence length is n, and [CLS] and [SEP] are marked at the beginning and end of the sequence respectively. A text S with a sequence length of n is input, and the word vector representation output after Roberta-wwm encoding is shown in formula (17):
[0116] V=([CLS],w1,w2,...,w n ,[SEP])=(h1,h2,...,h n )(17)
[0117] Among them, wi represents the i-th character of the input sequence, and hi represents the embedding vector corresponding to each character.
[0118] 2. Entity extraction module:
[0119] The word vector obtained from module 1 is used to design a global pointer network for entity extraction. The global pointer module considers the head and tail of an entity as a whole to judge, which can solve the problem of nested entities and is more global. The dot product attention mechanism and relative position RoPE are used to design the scoring matrix. The introduction of RoPE relative position can increase the sensitivity to the length and span of the entity. Figure 1 For a sentence of length n, m n×n span matrices are constructed, where m is the entity category and different types of entities are marked in different matrices. The rows marked as 1 in the matrix are marked as entity heads, and the corresponding columns are marked as entity tails. Therefore, the span{x1}{x2} entity marked in the figure is x1x2 and the span{x4}{x6} entity marked is x4x5x6. Based on this module, the scoring function S(s h ,s t),S(o h ,o t ) is greater than zero to extract all subjects and objects. The specific mathematical form is: the matrix [h1,h2,...,h n ], after the following transformation:
[0120] q i,a =w q,α h i +b q,α (18);
[0121] k i,a =w k,α h i +b k,α (19);
[0122] Get the sequence [q 1,α ,q 2,α ,...,q n,α ] and [k 1,α ,k 2,α ,...,k n,α ], which is the vector sequence used to identify the αth type of entity, and the scoring function is defined as shown in formula (20):
[0123]
[0124] s α (i,j) indicates that the continuous segment from i to j is a score of an entity of type α, and q i,α and k i,α The inner product of is used as the score of the entity segment type α. Then the relative position information is added, that is, the transformation matrix Ri is added to satisfy the relationship R i T R j =R j-i , and then apply it to q, k to get the scoring function injected with relative position, and finally judge the entity according to this scoring function:
[0125]
[0126] 3. Relationship filtering module:
[0127] The word vector obtained by the encoding module is used to design the relationship filtering and decoding module. The relationship is filtered to enhance the judgment of entity relationships in the global pointer network. The matrix [h1,h2,...,h n ] As input, first perform a pooling operation:
[0128]
[0129] in, is the dimensionality of the embedding.
[0130] The pooling result is then passed through a linear layer and a nonlinear activation function to obtain the relationship probability:
[0131]
[0132] Among them, P r ∈G k represents the probability of the text about the relationship, Sigmod() represents the nonlinear activation function, and its value range is (0,1), W r ∈G d×k represents the trainable weights, b r ∈G k Represents the bias term.
[0133] Finally, a threshold μ is set to filter redundant relations, which is modeled as a multi-label binary classification task. If the relationship probability Pr is greater than μ, it is considered as relationship filtering, and if it is less than μ, it is considered as redundant relationship. Finally, the relationship filter set is output.
[0134] 4. Entity relationship extraction module:
[0135] A global pointer network is designed through the entity relationship extraction module and the relationship filtering module to identify the head and tail positions of the subject entity and the object entity.
[0136] Consider the extraction of triples (s, r, o) as five-tuples (s h ,s t ,r,o h ,o t ), that is, <subject head, subject tail, relation, object head, object tail>. Design a five-tuple scoring function S(s h ,s t ,r,o h ,o t ), during training, the labeled quintuples are set to be greater than zero, and the rest are set to be less than zero. Then all quintuples are enumerated, and the part of quintuples greater than zero is output. Assuming the sentence length is m, there are n relations, and the total number of quintuples is formula (8):
[0137]
[0138] To simplify the above formula, we can use the following decomposition strategy:
[0139] S(s h ,s t ,r,oh ,o t )=S(s h ,s t )+S(o h ,o t )+S(s h ,o h |r)+S(s t ,o t |r) (25);
[0140] Where S(s h ,s t ) and S(o h ,o t ) represent the first and last scores of the subject entity and the object entity respectively. If both are greater than zero, the subject entity and the object entity can be determined. h ,o h |r) indicates that the relationship matching is performed using the header features of the subject and object entities as representations. If there are no nested entities, when S(s h ,o h |r)>0, then all relationships can be determined; if there are nested entities, then S(s t ,o t |r), that is, the tail features of the subject and object entities. Figure 1 The corresponding entities of the relation r1 are x1x2 and x4x5x6.
[0141] 5. Loss function design. Design loss functions for each module to train the model.
[0142] The loss function of the entity and relationship extraction module is:
[0143]
[0144] Among them, L GP_head Represents the loss value of the subject entity and object entity head based on a specific relationship, L GP_tail Represents the loss value of the tail of the subject entity and the object entity based on a specific relationship; M and N are the sets of positive and negative categories, Si represents the predicted i-th value corresponding to the i-th position in the label, e is a natural constant, A=M∪N, A is the set of positive and negative categories;
[0145] The loss function of the entity extraction task is:
[0146]
[0147] Among them, L GP_entity Represents the loss value of entity extraction.
[0148] For the entity relationship extraction task, it is divided into three subtasks: entity extraction, entity head extraction based on a certain relationship, and entity tail extraction based on a certain relationship. All of them use the sparse version of the multi-label cross entropy loss function. The multi-label cross entropy loss function compares the scores of the target category with the non-target category to balance the weight of each category and solve the problem of category imbalance. In addition, the sparse version reduces the transmission cost due to the excessive number of negative categories. Only the subscript corresponding to the positive category needs to be transmitted each time. The formula is expressed as:
[0149]
[0150] Among them, L GP S represents the loss of the global pointer network decoding module. M and N are the sets of positive and negative categories. A=M∪N and M are used to calculate the loss corresponding to the negative category, while the loss of the positive category remains unchanged. i Indicates that the predicted i-th value corresponds to the i-th position in the label.
[0151] For relationship filtering tasks, the cross entropy loss function is used
[0152]
[0153] Among them, L rel represents the loss of the relation filtering and decoding module, K represents the total number of relations in the predefined relation set, and Pir represents the probability corresponding to the i-th relation.
[0154] In order to solve the loss balance problem of multi-task learning, this paper adopts a dynamic weighted average method to let the four tasks learn at a similar speed. The relative decline rate is expressed by the change ratio of the loss function during training, thereby reflecting the learning difficulty of the task. It is believed that the task with a faster loss decline is easier to learn, so the task with a slower learning speed has a higher weight. The formula is as follows:
[0155]
[0156] Among them, S k (t-1) represents the relative rate of decrease of the loss function, L k (t-1) represents the current loss function value, L k (t-2) represents the previous loss function value. k (t) represents the weight value of each task, X represents the temperature coefficient used to adjust the task distribution. The larger the X, the more evenly each task is distributed. K represents a certain value to make the weight move within a range.
[0157] The final loss value is shown in formula (30):
[0158] Loss = wk (t)L GP_entity +w k (t)L GP_head +w k (t)L GP_tail +w k (t)L rel (30)
[0159] Among them, L GP_entity Represents the loss function value of entity extraction, L GP_head Represents the head loss value of the subject entity and the object entity based on a specific relationship, L GP_tail It represents the loss value of the tail of the subject entity and the object entity based on a specific relationship. The three loss value calculation formulas all use formula 26. L rel That is, it represents the loss value of the relationship filtering module, and the formula is shown in 27.
[0160] experiment:
[0161] In the present application study, in order to verify the effectiveness of the model of the present invention in Chinese entity relationship extraction, experiments were conducted on the Chinese public data set Chinese medical information extraction data set CMeIE and the relationship extraction data set DuIE2.0 provided by Baidu to evaluate the model of the present invention. The CMeIE data set is a Chinese medical information extraction data set released in CHIP2020. Its main sources are pediatric diseases and hundreds of common diseases, and it contains nearly 75,000 triples. The total number of its training set and validation set is 17,924, which are randomly divided to obtain training sets, validation sets, and test sets. In addition, in order to further verify the effectiveness of the model of the present invention on Chinese data sets in other fields, the DuIE2.0 data set is used. This data set is a data set for the 2020 Language and Intelligent Technology Competition. Its main sources are Baidu Encyclopedia and Baidu News, containing nearly 430,000 triple data, 210,000 Chinese sentences, and 48 predefined relationship types. The specific statistical analysis of the two data sets is shown in Table 1.
[0162] Table 1 Statistical analysis of the data set
[0163] Dataset Training set Test Set Validation set total Number of relationships DuIE2.0 137258 34044 30485 201787 48 CMeIE 11651 3584 2689 17924 53
[0164] Both of these data sets have complex overlapping and nesting problems. Therefore, the effectiveness of the model of the present invention based on the two problems of relationship overlap and entity nesting can be well verified by using these two data sets. The overlapping triple types mainly include normal class (Normal), entity pair overlap (EntityPairOverlap, EPO) and single entity overlap (SingleEntityOverlap, SEO). The examples of triple overlap and nested entity (Subject Object Overlap, SOO) are shown in Table 2.
[0165] Table 2 Three types of overlapping triples
[0166]
[0167] Experimental setup:
[0168] The results are evaluated using precision P, recall R, and F1 value. The higher the value, the more accurate the extraction of entity relationship quintuple information. The calculation formula is as follows:
[0169]
[0170] Among them, TP represents the number of correct predictions, FP represents the number of wrong predictions, and FN represents the number of incorrect predictions.
[0171] This experimental environment uses python3.8 and the Pytorch framework in deep learning. Model training and testing are mainly completed on the Windows operating system, 64GB running memory, and NVIDIA GeForce RTX 4090 graphics card. According to the laboratory hardware conditions, the Roberta-wwm model is selected in the experiment to obtain the sequence feature vector containing context information, and the Adam optimizer is used. The specific parameter values in the model are shown in Table 2.
[0172] Table 3 Statistics of model parameter values
[0173]
[0174] Experimental results analysis:
[0175] The experimental results of the comparison between the model of the present invention and several existing baseline models on the CMeIE dataset and the DuIE dataset are shown in Table 5. The training results in Table 5 show that the model of the present invention is effective on the CMeIE dataset, and obtains a precision of 57.7%, a recall of 50.6% and an F1 value of 53.5%. By comparison, it is found that the entity relationship joint extraction model is significantly better than the traditional pipeline method (NovelTagging), which also proves that the pipeline method has error propagation. At the same time, the model CasRel for the entity overlap problem converts the relationship into a function that maps the head entity to the tail entity, which can accurately identify more entities in the sentence, but cannot identify nested entities, so the accuracy rate obtained is lower than that of the model of the present invention. For TPLinker_plus, its annotation method is similar to the model of the present invention, but its model structure is more complex, which will cause a waste of resources and ultimately obtain a lower recall rate and F1 value. PRGC draws on the advantages of CasRel and TPLinker, and proposes relationship filtering. The model of the present invention is inspired by the prediction of its relationship filtering and applies it to the model of the present invention, but PRGC still has the shortcomings of TPLinker. The complex model structure easily causes waste of resources, and the final training time is long and the cost is high, resulting in low effect. The model of the present invention incorporates relationship filtering on the basis of GPLinker, reduces the judgment of redundant relationships, and adopts a multi-task learning strategy. By adjusting the weights of the loss functions of different tasks to balance each task, it finally obtains better accuracy, recall rate and F1 value. Compared with other baseline models, the F1 value has increased by 27.9%, 9.1%, 3.5%, and 2.2%, and compared with the original GPLinker model, it has also increased by 1.4%. In addition, from the analysis of experimental results, the model performance on the CMeIE dataset is generally low. The reason is that the dataset is a specific dataset in the medical field. The Chinese-BERT-WWM pre-trained language model used in the present invention is obtained after training and learning on a large-scale general corpus. The data in the medical field is highly professional, and the structural features of the data that are not identified by statistical analysis cannot be correctly identified for some medical professional symbols and medical terminology, which brings difficulties to the model's semantic understanding of the text. As shown in Table 4, proper nouns such as "submucous cleft palate" and "airway obstruction" cannot be recognized, resulting in inaccurate triple predictions, and because Chinese-BERT-WWM is a model for Chinese word segmentation, some English names cannot recognize their synonyms, resulting in a decrease in accuracy. Therefore, the Chinese-BERT-WWM model may have poor representation on medical field data, resulting in the incorrect extraction or non-extraction of more triples.
[0176] Table 4 Examples of incorrectly predicted triples
[0177] The correct triplet was not predicted Example (disease, related(symptoms), disease) (Submucous cleft palate, associated (symptoms), airway obstruction) (symptoms, synonyms, symptoms) (Gastritis, synonyms, gastritis) (disease, symptoms after treatment, symptoms) (Osteoarthritis, Post-treatment symptoms, Fatigue)
[0178] Therefore, in order to further verify the effectiveness of the model of the present invention on the Chinese data set, a comparative experiment was continued on the DuIE2.0 data set with other baseline models, and the experimental results are shown in Table 5. The training results show that the model of the present invention is effective on the DuIE2.0 data set, achieving a precision of 65.5%, a recall of 81.9% and an F1 value of 72.7%, which is 15.5%, 4.5%, 2.2% and 1.7% higher than the F1 value of other baseline models, and 1.6% higher than the original GPLinker model. The original GPLinker model is superior to other baseline models in terms of model structure, training strategy, training speed and training results, so the improvement of the GPLinker model by the present invention has improved and proved the significance of this experiment.
[0179] Table 5 Model comparison experimental results
[0180]
[0181]
[0182] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0183] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning, characterized in that: The method comprises: Step 1: Get the original text, map it to the input sequence of the pre-trained language module, mark the beginning and end of the input sequence, and use the pre-trained language module as an encoder to extract features from the sentences in the original text to obtain the output word vector; Step 2: Perform linear transformation on the word vector obtained in step 1 to obtain the transformed sequence, construct a first scoring function based on the transformed sequence, add relative position information to the first scoring function to obtain a second scoring function, construct a loss function for the entity extraction task, construct a global pointer network based on the second scoring function, train the global pointer network, and use the global pointer network for entity extraction; Step 3: Based on the word vector obtained in step 1, a pooling operation is performed to obtain a pooling result. The pooling result is passed through a linear layer and a non-activation function to obtain a relationship probability. A threshold is set to filter redundant relationships. A corresponding second loss function is constructed for the relationship filtering task to obtain a relationship filtering decoding module. The relationship filtering decoding module is trained. The relationship filtering decoding module is used to filter relationships. Step 4: Construct an entity and relationship extraction module based on the global pointer network and the relationship filtering and decoding module, process the text to be processed based on the entity and relationship extraction module, identify the first and last positions of the subject entity and the object entity respectively, and the relationship between the subject entity and the object entity to obtain the recognition result, and obtain the entity and relationship extraction result corresponding to the text to be processed based on the recognition result; Step 2 specifically includes: Based on the word vectors [h1,h2,...,h n ] is subjected to linear transformation to obtain the transformed sequence, and the sequence [q 1,α ,q 2,α ,...,q n,α ] and [k 1,α ,k 2,α ,...,k n,α ], sequence [q 1,α ,q 2,α ,...,q n,α ] and [k 1,α ,k 2,α ,...,k n,α ] is the vector sequence used to identify the αth type of entity, and the linear transformation method is: q i,a =w q,α h i +b q,α ; k i,a =w k,α h i +b k,α ; Among them, 1≤i≤n, w q,α is the weight parameter of the current linear transformation, b q,α is the bias term, w k,α is the weight parameter of the current linear transformation, b k,α is the bias term; The first scoring function for determining the entity is constructed based on the transformed sequence. The first scoring function is: Among them, s α (i,j) indicates that the continuous segment from i to j is a score of an entity of type α, and q i,α and k i,α The inner product of is used as the score of the entity fragment type α; The relative position information is added to the first scoring function to obtain the second scoring function, which is: Where T is the transpose matrix operator, R i and R j is the transformation matrix, which is used to increase the relative position information to satisfy the relationship Construct the loss function of the entity extraction task. The loss function of the entity extraction task is: Among them, L GP_entity Represents the loss value of entity extraction, M and N are the sets of positive and negative categories, Si represents the i-th position in the label corresponding to the predicted i-th value, e is a natural constant, A=M∪N, A is the set of positive and negative categories.
2. The global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning according to claim 1 is characterized in that: Step 4 specifically includes: A quintuple scoring function is constructed, and based on the quintuple scoring function, quintuple entity and relationship extraction is performed on the processed text to obtain entity and relationship extraction results. The quintuple entities and relationships include: subject entity head, subject entity tail, relationship between subject entity and object entity, object entity head and object entity tail.
3. The global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning according to claim 2 is characterized in that: The quintuple is (s h ,s t ,r,o h ,o t ), the scoring function of the quintuple is: S(s h ,s t ,r,o h ,o t )=S(s h ,s t )+S(o h ,o t )+S(s h ,o h |r)+S(s t ,o t |r); Among them, S(s h ,s t ) and S(o h ,o t ) represent the first and last scores of the subject entity and the object entity respectively. If both are greater than zero, the subject entity and the object entity are judged; S(s h ,o h |r) indicates that the relationship matching is performed using the head features of the subject entity and the object entity as representations. If there is no nested entity phenomenon where one entity contains another entity, and when S(s h ,o h |r)>0, all relationships are determined; if there is a nested entity phenomenon where one entity contains another entity, then S(s t ,o t |r), S(s t ,o t |r) is the tail feature of the subject entity and the object entity.
4. The global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning according to claim 2 is characterized in that: The step 4 also includes constructing a loss function for the entity and relationship extraction module. The loss function of the entity and relationship extraction module is: Among them, L GP_head Represents the loss value of the subject entity and object entity head based on a specific relationship, L GP_tail Represents the loss value of the tail of the subject entity and the object entity based on a specific relationship; M and N are the sets of positive and negative categories, Si represents the predicted i-th value corresponding to the i-th position in the label, e is a natural constant, A=M∪N, A is the set of positive and negative categories; And optimize the loss function of the entity and relationship extraction module in the following way: Among them, S k (t-1) represents the relative rate of decrease of the loss function, L k (t-1) represents the current loss function value, L k (t-2) represents the previous loss function value, w k (t) represents the weight value of each task, X represents the temperature coefficient used to adjust the task distribution, and K is the set value.
5. The global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning according to claim 1 is characterized in that: Step 1 specifically includes: The Roberta-wwm pre-trained language module is used as an encoder to extract the features of the sentence, and the vector representation of each word is learned according to the contextual semantic information. The original text S = [w1, w2, ..., w n ] is mapped to the input sequence of the Roberta-wwm model, the sequence length is n, and [CLS] and [SEP] are marked at the beginning and end of the sequence respectively. A text S with a sequence length of n is input, and the word vector output after Roberta-wwm encoding is expressed as: V=([CLS],w1,w2,...,w n ,[SEP])=(h1,h2,...,h n ); Among them, w i Represents the i-th character of the input sequence, h i Represents the embedding vector corresponding to each character, 1≤i≤n.
6. The global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning according to claim 1 is characterized in that: Step 3 specifically includes: Based on the word vectors [h1,h2,...,h n ] performs pooling operation to obtain the pooling result, 1≤i≤n, and the pooling operation is expressed as: in, is the pooling result of the i-th word vector, G is the predefined relationship set {r1,r2,...,r k }, d is the dimension of embedding, k is the size of the relation set, r i for a relationship; The pooling result is passed through a linear layer and a deactivation function to obtain the relationship probability, which is calculated as follows: Among them, P r ∈G k represents the relationship probability of the text about the relationship, Sigmod() represents the nonlinear activation function, and its value range is (0,1), W r ∈G d×k represents the trainable weights, b r ∈G k represents the bias term; Set the threshold μ to filter redundant relations. If the relation probability P r If it is greater than μ, it is considered as relationship filtering, and if it is less than μ, it is considered as redundant relationship; Constructing a corresponding second loss function for the relationship filtering task, obtaining a relationship filtering decoding module, and training the relationship filtering decoding module, wherein the relationship filtering decoding module is used to screen and filter the relationship; The second loss function is: Among them, L rel represents the loss of the relation filtering and decoding module, K represents the total number of relations in the predefined relation set, Represents the probability corresponding to the i-th relationship, y i is the true label of relation r.
7. The global pointer network entity relationship extraction method integrating relationship filtering and multi-task learning according to claim 1 is characterized in that: The method further comprises step 5: The entity and relationship extraction results obtained in step 4 are experimented on the Chinese dataset, and the precision P, recall R and F1 value are used to evaluate the results.
