Relation extraction algorithm based on three-dimensional network

Abstract

The invention discloses a relation extraction algorithm based on a three-dimensional network. The method comprises the following steps of: firstly, encoding an input text by using a pre-training language model as a text encoder; secondly, designing a three-dimensional network as an entity pair extractor for calculating the probability that each element in the head-tail cube serves as a head-tail entity pair, and comparing the probability with a predefined threshold to obtain a group of candidate head-tail entity pairs; then, marking each head-tail entity pair in all relation matrixes of the triple cube, and obtaining context representation of each candidate head-tail entity pair based on a shared text encoder; and finally, another new three-dimensional network is designed as a triple extractor for determining the relationship between each head-tail entity pair, and a final triple result is obtained. The method is suitable for a relation extraction task, and by designing a novel three-dimensional network framework combining entity and relation extraction, the extraction precision of the model triad is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, specifically to the technical field of text information extraction, and more specifically to providing a relationship extraction algorithm based on a stereo network. Background Art

[0002] As one of the important tasks in the field of natural language processing, the relation extraction task has become a key link in text understanding and information extraction. Its goal is to extract structured knowledge from unstructured text, thereby helping machines to understand human natural language more accurately. However, with the continuous expansion of data scale and the complexity of application scenarios, traditional relation extraction methods face performance bottlenecks and insufficient generalization capabilities. Therefore, relation extraction based on pre-trained language models came into being, aiming to make full use of the rich semantic information in large-scale corpora to improve the generalization ability and accuracy of the model.

[0003] Existing methods for relation extraction tasks can be mainly divided into two categories: pipeline-based extraction methods and joint extraction methods. Pipeline-based extraction methods use sequence labeling methods to identify all potential entities in a given text, and then use classifiers to determine the most appropriate relationship for each pair of head and tail entities. However, since these two stages are trained independently, they cannot share information during triple extraction, which limits the performance of the model. More importantly, such methods are prone to error propagation because errors in early stages cannot be corrected in subsequent stages. To address these problems, researchers have proposed joint extraction methods, which aim to extract entities and relations in one stage at the same time. Joint extraction methods can be further divided into three categories: 1) multi-task learning, 2) conversion task paradigm (for example, converting the relation extraction task into a multi-round question-answering task or a text generation task), and 3) table filling methods.

[0004] In recent years, table filling methods have achieved state-of-the-art performance in joint relation extraction tasks. Although existing table filling methods have achieved certain success in joint relation extraction tasks, they still have problems in efficiency and accuracy, especially in accuracy. The core idea of ​​current table filling methods is to use a relation-based two-dimensional matrix to label head and tail entity pairs. This type of labeling scheme aims to use multiple elements in the two-dimensional matrix to label a head and tail entity pair. This method requires a large number of labels to be marked for the head and tail entity pairs in the matrix, resulting in a surge in the number of marked label combinations, resulting in longer decoding time and lower efficiency. At the same time, this process will produce many invalid label combinations, which undoubtedly affect the accuracy. In addition, extracting nested entity pairs is also a severe challenge. When extracting nested entities, a single element in the two-dimensional matrix needs to correspond to multiple labels, which makes it difficult to accurately capture the semantic representation of the element during training, which seriously affects the accuracy of the extraction process.

[0005] In summary, a new relationship extraction algorithm needs to be designed to solve the above problems. Summary of the invention

[0006] In view of this, the present invention provides a relation extraction algorithm based on stereo network. First, a novel cube labeling scheme is designed. Its most notable feature is the use of a single element to label a head-tail entity pair, ensuring high efficiency and effectiveness. Second, a relation extraction method based on stereo network is proposed, and a new stereo network for joint entity and relation extraction is designed, with two different stereo networks: one for identifying head-tail entity pairs and the other for judging their relationship.

[0007] The technical route of the present invention is implemented in the form of: first, use a pre-trained language model as a text encoder to encode the input text. Then, design a stereo network as an entity pair extractor to identify a set of candidate head-tail entity pairs. Then, mark each head-tail entity pair in all relationship matrices of the predefined triple cube. Finally, design another new stereo network as a triple extractor to determine the relationship between the head-tail entity pairs and obtain the final triple result.

[0008] The specific steps are:

[0009] S1. Encode the input text using the pre-trained language model as a text encoder.

[0010] S2, design the head-tail cube and triple cube to mark the head-tail entity pairs and triples respectively;

[0011] S3. Design a stereo network as an entity pair extractor to calculate the probability of each element in the head-tail cube being a head-tail entity pair, and compare it with the threshold to identify a set of candidate head-tail entity pairs;

[0012] S4. Through the shared text encoder, the semantic vector representation of the entity pair in all the relationship matrices of the triple cube is obtained; and another new cubic network is designed as a triple extractor to determine the relationship between each head and tail entity pair and obtain the final triple result.

[0013] S11. Take the text C as input and use the context-aware pre-trained language model BERT to output context-aware representations for each input token:

[0014] H=PLM([ <cls>,c1,c2,…,c n , <sep>]),

[0015] in <cls>and <sep>It is a special token in the BERT model.

[0016] S21. Design a cube as a head-tail cube, where each different head-tail entity pair is uniquely associated with a specific element in the head-tail cube;

[0017] S22. Design a new cube as a triple cube, where each different triple can be represented by one or more elements in the cube.

[0018] S31. Based on the head-tail cube, design a stereo network as an entity pair extractor. Construct m matrices M∈R n ×n , for the i-th matrix, each element Indicates the probability that the starting character of the head entity and the tail entity are located at the jth and kth positions respectively. It can be expressed as:

[0019]

[0020] Among them, W1, W2 and b1, b2 are trainable weight parameters. j , R k is the rotation matrix;

[0021] S32. Define the loss function of the entity pair extractor, which is used to learn the parameters in the entity pair extractor:

[0022]

[0023] Among them, Ω = {(j,k) | 1 ≤ j, k ≤ n} is the set of all possible head-tail entity pairs in the i-th matrix, Contains all true head-tail entity pairs;

[0024] S33, comparing the probability of each element in the head-tail cube being a head-tail entity pair with a predefined threshold, and those exceeding the threshold are selected as candidate head-tail entity pairs;

[0025] S34. For the obtained set of head and tail entity pairs, convert them into quadruplets with clearer entity boundary information. S41. For the set of entity pair tags (h_s, h_e, t_s, t_e) obtained by the entity pair extractor, obtain the semantic vector representation of the entity pair in all relationship matrices of the triple cube through the shared text encoder:

[0026]

[0027] Among them, M′ i,j The vector representing the i-th head entity tag and the j-th tail entity tag has the training parameters of another stereo network; S42. Based on the triple cube, design a stereo network as a triple extractor. Construct l matrices M′∈R n×n , M′ i,j Represents the vector of the i-th head entity label and the j-th tail entity label obtained by the head-tail cube. i,j Input into a multilayer perceptron to obtain the probability of entity pairs being associated with various relationships:

[0028]

[0029] Among them, W r and b r is a trainable weight, σ represents the sigmoid function;

[0030] S43. Objective optimization function of triple extractor and objective function of entity extractor Same, used to learn the parameters in the triplet extractor;

[0031] S44. In the reasoning stage, the head and tail entity pairs are combined with the top-1 relation type obtained by the corresponding relation extractor to form the final triple. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall framework of the relationship extraction framework based on the three-dimensional network of the present invention;

[0033] Figure 2 It is a flow chart of the relationship extraction framework based on the three-dimensional network of the present invention; DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solutions of the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more obvious and understandable, the present invention is further described in detail below in conjunction with embodiments and accompanying drawings of the embodiments.

[0035] See also Figure 1 , Figure 1 It is a schematic diagram of the overall framework of the relationship extraction framework based on the stereo network of the present invention. The method includes: first, using a pre-trained language model as a text encoder to encode the input text; second, designing a head-tail cube, the head-tail cube represents all head-tail entity pair combinations in the input text, and each different head-tail entity pair is uniquely associated with a specific element in the head-tail cube. Design a triple cube, the triple cube represents all candidate triples, and each different triple can be represented by one or more elements in the triple cube; then, design a stereo network as an entity pair extractor, which is used to calculate the probability of each element in the head-tail cube as a head-tail entity pair, and compare it with a predefined threshold to obtain a set of candidate head-tail entity pairs; then, mark each head-tail entity pair in all relationship matrices of the triple cube, and obtain the contextual representation of each head-tail entity pair based on the shared text encoder. In addition, another stereo network is designed as a triple extractor to determine the relationship between head-tail entity pairs.

[0036] See also Figure 2 , Figure 2 It is a flow chart of the relationship extraction framework based on the three-dimensional network of the present invention, and the specific steps include:

[0037] S1. Input text, use the pre-trained language model as a text encoder to encode the input text to obtain the encoding of the input text;

[0038] S11. Given a text C, C = [c1, c2, ..., c n ], using the context-aware pre-trained language model BERT to output a context-aware representation for each input token ci:

[0039] H=PLM([ <cls>,c1,c2,…,c n , <sep>]),

[0040] in <cls>and <sep>It is a special marker in the BERT model. PLM represents the pre-trained language model. H = [h <cls>< / cls> ,h1,h2,…,h n ,h <sep>< / sep> ] is the context representation of text C, h i is the contextual representation of the i-th character in text C.

[0041] S2. Design a head-tail cube, where each different head-tail entity pair is uniquely associated with a specific element in the head-tail cube; design a triple cube, where each different triple can be represented by one or more elements in the triple cube;

[0042] S21. Design a cube As a head-tail cube, where n represents the length of the input text, m represents the number of all head entity and tail entity length combinations, each different head-tail entity pair can be uniquely associated with a specific element in the cube, expressed as (h_s, t_s, [h_len, t_len]); h_s of the first dimension and t_s of the second dimension are the starting character positions of the head entity and the tail entity, respectively; correspondingly, h_len and t_len of the third dimension represent the lengths of the head entity and the tail entity, respectively;

[0043] S22. Design a cube As a triple cube, where n and l represent the length of the input text and the number of predefined relations, respectively, each different triple can be represented by one or more elements in the cube. Figure 2 As shown, the triple "(Melanie joyceon, r1, dallas)" can be composed of the elements "(Melanie, r1, dallas)" and "(joyceon, r1, dallas)", where r1 is the relationship between "Melanie joyceon" and "dallas".

[0044] S3. Design a stereo network as an entity pair extractor to calculate the probability of each element in the head-tail cube being a head-tail entity pair, and compare it with a predefined threshold to obtain a set of candidate head-tail entity pairs;

[0045] S31. Based on the head-tail cube, design a stereo network as an entity pair extractor. Construct m matrices M∈R n ×n , n represents the length of the input text, m represents the number of all head entity and tail entity length combinations, for the i-th matrix, each element Indicates the probability that the starting characters of the head entity and the tail entity are located at the jth and kth positions respectively (the lengths of the head entity and the tail entity are represented as h_len and t_len respectively). Based on the contextual representation of the input text, It can be expressed as:

[0046]

[0047] Among them, W1, W2 and b1, b2 are trainable weight parameters. j , R k are rotation matrices, which are used to incorporate relative position information into h j and h k middle;

[0048] S32. Define the loss function of the entity pair extractor, which is used to learn the parameters in the entity pair extractor:

[0049]

[0050] Where e represents a natural constant, Ω = {(j,k)|1≤j,k≤n} is the set of all possible head-tail entity pairs in the i-th matrix, Ω P Contains all true head-tail entity pairs;

[0051] S33, using an entity pair extractor to calculate the probability of each element in the head-tail cube being a head-tail entity pair, and compare it with a predefined threshold, and those exceeding the threshold are selected as candidate head-tail entity pairs;

[0052] S34. For a set of head and tail entity pairs (g_s, t_s, [h_len, t_len]) obtained, convert them into a four-tuple with clearer entity boundary information, namely (h_s, h_e, t_s, t_e). h_s and h_e represent the starting and ending character positions of the head entity, respectively, and t_s and t_e represent the starting and ending character positions of the tail entity, respectively. This conversion can be achieved by the formula e=s+len-1, where s, e and len are the starting character position, ending character position and length of the entity, respectively.

[0053] S4. Based on the candidate head and tail entity pairs obtained by the entity pair extractor, the semantic vector representation of the entity pairs in all relationship matrices of the triple cube is obtained through the shared text encoder; and another new cubic network is designed as a triple extractor to determine the relationship between each head and tail entity pair and obtain the final triple result;

[0054] S41. For a set of entity pair tags (h_s, h_e, t_s, t_e) obtained by the entity pair extractor, obtain the semantic vector representation of the entity pair in all relationship matrices of the triple cube through the shared text encoder:

[0055]

[0056] Among them, M′ i,j The vector representing the i-th head entity tag and the j-th tail entity tag has the same training parameters as the entity pair extractor;

[0057] S42. Based on the triple cube, design a three-dimensional network as a triple extractor. Construct l matrices M′∈R n×n (n represents the length of the input text, l represents the number of predefined relations), M′ i,j Represents the vector of the i-th head entity label and the j-th tail entity label obtained by the head-tail cube. i,j Input into a multilayer perceptron to obtain the probability of entity pairs being associated with various relationships:

[0058]

[0059] Among them, W r and b r are trainable weights, is the semantic vector representation of the candidate entity pair in all relation matrices of the triple cube, and σ represents the sigmoid function;

[0060] S43. Objective optimization function of triple extractor and objective optimization function of entity extractor Same, used to learn the parameters in the triplet extractor;

[0061] S44. In the reasoning stage, the head and tail entity pairs obtained by the entity pair extractor are combined with the highest probability relationship type obtained by the corresponding relationship extractor to form the final triple.

[0062] The present invention has conducted extensive experiments on five public datasets. The results on the Chinese relation extraction dataset CMeIE-V2 are shown in Table 1. From the table we can conclude that: 1) The method of the present invention outperforms all baselines in F1 score, proving the effectiveness of the proposed method. In particular, the method of the present invention exceeds the current optimal GPLinker method by 1.36% in F1 score. 2) Although GPLinker is slightly higher than the method of the present invention in precision, it lags behind significantly in recall. This difference is mainly due to the fact that the entity length in CMeIE-V2 is usually longer, resulting in our head and tail cubes having abnormally many dimensions on the depth axis, resulting in an abnormally sparse cube. Therefore, each layer corresponding to the depth axis of the cube has less data, which poses a challenge to training an efficient model.

[0063] Table 1 Model comparisons on CMeIE-V2 dataset

[0064]

[0065] < / sep> < / cls> < / sep> < / cls> < / sep> < / cls> < / sep> < / cls>

Claims

1. A relationship extraction algorithm based on stereo network, characterized by The following steps are involved: S1. Input text, use the pre-trained language model as a text encoder to encode the input text to obtain the encoding of the text; S2. Design a head-tail cube, which is used to mark all combinations of head-tail entity pairs in the input text, and each different head-tail entity pair is uniquely associated with a specific element in the head-tail cube; design a triple cube, which is used to mark all possible triples, and each different triple can be represented by one or more elements in the triple cube; S3. Based on the context representation obtained by the text encoder, a stereo network is designed as an entity pair extractor. The entity pair extractor is used to calculate the probability of each element in the head-tail cube being a head-tail entity pair, and compare it with a predefined threshold to obtain a set of candidate head-tail entity pairs. S4. Based on the candidate head and tail entity pairs obtained by the entity pair extractor, the semantic vector representation of the entity pairs in all relationship matrices of the triple cube is obtained through a shared text encoder; and another new cubic network is designed as a triple extractor. The triple extractor is used to determine the relationship between each head and tail entity pair and obtain the final triple result.

2. According to the three-dimensional network-based relationship extraction algorithm of claim 1, it is characterized in that: The step S1 specifically includes: S11, take text C as input, C = [c1, c2, ..., c n ], using the context-aware pre-trained language model BERT to output a context-aware representation for each ci in the input text: H=PLM([ <cls>,c1,c2,...,c n , <sep> ])，< / sep> < / cls> in <cls>and <sep>It is a special marker in the BERT model. PLM represents the pre-trained language model. H = [h <CLS) ,h1,h2,...,h n ,h <sep>< / sep> ] is the context representation of text C, h i is the contextual representation of the i-th character in text C.< / sep> < / cls> 3. The relationship extraction algorithm based on a stereo network according to claim 1 is characterized in that: The step S2 specifically includes: S21. Design a cube As a head-tail cube, where n represents the length of the input text, m represents the number of all head entity and tail entity length combinations, and each different head-tail entity pair can be uniquely associated with a specific element in the cube, expressed as (h_s, t_s, [h_len, t_len]); h_s of the first dimension and t_s of the second dimension are the starting character positions of the head entity and the tail entity, respectively, and h_len and t_len of the third dimension represent the length of the head entity and the tail entity, respectively; S22. Design a cube As a triple cube, where n and l represent the length of the input text and the number of predefined relations, respectively, each different triple can be represented by one or more elements in the cube.

4. The relationship extraction algorithm based on stereo network according to claim 1 is characterized in that: The step S3 specifically includes: S31. Based on the head-tail cube, design a stereo network as an entity pair extractor; construct m matrices M∈R n×n , n represents the length of the input text, m represents the number of all head entity and tail entity length combinations, for the i-th matrix, each element Indicates the probability that the starting characters of the head entity and the tail entity are located at the jth and kth positions respectively, and the corresponding lengths of the head entity and the tail entity are h_len and t_len respectively; based on the context representation of the input text, It can be expressed as: Among them, W1, W2 and b1, b2 are trainable weight parameters. j , R k are rotation matrices, which are used to incorporate relative position information into h j and h k middle; S32. Define the loss function of the entity pair extractor, which is used to learn the parameters in the entity pair extractor: Where e represents a natural constant, Ω = {(j, k) | 1 ≤ j, k ≤ n} is the set of all possible head-tail entity pairs in the i-th matrix, Ω P Contains all true head-tail entity pairs; S33, comparing the probability of each element in the head-tail cube being a head-tail entity pair with a predefined threshold, and those exceeding the threshold are selected as candidate head-tail entity pairs; S34. For a set of head and tail entity pairs obtained (h_s, t_s, [h_len, t_len]), convert them into four-tuples with clearer entity boundary information, namely (h_s, h_e, t_s, t_e); h_s and h_e respectively represent the starting and ending character positions of the head entity, and t_s and t_e respectively represent the starting and ending character positions of the tail entity.

5. The three-dimensional network-based relationship extraction algorithm according to claim 1, characterized in that: The step S4 specifically includes: S41. For a set of entity pair tags (h_s, h_e, t_s, t_e) obtained by the entity pair extractor, obtain the semantic vector representation of the entity pair in all relationship matrices of the triple cube through the shared text encoder: Among them, M′ i，j The vector representing the i-th head entity tag and the j-th tail entity tag has the same training parameters as the entity pair extractor; S42. Based on the triple cube, design a three-dimensional network as a triple extractor and construct l matrices M′∈R n×n , n represents the length of the input text, l represents the number of predefined relations, M′ i，j Represents the vector of the i-th head entity tag and the j-th tail entity tag obtained by the head-tail cube, and M′ i，j Input into a multilayer perceptron to obtain the probability of entity pairs being associated with various relations: Among them, W r and b r are trainable weights, is the semantic vector representation of the candidate entity pair in all relation matrices of the triple cube, and σ represents the sigmoid function; S43. Objective optimization function of triple extractor and objective optimization function of entity extractor Same, used to learn the parameters in the triplet extractor; S44. Combine the head and tail entity pairs obtained by the entity pair extractor with the highest probability relationship type obtained by the corresponding relationship extractor to form a final triple.

Images