A relation extraction algorithm based on three-dimensional networks

Abstract

This invention discloses a relation extraction algorithm based on a stereo network. The invention includes the following steps: First, a pre-trained language model is used as a text encoder to encode the input text; second, a stereo network is designed as an entity pair extractor to calculate the probability of each element in the head-tail cube being a head-tail entity pair, and a set of candidate head-tail entity pairs is obtained by comparing it with a predefined threshold; next, each head-tail entity pair is labeled in all relation matrices of the triplet cube, and the context representation of each candidate head-tail entity pair is obtained based on a shared text encoder; finally, another novel stereo network is designed as a triplet extractor to determine the relationship between each head-tail entity pair, obtaining the final triplet result. This invention is applicable to relation extraction tasks, and by designing a novel stereo network framework that jointly extracts entities and relations, it significantly improves the accuracy of triplet extraction.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically to the field of text information extraction technology, and more specifically, to a relation extraction algorithm based on a three-dimensional network. Background Technology

[0002] As a crucial task in the field of natural language processing, relation extraction has become a key component of text understanding and information extraction. Its goal is to extract structured knowledge from unstructured text, thereby helping machines understand human natural language more accurately. However, with the continuous expansion of data scale and the increasing complexity of application scenarios, traditional relation extraction methods face performance bottlenecks and insufficient generalization ability. Therefore, relation extraction based on pre-trained language models has emerged, aiming to fully utilize the rich semantic information in large-scale corpora to improve the model's generalization ability and accuracy.

[0003] Existing methods for relation extraction tasks are mainly divided into two categories: pipeline-based extraction methods and joint extraction methods. Pipeline-based extraction methods use sequence labeling to identify all latent entities in a given text, and then use a classifier to determine the most suitable relation for each head-and-tail entity pair. However, since these two stages are trained independently, they cannot share information during triple extraction, resulting in limited model performance. More importantly, these methods are prone to error propagation because errors in early stages cannot be corrected in later stages. To address these issues, researchers have proposed joint extraction methods, aiming to extract entities and relations simultaneously in one stage. Joint extraction methods can be further divided into three categories: 1) multi-task learning, 2) task paradigm shifting (e.g., converting relation extraction tasks into multi-turn question answering or text generation tasks), and 3) table filling methods.

[0004] In recent years, table-filling methods have achieved state-of-the-art performance in union relation extraction tasks. While existing table-filling methods have achieved some success in union relation extraction, they still suffer from issues in efficiency and accuracy, particularly accuracy. The core idea of ​​current table-filling methods is to use a relation-based two-dimensional matrix to label head-and-tail entity pairs. These labeling schemes aim to use multiple elements in the two-dimensional matrix to label a head-and-tail entity pair. This method requires a large number of labels for head-and-tail entity pairs within the matrix, resulting in a surge in the number of label combinations, leading to increased decoding time and reduced efficiency. Simultaneously, this process generates many invalid label combinations, which undoubtedly affect accuracy. Furthermore, extracting nested entity pairs is also a significant challenge. When extracting nested entities, a single element in the two-dimensional matrix needs to correspond to multiple labels, making it difficult to accurately capture the semantic representation of the element during training, severely impacting the accuracy of the extraction process.

[0005] In conclusion, a new relation extraction algorithm needs to be designed to solve the above problems. Summary of the Invention

[0006] In view of this, the present invention provides a relation extraction algorithm based on a three-dimensional network. First, a novel cube labeling scheme is designed. Its most significant feature is that a single element is used to label a head-and-tail entity pair, ensuring high efficiency and effectiveness. Second, a relation extraction method based on a three-dimensional network is proposed, designing a novel three-dimensional network for joint entity and relation extraction, consisting of two distinct three-dimensional networks: one for identifying head-and-tail entity pairs, and the other for determining their relationships.

[0007] The technical approach of this invention is as follows: First, a pre-trained language model is used as a text encoder to encode the input text. Next, a stereo network is designed as an entity pair extractor to identify a set of candidate head-and-tail entity pairs. Then, each head-and-tail entity pair is labeled in all relation matrices of a predefined triplet cube. Finally, another new stereo network is designed as a triplet extractor to determine the relations between the head-and-tail entity pairs, obtaining the final triplet result.

[0008] The specific steps are as follows:

[0009] S1. Use a pre-trained language model as a text encoder to encode the input text;

[0010] S2. Design the head and tail cubes and the triplet cubes to mark the head and tail entity pairs and triplets, respectively.

[0011] S3. Design a 3D network as an entity pair extractor to calculate the probability of each element in the head and tail cube being a head and tail entity pair, and compare it with a threshold to identify a set of candidate head and tail entity pairs.

[0012] S4. Obtain the semantic vector representation of entity pairs in all relation matrices of the triple cube through a shared text encoder; and design another new 3D network as a triple extractor to determine the relationship between each head and tail entity pair and obtain the final triple result.

[0013] S11. Using text C as input, the context-aware pre-trained language model BERT is used to label each input and output a context-aware representation:

[0014] H = PLM([ <cls>,c1,c2,…,c n , <sep>]),

[0015] in <cls>and <sep>It is a special marker in the BERT model.

[0016] S21. Design a cube as the head and tail cube, and establish a unique association between each distinct head and tail entity pair and a specific element in the head and tail cube.

[0017] S22. Design a new cube as a triplet cube, where each distinct triplet can be represented by one or more elements in the cube.

[0018] S31. Based on the head-and-tail cube, design a 3D network as an entity pair extractor. Construct m matrices M∈R n ×n For the i-th matrix, each element This represents the probability that the starting character of the head entity and the tail entity are located at the j-th and k-th positions, respectively. It can be represented as:

[0019]

[0020] Where W1, W2 and b1, b2 are trainable weight parameters. R j R k It is a rotation matrix;

[0021] S32. Define the loss function for the entity pair extractor, used to learn the parameters in the entity pair extractor:

[0022]

[0023] Where Ω={(j,k)|1≤j,k≤n} is the set of all possible head and tail entity pairs in the i-th matrix. Includes all real head and tail entity pairs;

[0024] S33. Compare the probability of each element in the head and tail cube as a head and tail entity pair with a predefined threshold. Those exceeding the threshold are selected as candidate head and tail entity pairs.

[0025] S34. For the obtained set of head and tail entity pairs, convert them into quadruples with clearer entity boundary information. S41. For a set of entity pair tags (h_s, h_e, t_s, t_e) obtained by the entity pair extractor, use a shared text encoder to obtain the semantic vector representation of the entity pairs in all relation matrices of the triplet cube:

[0026]

[0027] Among them, M′ i,j The vector representing the i-th head entity label and the j-th tail entity label has the training parameters of another 3D network; S42. Based on the triplet cube, design a 3D network as a triplet extractor. Construct l matrices M′∈R n×n M′ i,j Let M′ represent the vector containing the i-th head entity tag and the j-th tail entity tag obtained through the head-tail cube. i,j The input is fed into a multilayer perceptron to obtain the probabilities of entity pairs being associated with various relations:

[0028]

[0029] Among them, W r and b r These are trainable weights, and σ represents the sigmoid function;

[0030] S43. Objective optimization function of triplet extractor and objective function of entity extractor The same applies to learning the parameters in the triplet extractor;

[0031] S44. During the reasoning phase, the head and tail entity pairs are combined with the top-1 relation types obtained by the corresponding relation extractor to form the final triples. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the overall framework of the relation extraction framework based on three-dimensional networks of the present invention;

[0033] Figure 2 This is a flowchart of the relationship extraction framework based on three-dimensional networks of the present invention; Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0035] Please see Figure 1 , Figure 1 This is a schematic diagram of the overall framework of the relation extraction framework based on a 3D network of the present invention. The method includes: First, encoding the input text using a pre-trained language model as a text encoder; second, designing a head-tail cube, which represents all head-tail entity pair combinations in the input text, with each distinct head-tail entity pair uniquely associated with a specific element in the head-tail cube. A triplet cube is also designed, representing all candidate triplets, with each distinct triplet represented by one or more elements in the triplet cube; then, a 3D network is designed as an entity pair extractor to calculate the probability of each element in the head-tail cube being a head-tail entity pair and compare it with a predefined threshold to obtain a set of candidate head-tail entity pairs; next, each head-tail entity pair is labeled in all relation matrices of the triplet cube, and the contextual representation of each head-tail entity pair is obtained based on a shared text encoder. Furthermore, another 3D network is designed as a triplet extractor to determine the relationships between head-tail entity pairs.

[0036] Please see Figure 2 , Figure 2 This is a flowchart of the relationship extraction framework based on three-dimensional networks of the present invention, and the specific steps include:

[0037] S1. Input text: Use a pre-trained language model as a text encoder to encode the input text and obtain the encoding of the input text.

[0038] S11. Given text C, C = [c1, c2, ..., c n The context-aware pre-trained language model BERT outputs a context-aware representation for each input tag ci:

[0039] H = PLM([ <cls>,c1,c2,…,c n , <sep>]),

[0040] in <cls>and <sep>This is a special notation in the BERT model; PLM stands for Pre-trained Language Model, where H = [h <cls>< / cls> ,h1,h2,…,h n ,h <sep>< / sep> ] is the context representation of text C, h i It is the context representation of the i-th character in text C.

[0041] S2. Design a head-and-tail cube, where each distinct head-and-tail entity pair is uniquely associated with a specific element in the head-and-tail cube; design a triplet cube, where each distinct triplet can be represented by one or more elements in the triplet cube.

[0042] S21. Design a cube. As a head-tail cube, where n represents the length of the input text and m represents the number of all head and tail entity length combinations, each distinct head-tail entity pair can be uniquely associated with a specific element in the cube, denoted as (h_s, t_s, [h_len, t_len]); the first dimension h_s and the second dimension t_s are the starting character positions of the head and tail entities, respectively; correspondingly, the third dimensions h_len and t_len represent the lengths of the head and tail entities, respectively.

[0043] S22. Design a cube. As a triple cube, where n and l represent the length of the input text and the number of predefined relations, respectively, each distinct triple can be represented by one or more elements in the cube. (See attached...) Figure 2 As shown, the triple "(Melanie joyceon,r1,dallas)" can be composed of the elements "(Melanie,r1,dallas)" and "(joyceon,r1,dallas)", where r1 is the relationship between "Melanie joyceon" and "dallas".

[0044] S3. Design a 3D network as an entity pair extractor to calculate the probability of each element in the head and tail cubes being a head and tail entity pair, and compare it with a predefined threshold to obtain a set of candidate head and tail entity pairs.

[0045] S31. Based on the head-and-tail cube, design a 3D network as an entity pair extractor. Construct m matrices M∈R n ×n n represents the length of the input text, and m represents the number of all combinations of head and tail entity lengths. For the i-th matrix, each element... This represents the probability that the starting character of the head entity and the tail entity are located at the j-th and k-th positions, respectively (the lengths of the head entity and the tail entity are represented by h_len and t_len, respectively). Based on the context representation of the input text, It can be represented as:

[0046]

[0047] Where W1, W2 and b1, b2 are trainable weight parameters. R j R k These are rotation matrices, used to incorporate relative position information into h. j and h k middle;

[0048] S32. Define the loss function for the entity pair extractor, used to learn the parameters in the entity pair extractor:

[0049]

[0050] Where e represents the natural constant, and Ω = {(j,k)|1≤j,k≤n} is the set of all possible head and tail entity pairs in the i-th matrix. Ω P Includes all real head and tail entity pairs;

[0051] S33. Use an entity pair extractor to calculate the probability of each element in the head and tail cube being a head and tail entity pair, and compare it with a predefined threshold. Those exceeding the threshold are selected as candidate head and tail entity pairs.

[0052] S34. For the obtained set of head and tail entity pairs (g_s, t_s, [h_len, t_len]), convert them into quadruplets with clearer entity boundary information, i.e., (h_s, h_e, t_s, t_e). h_s and h_e represent the start and end character positions of the head entity, respectively, and t_s and t_e represent the start and end character positions of the tail entity, respectively. This conversion can be achieved by the formula e = s + len - 1, where s, e, and len are the start character position, end character position, and length of the entity, respectively.

[0053] S4. Based on the candidate head and tail entity pairs obtained by the entity pair extractor, the semantic vector representation of the entity pairs in all relation matrices of the triple cube is obtained through a shared text encoder; and another new 3D network is designed as a triple extractor to determine the relationship between each head and tail entity pair and obtain the final triple result.

[0054] S41. For a set of entity pair tags (h_s, h_e, t_s, t_e) obtained by the entity pair extractor, the semantic vector representation of the entity pair in all relation matrices of the triplet cube is obtained through a shared text encoder:

[0055]

[0056] Among them, M′ i,j The vector representing the i-th head entity tag and the j-th tail entity tag has the same training parameters as the entity pair extractor.

[0057] S42. Based on triplet cubes, design a 3D network as a triplet extractor. Construct l matrices M′∈R n×n (n represents the length of the input text, l represents the number of predefined relations), M′ i,j Let M′ represent the vector containing the i-th head entity tag and the j-th tail entity tag obtained through the head-tail cube. i,j The input is fed into a multilayer perceptron to obtain the probabilities of entity pairs being associated with various relations:

[0058]

[0059] Among them, W r and b r These are trainable weights. It is the semantic vector representation of candidate entity pairs in all relation matrices of the triple cube, where σ represents the sigmoid function;

[0060] S43. Objective optimization functions of triplet extractor and entity extractor The same applies to learning the parameters in the triplet extractor;

[0061] S44. During the reasoning phase, the head and tail entity pairs obtained by the entity pair extractor are combined with the highest probability relation type obtained by the corresponding relation extractor to form the final triple.

[0062] This invention underwent extensive experiments on five publicly available datasets. The results on the Chinese relation extraction dataset CMeIE-V2 are shown in Table 1. From the table, we conclude that: 1) The proposed method outperforms all baselines in F1 score, demonstrating its effectiveness. In particular, the proposed method surpasses the current state-of-the-art GPLinker method by 1.36% in F1 score. 2) While GPLinker slightly surpasses the proposed method in accuracy, it significantly lags behind in recall. This difference is primarily due to the typically long entity lengths in CMeIE-V2, resulting in an unusually high number of dimensions on the depth axis for our head and tail cubes, leading to exceptionally sparse cubes. Consequently, each layer corresponding to the depth axis of the cube has relatively little data, posing a challenge to training an efficient model.

[0063] Table 1. Model comparisons on the CMeIE-V2 dataset.

[0064]

[0065] < / sep> < / cls> < / sep> < / cls> < / sep> < / cls> < / sep> < / cls>

Claims

1. A relation extraction algorithm based on a three-dimensional network, characterized in that... Includes the following steps: S1. Input text, use a pre-trained language model as a text encoder to encode the input text, and obtain the text encoding; S2. Design a head-tail cube to label all head-tail entity pair combinations in the input text. Each distinct head-tail entity pair is uniquely associated with a specific element in the head-tail cube. Design a triple cube to label all triples. Each distinct triple is represented by one or more elements in the triple cube. S3. Based on the context representation obtained by the text encoder, design a 3D network as an entity pair extractor. The entity pair extractor is used to calculate the probability of each element in the head and tail cube as a head and tail entity pair and compare it with a predefined threshold to obtain a set of candidate head and tail entity pairs. S4. Based on the candidate head and tail entity pairs obtained by the entity pair extractor, the semantic vector representation of the entity pairs in all relation matrices of the triple cube is obtained through a shared text encoder. A novel 3D network is designed as a triplet extractor. This triplet extractor is used to determine the relationship between each head and tail entity pair and obtain the final triplet result, specifically including: S41. For a set of entity pair tags (h_s, h_e, t_s, t_e) obtained by the entity pair extractor, the semantic vector representation of the entity pair in all relation matrices of the triplet cube is obtained through a shared text encoder: Among them, M ′ i,j The vector represents the i-th head entity label and the j-th tail entity label, and has the same training parameters as the entity pair extractor. h_len and t_len represent the lengths of the head entity and the tail entity, respectively. S42. Based on triplet cubes, design a 3D network as a triplet extractor to construct l matrices M′∈R. n×n n represents the length of the input text, l represents the number of predefined relations, and M represents the length of the input text. ′ i,j This represents the vector containing the i-th head entity tag and the j-th tail entity tag obtained through the head-tail cube, where M... ′ i,j The input is fed into a multilayer perceptron to obtain the probability that entity pairs are associated with various relationships: Among them, W r and b r These are the training weights. It is the semantic vector representation of candidate entity pairs in all relation matrices of the triple cube, where σ represents the sigmoid function; S43. Objective optimization functions of triplet extractor and entity extractor The same applies to learning the parameters in the triplet extractor; S44. Combine the head and tail entity pairs obtained by the entity pair extractor with the highest probability relation type obtained by the corresponding relation extractor to form the final triple.

2. The relation extraction algorithm based on a three-dimensional network according to claim 1, characterized in that, Step S1 specifically includes: S11. Input text C, where C = [c1, c2, ..., c n The context-aware pre-trained language model BERT outputs a context-aware representation for each ci in the input text. H=PLM([ <cls>,c1,c2,…,c n , <sep> ]),< / sep> < / cls> in <cls>and <sep>This is a special notation in the BERT model; PLM stands for Pre-trained Language Model, where H = [h <cls>< / cls> ,h1,h2,…,h n ,h <sep>< / sep> ] is the context representation of text C, h i It is the context representation of the i-th character in text C.< / sep> < / cls> 3. The relation extraction algorithm based on a three-dimensional network according to claim 1, characterized in that, Step S2 specifically includes: S21. Design a cube. As a head-tail cube, where n represents the length of the input text and m represents the number of all head and tail entity length combinations, each distinct head-tail entity pair is uniquely associated with a specific element in the cube, denoted as (h_s, t_s, [h_len, t_len]); the first dimension h_s and the second dimension t_s are the starting character positions of the head and tail entities, respectively, and the third dimension h_len and t_len represent the lengths of the head and tail entities, respectively. S22. Design a cube. As a triple cube, n and l represent the length of the input text and the number of predefined relations, respectively, and each distinct triple is represented by one or more elements in the cube.

4. The relation extraction algorithm based on a three-dimensional network according to claim 1, characterized in that, Step S3 specifically includes: S31. Based on the head-and-tail cube, design a 3D network as an entity pair extractor; construct m matrices M∈R n×n n represents the length of the input text, m represents the number of all combinations of head and tail entity lengths, and for the i-th matrix, each element... This represents the probability that the starting character of the head entity and the tail entity are located at the j-th and k-th positions, respectively, and the lengths of the head entity and the tail entity are h_len and t_len, respectively; based on the context representation of the input text, Represented as: Where W1, W2 and b1, b2 are the training weight parameters, R j R k These are rotation matrices, used to incorporate relative position information into h. j and h k middle; S32. Define the loss function for the entity pair extractor, used to learn the parameters in the entity pair extractor: Where e represents the natural constant, and Ω = {(j,k)|1≤j,k≤n} is the set of all possible head and tail entity pairs in the i-th matrix. Ω P Includes all real head and tail entity pairs; S33. Compare the probability of each element in the head and tail cube as a head and tail entity pair with a predefined threshold. Those exceeding the threshold are selected as candidate head and tail entity pairs. S34. For the obtained set of head and tail entity pairs (h_s, t_s, [h_len, t_len]), convert them into quadruplets with clearer entity boundary information, i.e. (h_s, h_e, t_s, t_e); h_s and h_e represent the start and end character positions of the head entity, respectively, and t_s and t_e represent the start and end character positions of the tail entity, respectively.

Images