A Joint Entity Relation Extraction Method Based on Dual-Query Network

Abstract

This invention discloses a joint entity relation extraction method based on a dual-query network. Unlike previous works that treat entity relation extraction as a multi-turn question-answering task, this invention uses two sets of queries for extraction, where each entity query extracts one entity and each relation query retrieves one relation. Since these learnable queries are type-independent, this invention avoids manually constructing natural queries for each relation category, which is inefficient and labor-intensive. To obtain the final relation triples, this invention designs relation pointers to associate the results of the two sets of queries. Furthermore, through an attention mechanism, this invention can naturally model the interdependencies between the two tasks, which is difficult to achieve in span-based classification methods.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of entity relation extraction, and particularly relates to a joint entity relation extraction method based on a double query network. BACKGROUND

[0002] Joint entity relation extraction has always been a core task in the field of information extraction. The purpose of the joint entity relation extraction task is to mine high-quality relation entity triples from unstructured text.

[0003] Existing entity recognition and relation extraction methods are divided into two categories: pipeline methods and joint methods. The pipeline method marks the entities in the sentence through an entity recognition model, and then predicts the relationship between them through a relation extraction model. Although it is flexible to build a pipeline method, these methods are more prone to error propagation, in which prediction errors from entity recognition will affect relation extraction. In addition, there is a lack of effective interaction between entity recognition and relation extraction, ignoring the internal relationship and dependence between the two. Therefore, many methods of connecting entity and relation extraction have been proposed to unify the entity recognition model and the relation extraction model through different strategies.

[0004] Recently, more and more researches focus on how to better model the interaction between entity recognition NER and relation extraction RE. Wang et al. use two different encoders for the two tasks to interact with each other. Shen et al. propose a memory flow attention mechanism to enhance the interaction between the two tasks. Yan et al. design a partition filtering network to model the interaction between tasks appropriately while encoding task-specific features. However, these methods only consider the interaction between tasks, ignoring the internal dependence between entities and relations. In addition, although the search space can be reduced through length limitation or filtering strategy, these methods still need to classify all candidate entity pairs, which is low in efficiency. SUMMARY

[0005] The present application aims at the deficiencies of the prior art, and provides a joint entity relation extraction method based on a double query network. The present application proposes a novel network to predict entities and relations in text. Two groups of queries are proposed to obtain entity semantics and relation semantics from context. Each entity query is used to predict an entity, and each relation query is used to predict a relation triple. Considering that each query corresponds to an entity or a relation, the present application can naturally capture the dependence between entities and relations through the interaction between queries. After obtaining the entity and relation representations, the present application designs entity pointers to connect the relation query to its corresponding head and tail entity queries. In addition, since the number of these learnable queries is limited, the present application does not need to classify all possible entity pairs.

[0006] The technical scheme adopted by the present application to solve its technical problems is as follows: a joint entity relation extraction method based on a double query network, the method comprising the following steps:

[0007] 1) text representation: using a pre-trained language model and a bidirectional LSTM to obtain a context vector representation;

[0008] 2) joint decoding of entity relations, comprising the following three parts:

[0009] 2.1) entity prediction: using a set of globally learnable query vectors to represent entity queries, each entity query is responsible for extracting an entity; first, use the entity decoder to convert the entity query to obtain the entity embedding representation; then use the fully connected layer to construct the entity classifier and the entity locator to predict the entity type and the entity boundary respectively;

[0010] 2.2) relationship prediction: using a set of globally learnable query vectors to represent relationship queries, each relationship query is responsible for extracting a relationship; first, use the relationship decoder to convert the relationship query to obtain the relationship embedding representation; then use the fully connected layer to construct the relationship classifier and the relationship pointer to predict the relationship type and the relationship object respectively;

[0011] 2.3) interaction between entity relation tasks: use the interaction between entity queries / relationship queries to capture the dependency between entity extraction / relationship extraction; first, use the fully connected layer to linearly project the entity / relationship query converted by the decoder; then use the multi-head attention mechanism layer to obtain the entity / relationship embedding representation;

[0012] 3) training of entity relation extraction model: according to the dichotomy matching algorithm, the optimal labels are assigned to entities and relationships respectively, and then the entity relation extraction model training is completed through the negative logarithmic loss based on cross entropy;

[0013] 4) entity relation extraction: after fine-tuning the entity relation extraction model parameters through the training data set, input the test text to be extracted into the entity relation extraction model; the entity classifier and the entity locator calculate the predicted entity probability, and the relationship classifier and the relationship pointer calculate the predicted relationship probability; extract entities and relationships by taking the maximum probability operation.

[0014] Further, the step 1) is specifically:

[0015] Given an input sentence s, first use the pre-trained BERT model to obtain the context vector representation of each word; in order to capture the long-range dependency, the sentence is expanded to a fixed window size; then, in order to better consider the sequential information of the sentence, use Bi-LSTM to obtain the final sentence vector representation;

[0016] Assuming that the sentence s is composed of n words, define the vector representation of the i-th word of s as where is the hidden dimension of the pre-trained BERT model; in Bi-LSTM, the sentence is encoded into hidden states (h1, h2,..., hn) n , n is the number of hidden states of Bi-LSTM, which is the same as the number of words, and the encoding formula of the forward and reverse is as follows:

[0017]

[0018]

[0019] where and represent the hidden states of the forward and backward LSTM at the t-th time step, respectively, d c is the hidden dimension of Bi-LSTM, and represent the forward LSTM and the backward LSTM, respectively, v t is the vector representation of the word w t at the t-th time step; let as the output result of the word, where represents the concatenation of vectors; use h1, h2,..., hn n as the sentence vector representation H.

[0020] Further, the entity prediction in step 2.1) is specifically:

[0021] A set of globally learned query vectors is used, denoted as entity queries Each query is represented as a vector of size d, which is used to extract an entity; these queries are randomly initialized, and the number of queries M e is set by experiment;

[0022] In entity prediction, first, the entity decoder is used to convert the entity query to obtain the entity embedding representation; then, the entity classifier and the entity locator are designed to obtain the predicted entity;

[0023] In the entity decoder, L Transformer layers are stacked for decoding, and the multi-head self-attention mechanism between the entity queries Q e is used to model the dependency between entities, and the multi-head cross-attention mechanism is used to obtain the context information perceived by the entity from the sentence vector representation H; through the L decoding layers, the entity query Q e is decoded into an intermediate vector E', and the above process is represented as:

[0024] E' = Entity-Decoder (Q e ; H)

[0025] wherein

[0026] Then pass through the task interaction module, which is detailed in step 2.3), obtain M e entity embedding representations E according to E'

[0027] The implementation of the entity classifier is as follows: for M e entity embedding representations E, input them into a fully connected layer FFN with the same dimension to predict the class of the corresponding entity; since M e is greater than the number of gold entities in the sentence s, an additional empty label is added to represent no query entity; then the probability that the i-th entity belongs to class c is calculated The calculation formula is as follows:

[0028]

[0029] wherein and represent trainable parameters, E i represents the i-th entity embedding representation, represents the set of possible entity classes;

[0030] The implementation of the entity locator is as follows: in order to predict the left and right boundaries of each entity, first concatenate the entity embedding representation E with the sentence vector representation H to obtain the concatenated vector Then use a fully connected layer FFN with the same dimension to obtain the probability that the k-th word in the sentence is the left boundary of entity i The calculation formula is as follows:

[0031]

[0032] wherein n is the number of words in the sentence, and represent trainable parameters, and Sik represents the concatenated vector of the i-th entity embedding representation and the vector representation of the k-th word; the probability of the right boundary The calculation is the same as above.

[0033] Further, the relationship prediction in step 2.2) is specifically:

[0034] A set of globally learnable query vectors are used, denoted as relationship queries Each query is represented as a vector of size d, used to extract a relationship; these queries are randomly initialized, and the number of queries M r is set by experiment;

[0035] In relation prediction, first, the relation decoder is used to convert the relation query to obtain the relation embedding representation; then, the relation classifier and the relation pointer are designed to obtain the relation triplets;

[0036] In the relation decoder, L stacked Transformer layers are used for decoding, the dependency between relations is modeled by using the multi-head self-attention mechanism between the relation queries Qr, and the context information perceived by the relation is obtained from the sentence vector representation H by using the multi-head cross-attention mechanism; through the L decoding layers, the relation query Q r is decoded into the intermediate vector R', and the above process is represented as:

[0037] R' = Relation-Decoder (Q r ; H)

[0038] wherein

[0039] Then, M r relation embedding representations R are obtained from R' by using the task interaction module, which is described in detail in step 2.3), wherein

[0040] The implementation of the relation classifier is as follows: for M r relation embedding representations R, they are input into a one-dimensional fully connected layer FFN to predict the category of the corresponding relation; since M r is greater than the number of golden relations in the sentence s, an additional empty label is added to represent that no relation is queried; then, the probability that the i-th relation belongs to the category c is calculated The calculation formula is as follows:

[0041]

[0042] wherein and represent trainable parameters, R i represents the i-th relation embedding representation, represents the set of possible relation categories;

[0043] The implementation of the relation pointer is as follows: in order to predict the relation triplets, the relation pointer is used to connect the relation embedding representation with the corresponding subject-object entity embedding representation; first, the relation embedding representation R is spliced with the entity embedding representation E to obtain the spliced vector Then, a one-dimensional fully connected layer FFN is used to obtain the probability that the k-th entity belongs to the subject object of the i-th relation The calculation formula is as follows:

[0044]

[0045] where M e is the number of entity embedding representations, and denote trainable parameters, U ik denotes the concatenation vector of the ith relation embedding representation and the kth entity embedding representation; the probability of the object is calculated as above.

[0046] Further, in the step 2.3), the entity relation task interaction is realized through a task interaction module, specifically:

[0047] The interaction between the entity query and the relation query is used to capture the dependency between the entity extraction NER and the relation extraction RE; first, the intermediate vector E' obtained by decoding the entity query and the intermediate vector R' obtained by decoding the relation query are linearly projected by using a full connection layer FFN; then, the projection results thereof are spliced, and an entity embedding representation and a relation embedding representation are obtained through a multi-head attention mechanism layer Interaction. The above process is represented as:

[0048]

[0049] where W' e , W' r denote trainable parameters.

[0050] Further, in the step 3), the optimal labels for entities and relations are assigned according to a bipartite matching algorithm, and the model training is completed based on a cross-entropy negative logarithm loss, specifically:

[0051] After the predicted entity probability and the predicted relation probability are calculated, in order to perform end-to-end training, a label matcher is designed to assign the predicted to the gold label y, and the labels are filled to the same number as the prediction of non-entity or non-relation; taking the matching entity as an example, the predicted entity is defined as where is the entity category probability, is the entity boundary probability; the gold entity y i is defined as y i = (t i , l i , r i ), where t i is the entity category, l i , r i is the entity boundary; in order to find the optimal match, the minimum cost permutation σ ∈ Θ between M e entity queries and the gold entity is searched:

[0052]

[0053] where Θ is all cost arrangements, defined as

[0054] The optimal matching is obtained by using the bipartite matching algorithm

[0055] Then, the model training loss is defined according to the optimal matching and entity prediction probability

[0056]

[0057] where M is the number of entity queries, e is the category probability of the i th entity query matching entity, is the boundary probability of the i th entity query matching entity; the matching relationship is the same as above. Further, in the step 4), after the entity relationship extraction parameters are fine-tuned through the training data set, the test text to be extracted is input into the entity relationship extraction model, the entity classifier and the entity locator calculate the predicted entity probability, the relationship classifier and the relationship pointer calculate the predicted relationship probability, and then the entity and relationship are extracted by taking the maximum probability operation, and specifically:

[0058] For entity extraction, the predicted entity of the i th entity query is

[0059] and is the boundary of the entity, is the category of the entity; the entity with a predicted type of null will be filtered out; For relationship extraction, the predicted relationship of the i th relationship query is

[0060] and is the object of the relationship, is the category of the relationship; the relationship with a predicted type of null will be filtered out. The method of the present application has the following beneficial effects compared with the prior art:

[0061] 1. The present method uses a learnable query to predict all entities and relationships at one time. At the same time, the self-attention mechanism is used to naturally capture the dependency relationship between different types of entities and different types of relationships.

[0062]

[0063] ​​​2. The number of queries that can be learned in the method is limited, and there is no need to classify all possible spans, improving the prediction efficiency.

[0064] 3. The method provides a set of learnable queries to replace the manually constructed natural language queries in the multi-round query model. Since these queries are type-independent, the model in the invention can predict all types of relationship triples at once. This way is more efficient in inference.

[0065] 4. The method has high extraction accuracy and can accurately extract entities and their relationships in sentences. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is the overall flowchart of the method of the invention;

[0067] Figure 2 is the neural network model structure diagram in the specific embodiment of the invention. DETAILED DESCRIPTION

[0068] The invention will be further described in detail below in conjunction with the drawings and specific embodiments.

[0069] As shown in Figure 1 , the invention provides a joint entity relationship extraction method based on a double query network, including the following steps:

[0070] 1) Text representation: use a pre-trained language model and a bidirectional LSTM to obtain context vector representation;

[0071] 2) Entity relationship joint decoding, including the following three parts:

[0072] 2.1) Entity prediction: use a set of globally learnable query vectors to represent entity queries, each entity query is responsible for extracting an entity; first use the entity decoder to convert the entity query to obtain the entity embedding representation; then use the fully connected layer to construct the entity classifier and the entity locator, respectively, to predict the entity type and the entity boundary;

[0073] 2.2) Relationship prediction: use a set of globally learnable query vectors to represent relationship queries, each relationship query is responsible for extracting a relationship; first use the relationship decoder to convert the relationship query to obtain the relationship embedding representation; then use the fully connected layer to construct the relationship classifier and the relationship pointer, respectively, to predict the relationship type and the relationship object;

[0074] 2.3) Entity relationship task interaction: use the interaction between entity queries / relationship queries to capture the dependency between entity extraction / relationship extraction; first use the fully connected layer to linearly project the entity / relationship queries converted by the decoder; then use the multi-head attention mechanism layer to obtain the entity / relationship embedding representation;

[0075] 3) Training entity relation extraction model: according to the dichotomy matching algorithm, the optimal label is assigned to the entity and the relation respectively, and then the entity relation extraction model training is completed through the negative logarithm loss based on cross entropy;

[0076] 4) Entity relation extraction: after the entity relation extraction model parameters are fine-tuned through the training data set, the test text to be extracted is input into the entity relation extraction model; the entity classifier and the entity locator calculate the predicted entity probability, the relation classifier and the relation pointer calculate the predicted relation probability; the entity and the relation are extracted by taking the maximum probability operation.

[0077] Specifically, the step 1) is specifically:

[0078] Given the input sentence s, first use the pre-trained BERT model to obtain the context vector representation of each word; in order to capture the long-range dependency, the sentence is expanded to a fixed window size; then, in order to better consider the sequential information of the sentence, the Bi-LSTM is used to obtain the final sentence vector representation;

[0079] Suppose the sentence s is composed of n words, and the vector representation of the i-th word in s is defined as where d m is the hidden dimension of the pre-trained BERT model; in Bi-LSTM, the sentence is encoded into hidden state (h1, h2,..., hn), n is the number of hidden states of Bi-LSTM, which is the same as the number of words, and the encoding formula of forward and reverse is as follows: n

[0080]

[0081]

[0082] where, and represent the hidden states of the forward and backward LSTM at the t-th time step, d c is the hidden dimension of Bi-LSTM, and represent the forward LSTM and the backward LSTM, respectively, v t is the vector representation of the word w t at the t-th time step; let as the output result of the word, where represents the concatenation of vectors; use h1, h2,..., hn n as the sentence vector representation H.

[0083] Specifically, the entity prediction in step 2.1) is specifically: ​

[0084] A set of global learnable query vectors, denoted as entity queries Q Each query is represented as a vector of size d, used to extract one entity; these queries are randomly initialized, the number of queries M e is set by experiment;

[0085] In entity prediction, first, the entity decoder is used to convert the entity query to obtain the entity embedding representation; then, the entity classifier and the entity locator are designed to obtain the predicted entity;

[0086] In the entity decoder, L Transformer layers are stacked for decoding, using the multi-head self-attention mechanism between the entity queries Q e to model the dependency between entities, and using the multi-head cross-attention mechanism to obtain the entity-aware context information from the sentence vector representation H; through the L decoding layers, the entity query Q e is decoded into the intermediate vector E', and the above process is represented as:

[0087] E' = Entity-Decoder(Q e ; H)

[0088] Wherein

[0089] Then through the task interaction module, which is described in detail in step 2.3), M e entity embedding representations E are obtained from E', wherein

[0090] The implementation of the entity classifier is as follows: for M e entity embedding representations E, input them into a one-dimensional fully connected layer FFN to predict the class of the corresponding entity; since M e is greater than the number of gold entities in the sentence s, an additional empty label is added to represent no query entity; then the probability that the i-th entity belongs to class c is calculated The calculation formula is as follows:

[0091]

[0092] Wherein and represent trainable parameters, E i represents the i-th entity embedding representation, represents the set of possible entity classes;

[0093] The implementation of the entity locator is as follows: in order to predict the left and right boundaries of each entity, first, the entity embedding representation E is concatenated with the sentence vector representation H to obtain the concatenated vector Then use a one-dimensional full connection layer FFN to get the probability that the kth word in the sentence is the left boundary of entity i The calculation formula is as follows:

[0094]

[0095] Where n is the number of words in the sentence, And Indicates a trainable parameter, S ik Indicates the splicing vector of the i-th entity embedding representation and the k-th word vector representation; the probability of the right boundary The calculation is the same as above.

[0096] Specifically, the relationship prediction in step 2.2) is specifically:

[0097] A set of globally learnable query vectors are used, denoted as relationship queries Each query is represented as a vector of size d, used to extract a relationship; these queries are randomly initialized, and the number of queries M r is set by experiment;

[0098] In relationship prediction, first use the relationship decoder to convert the relationship query to obtain the relationship embedding representation; then design the relationship classifier and the relationship pointer to obtain the relationship triplets;

[0099] In the relationship decoder, L Transformer layers are stacked for decoding, the multi-head self-attention mechanism between relationship queries Qr is used to model the dependency relationship between relationships, and the multi-head cross-attention mechanism is used to obtain the context information from the sentence vector representation H; through L decoding layers, the relationship query Q r is decoded into an intermediate vector R', and the above process is represented as:

[0100] R' = Relation-Decoder(Q r ; H)

[0101] Where,

[0102] Then through the task interaction module, which is detailed in step 2.3), obtain M r relationship embedding representations R from R', where

[0103] The implementation of the relationship classifier is as follows: for M r embedding representations R, input them into a one-dimensional full connection layer FFN to predict the class of the corresponding relationship; since M rThe number of golden relations in the sentence s is greater than the number of golden relations in the sentence s, so an additional empty label is added to represent that no relation is found; then the probability that the ith relation belongs to the category c is calculated The calculation formula is as follows:

[0104]

[0105] Wherein And Denote trainable parameters, R i Denote the ith relation embedding representation, Denote the set of possible relation categories;

[0106] The implementation of the relation pointer is as follows: in order to predict the relation triple, the relation pointer is used to connect the relation embedding representation with the corresponding subject-object entity embedding representation; first, the relation embedding representation R is spliced with the entity embedding representation E to obtain a spliced vector Then, a one-dimensional full connection layer FFN is used to obtain the probability that the kth entity belongs to the subject object of the ith relation The calculation formula is as follows:

[0107]

[0108] Wherein M e is the number of entity embedding representations, And Denote trainable parameters, U ik Denote the spliced vector of the ith relation embedding representation and the kth entity embedding representation; the probability of the object object The calculation is the same as above.

[0109] Specifically, in the step 2.3), the entity relation task interaction is realized through the task interaction module, specifically:

[0110] The interaction between entity query / relation query is used to capture the dependency between entity extraction NER / relationship extraction RE. First, the intermediate vectors E' and R' are linearly projected by the full connection layer FFN; then, the projection results thereof are spliced, and the entity embedding representation And the relation embedding representation The above process is represented as:

[0111]

[0112] Wherein W' e , W' r Denote trainable parameters.

[0113] Specifically, in step 3), the optimal label is assigned to the entity and the relationship according to the bipartite matching algorithm, and the model training is completed based on the negative logarithmic loss of cross entropy, specifically:

[0114] After the predicted entity probability and the predicted relationship probability are calculated, in order to perform end-to-end training, the present application designs a label matcher to match the predicted Gold label y, the label is filled in the same number as the prediction of non-entity or non-relation. Taking the matched entity as an example, the predicted entity is defined as Wherein is the entity category probability, is the entity boundary probability; the gold entity y i is defined as y i =(t i , l i , r i ), wherein t i is the entity category, l i , r i is the entity boundary; in order to find the optimal matching, search for the minimum cost permutation σ∈Θ between M e entity queries and gold entities:

[0115]

[0116] Wherein, Θ is all cost permutations, defined as:

[0117]

[0118] The bipartite matching algorithm is used to obtain the optimal matching After that, the model training loss is defined according to the optimal matching and the entity prediction probability As:

[0119]

[0120] Wherein, M e is the number of entity queries, is the category probability of the matched entity of the i-th entity query, is the boundary probability of the matched entity of the i-th entity query; the method of matching relationship is the same as above.

[0121] Specifically, in step 4), after the entity relationship extraction model parameters are fine-tuned through the training data set, the test text to be extracted is input into the model, the entity classifier and the entity locator calculate the predicted entity probability, the relationship classifier and the relationship pointer calculate the predicted relationship probability, and then the entity and the relationship are extracted by taking the maximum probability operation, specifically:

[0122] For entity extraction, the i-th entity query predicted entity is and is the boundary of the entity, is the category of the entity; the entity with the predicted type of empty will be filtered out.

[0123] For relation extraction, the i-th relation query predicted relation is and is the object of the relation, is the category of the relation; the relation with the predicted type of empty will be filtered out.

[0124] Embodiment

[0125] The specific steps of the embodiment are described in detail as follows in combination with the method of the application:

[0126] In this embodiment, the method of the application is applied to a commonly used data set SciERC for joint extraction of entities and relations, which includes annotations of scientific entities and relations therebetween, and in addition, 500 scientific abstracts of coreference relations. This data set annotates 6 entity types and 7 relation types.

[0127] 1) Divide the data set. We randomly extract 15% of the training data set as a development set and a test set. In addition, each undirected relation in the training is regarded as two directed relations.

[0128] 2) Text representation. We use scibert as a pre-trained language model, and use a bidirectional LSTM with a hidden layer vector dimension of 384.

[0129] 3) Joint decoding of entities and relations. The number of decoder layers L is set to 5, and the optimizer is AdamW with a linear warm-up decay learning rate plan. The peak learning rate of the pre-trained model is set to 1e-5, and other parameters are set to 4e-5. The model is trained for 100 epochs, and the dropout is set to 0.1.

[0130] 4) Train the entity and relation extraction model. In this method, the network model is built using the PyTorch framework, and the Hungarian algorithm is used to realize bipartite matching.

[0131] 5) Entity relation extraction. After the training of the neural network is completed, we can input the test set data into the neural network, which is not in the previous training process. After fine-tuning the model parameters, the confidence threshold is set to 0.4 to make predictions. We use F1 value as the evaluation index of entity extraction NER and relation extraction RE. For NER, only when the type and boundary of the predicted entity match the type and boundary of the gold entity, the predicted entity is correct. For RE, we use a strict evaluation standard, that is, the type of the relationship is correct, and the type and boundary of the corresponding subject and object entities are correct. The test results are shown in Table 1, where SPE, PURE, UniRE are all joint entity relation extraction methods proposed, and the F1 performance index of the method is consistently better than these methods in two tasks.

[0132] Table 1 F1 evaluation of prediction results

[0133] NER RE The method 0.698 0.389 SPE 0.680 0.346 PURE 0.689 0.368 UniRE 0.684 0.369

[0134] The above only describes the preferred embodiments of one or more embodiments of the present specification, and does not limit one or more embodiments of the present specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present specification shall be included in the protection scope of one or more embodiments of the present specification.

Claims

1. A method for joint entity relation extraction based on a double query network, characterized in that, The method comprises the following steps: 1) text representation: using a pre-trained language model and a bidirectional LSTM to obtain a context vector representation; 2) entity and relation joint decoding, comprising the following three parts: 2.1) entity prediction: using a set of globally learnable query vectors as entity queries, each of which is responsible for extracting an entity; first, use the entity decoder to convert the entity query to obtain an entity embedding representation; then use a fully connected layer to construct an entity classifier and an entity locator to predict the entity type and entity boundary, respectively; 2.2) relation prediction: using a set of globally learnable query vectors as relation queries, each of which is responsible for extracting a relation; first, use the relation decoder to convert the relation query to obtain a relation embedding representation; then use a fully connected layer to construct a relation classifier and a relation pointer to predict the relation type and relation object, respectively; 2.3) entity and relation task interaction: using the interaction between entity queries / relation queries to capture the dependency between entity extraction / relation extraction; first, use a fully connected layer to linearly project the decoder-converted entity / relation query; then use a multi-head attention mechanism layer to obtain the entity / relation embedding representation; 3) training the entity and relation extraction model: according to the dichotomy matching algorithm, the optimal labels for entities and relations are assigned, and then the entity and relation extraction model training is completed based on the cross-entropy-based negative logarithm loss; 4) entity and relation extraction: after fine-tuning the entity and relation extraction model parameters through the training data set, input the test text to be extracted into the entity and relation extraction model; the entity classifier and entity locator calculate the predicted entity probability, and the relation classifier and relation pointer calculate the predicted relation probability; extract the entity and relation by taking the maximum probability operation.

2. The joint entity relation extraction method based on the double query network according to claim 1, characterized in that, The step 1) is specifically: Given an input sentence s, use the pre-trained BERT model to obtain the context vector representation of each word; expand the sentence to a fixed window size; Use Bi-LSTM to obtain the final sentence vector representation; Suppose a sentence s consists of n characters, and the vector representation of the i-th character in s is defined as follows: Where d m This refers to the hidden dimension of the pre-trained BERT model; in Bi-LSTM, sentences are encoded into hidden states (h1, h2, ..., h...). n ), where n is the number of hidden states in the Bi-LSTM, which is the same as the number of words. The encoding formulas for the forward and reverse directions are as follows: where and denote the hidden states of the forward and backward LSTM at the t-th time step, respectively, d c is the hidden dimension of the Bi-LSTM, and denote the forward and backward LSTM, respectively, v t is the word w t the vector representation at the t-th time step; let as the output result of the word, where denotes the concatenation of vectors; use h1, h2,..., h n as the sentence vector representation H.

3. The joint entity relation extraction method based on the double query network according to claim 1, characterized in that, The entity prediction in step 2.1) is specifically: Using a set of globally learnable query vectors, denoted as entity queries Each query is represented as a vector of size d, used to extract one entity; these queries are randomly initialized, the number of queries M e is experimentally set; In entity prediction, first convert the entity query using the entity decoder to obtain the entity embedding representation; then design the entity classifier and entity locator to obtain the predicted entity; In the entity decoder, L stacked Transformer layers are used for decoding, using multi-head self-attention mechanisms between the entity queries Q e to model the dependency between entities, and using multi-head cross-attention mechanisms to obtain entity-aware contextual information from the sentence vector representation H; through L decoding layers, the entity queries Q e are decoded into intermediate vectors E′, and the above process is represented as: E' = Entity-Decoder(Q e ; H) wherein According to E' obtain M e An entity embedding representation E, wherein The implementation of the entity classifier is as follows: for M e entity embedding representations E, input them into a one-dimensional full connection layer FFN to predict the class of the corresponding entity; since M e is greater than the number of gold entities in the sentence s, an additional empty label is added to represent no query entity; then the probability that the ith entity belongs to class c is calculated The calculation formula is as follows: wherein and denote trainable parameters, E i denotes the i-th entity embedding representation, denotes a set of possible entity classes; The implementation of the entity locator is as follows: in order to predict the left and right boundaries of each entity, the entity embedding representation E is first spliced with the sentence vector representation H to obtain a spliced vector Then the probability that the kth word in the sentence is the left boundary of the entity i is obtained by using a one-dimensional full connection layer FFN The calculation formula is as follows: where n is the number of words in the sentence, and denote trainable parameters, S ik denotes the i-th entity embedding representation concatenated with the vector representation of the k-th word; the probability of the right boundary computed as above.

4. The joint entity relation extraction method based on the double query network according to claim 1, characterized in that, The relation prediction in step 2.2) is specifically: Using a set of global learnable query vectors, denoted as relational queries Each query is represented as a vector of size d, used to extract one relation; These queries are randomly initialized, the number M of queries r is set by the experiment; In relation prediction, first convert the relation query using the relation decoder to obtain the relation embedding representation; Then, design the relation classifier and relation pointer to obtain the relation triple; In the relation decoder, L stacked Transformer layers are used for decoding, utilizing multi-head self-attention mechanisms between the relations Q r to model the dependency between relations, and utilizing multi-head cross-attention mechanisms to obtain relation-aware context information from the sentence vector representation H; through L decoding layers, the relation query Q r is decoded into an intermediate vector R ′ , and the above process is represented as: R ′ = Relation-Decoder(Q r ; H) wherein According to R' obtain M r A relational embedding representation R, wherein The implementation of the relation classifier is as follows: for M r relation embedding representations R, input them into a one-dimensional full connection layer FFN to predict the category of the corresponding relation; since M r is greater than the number of golden relations in the sentence s, an additional empty label is added to represent that no relation is found; then the probability that the ith relation belongs to category c is calculated The calculation formula is as follows: wherein and denote trainable parameters, R i denotes the i-th relation embedding representation, denotes a set of possible relation classes; The implementation of the relation pointer is as follows: in order to predict the relation triple, the relation pointer is used to connect the relation embedding representation with the corresponding subject-object entity embedding representation; first, the relation embedding representation R is spliced with the entity embedding representation E to obtain a spliced vector Then the probability that the kth entity belongs to the subject object of the ith relation is obtained by using a one-dimensional full connection layer FFN The calculation formula is as follows: where M e is the number of entity embedding representations, and denote trainable parameters, U ik denotes a concatenation vector of the ith relation embedding representation and the kth entity embedding representation; the probability of the object being the subject is computed as above.

5. The joint entity relation extraction method based on the double query network according to claim 1, characterized in that, In step 2.3), the entity and relation task interaction is realized through a task interaction module, specifically: Interaction between entity query / relation query is used to capture the dependency between entity extraction NER / relationship extraction RE; first, the intermediate vector E' obtained by decoding the entity query and the intermediate vector R' obtained by decoding the relationship query are linearly projected by using the full connection layer FFN; then, the projection results of them are spliced, and the entity embedding representation is obtained through the multi-head attention mechanism layer Interaction and the relationship embedding representation The above process is represented as: where W' = W + W e W' = W + W r represent trainable parameters.

6. The joint entity relation extraction method based on the double query network according to claim 1, characterized in that, In step 3), the optimal labels for entities and relations are assigned according to the dichotomy matching algorithm, and the model training is completed based on the cross-entropy-based negative logarithm loss, specifically: After the predicted entity probabilities and predicted relation probabilities are computed, to perform end-to-end training, a label matcher is designed to match the predictions to the gold labels y, the labels are padded to the same number of predictions for non-entities or non-relations; for matching entities, the predicted entity is defined as where is the entity class probability, is the entity boundary probability; the gold entity y i is defined as y i = (t i , l i , r i ), where t i is the entity class, l i , r i is the entity boundary; to find the optimal match, the minimum cost permutation σ ∈ Θ between M e entity queries and gold entities is searched: where Θ is the set of all cost permutations, defined as is: wherein, are the permutations of the set of Θ corresponding to the i-th entity query, and the optimal matching in the set of Θ is obtained using the bipartite matching algorithm i are the corresponding entity class probability, entity left boundary probability, entity right boundary probability, and σ i is the permutation of the i-th entity query in a set of Θ, and the optimal matching in the set of Θ is obtained using the bipartite matching algorithm Then, the model training loss is defined according to the optimal matching and the entity prediction probability is: where M e is the number of entity queries, is the class probability of the i-th entity query matching an entity, is the boundary probability of the i-th entity query matching an entity; the matching relationship is the same as above.

7. The joint entity relation extraction method based on the double query network according to claim 1, characterized in that, In step 4), after fine-tuning the entity and relation extraction model parameters through the training data set, input the test text to be extracted into the entity and relation extraction model; the entity classifier and entity locator calculate the predicted entity probability, and the relation classifier and relation pointer calculate the predicted relation probability; then extract the entity and relation by taking the maximum probability operation, specifically: For entity extraction, the predicted entity for the ith entity query is and is the boundary of the entity, is the class of the entity; wherein, is the probability that the kth word in the sentence is the left boundary of entity i, is the probability that the kth word in the sentence is the right boundary of entity i, is the probability that the ith entity belongs to class c; entities with predicted type null will be filtered out; For relation extraction, the predicted relation of the ith relation query is and is the object of the relation, is the category of the relation; wherein, is the probability that the kth entity is the object object of the ith relation, is the probability that the kth entity is the subject object of the ith relation, is the probability that the ith relation belongs to the category c; the predicted relation of type null will be filtered out.

Images