A document-level relation extraction method and device, electronic equipment and storage medium

Abstract

The application relates to a document-level relation extraction method and device, electronic equipment and a storage medium. The document-level relation extraction method comprises the following steps: obtaining a plurality of entity pairs and a document to be extracted; sequentially inputting the entity pairs and the document into an embedding layer and an encoding layer to obtain a hidden state vector; calculating the attention of each word in the document according to the inter-entity relation between each entity and other entities in the document; multiplying the actual mention set corresponding to each entity and the hidden state vector corresponding to the document to obtain the vector of each entity pair; using a graph attention mechanism to obtain the vector of the head entity and the tail entity in each entity pair respectively; and inputting the head entity vector and the tail entity vector of each entity pair into an output layer to obtain the entity relation with the highest probability as output through the output layer. The document-level relation extraction method solves the problem that some entity mentions may be irrelevant to the relation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a document-level relation extraction method and device, electronic equipment and storage medium. BACKGROUND

[0002] Natural Language Processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language.

[0003] Among them, the document-level relation extraction aims to identify the relationship entities between the two. The entity may correspond to multiple entity mentions across sentences. Document-level relation extraction is usually after entity extraction and entity linking. After identifying the key entities in the document, the semantic relationship between two entities or multiple entities also needs to be extracted. Semantic relationship is usually used to connect two entities and express the main meaning of the text together with the entity.

[0004] Generally, the extraction of the relationship vector between the entity pairs of the document-level relation extraction method based on the pre-training model is to input the entire document into the pre-training language model (such as Bert) to obtain the attention vector of all words in the entire document. Then the attention vectors of the entity mentions corresponding to the entities are added, which is the relationship vector between the entity pairs. However, the attention is the attention obtained by training the pre-training model on the document text. Because the attention of the pre-training model is more for the pre-training task, not the document-level relation extraction task. Therefore, the attention mechanism cannot well pay attention to the words that should be focused on in the document-level relation extraction.

[0005] In addition, the pre-training model based on the addition of the word vectors of the entity mentions corresponding to the entity pairs in the document as the word vector of the entity. The above method will produce a problem: these methods treat each entity mention equally, and then generate a word vector for the entity. Then, the entity word vector is used for relationship classification of all entity pairs. When classifying the relationship of a specific entity pair, some entity mentions may be irrelevant to the relationship. SUMMARY

[0006] Based on this, the purpose of the present application is to provide a document-level relation extraction method and device, electronic equipment and storage medium, and propose an entity relationship information attention mechanism and an entity mention selection attention mechanism. Thus, the problem of the structured attention network adding multiple structured information to the pre-training model for re-training, thereby causing a burden to the pre-training model and producing a negative effect, is avoided, and the problem of previous entities being treated equally, and some entity mentions being irrelevant to the relationship, is solved.

[0007] In a first aspect, the present application provides a document-level relation extraction method, comprising the following steps:

[0008] Obtaining a plurality of entity pairs and a document to be extracted;

[0009] Sequentially inputting the entity pairs and the document into an embedding layer and an encoding layer to obtain a hidden state vector corresponding to the document, an entity mention set corresponding to each of the entity pairs, and a hidden state vector of the entity mention set;

[0010] According to the inter-entity relationship between each entity and other entities in the document, an inter-entity relationship matrix corresponding to the document is constructed, and the attention of each word in the document is calculated;

[0011] According to the multiplication of the actual mention set corresponding to each entity and the hidden state vector corresponding to the document, a vector of each entity pair is obtained;

[0012] Using a graph attention mechanism, an entity pair is used as a main node, and the entity mentions of the entities in the entity pair are used as slave nodes to construct an entity mention graph; and according to the entity mention graph, a graph attention mechanism is used to obtain vectors of the head entity and the tail entity in each entity pair, respectively;

[0013] For each entity pair, the head entity vector and the tail entity vector are input into an output layer, so that the output layer splices the head entity vector and the tail entity vector together, and the probability distribution of the entity relationship corresponding to the head entity and the tail entity is obtained through the output layer, and the entity relationship with the highest probability is selected as the output.

[0014] Further, according to the inter-entity relationship between each entity and other entities in the document, an inter-entity relationship matrix corresponding to the document is constructed, and the attention of each word in the document is calculated, comprising:

[0015] According to the following relationship, the inter-entity relationship matrix C corresponding to the document is constructed: if two entities have a relationship, the values of their entity mentions at the intersection position of the matrix are 1, and if there is no relationship, the values are 0;

[0016] The query / key / value vector values about the inter-entity relationship information attention mechanism are constructed using the following formula:

[0017]

[0018]

[0019]

[0020] wherein W Q ,W K ,W V are randomly initialized weight values, and i is the position of the word in the text;

[0021] The score of each word in the word is calculated using the following formula:

[0022]

[0023] wherein s ij represents the score of the jth word in the ith word, A ij represents the structured information hidden state value, and if the value of C ij is 1, the initialized value of A ij is a random value, otherwise the initialized value of A ij is 0;

[0024] The attention of each word is calculated according to the score of each word using the following formula:

[0025]

[0026] wherein a i is the attention of the ith word, m is the number of words, j and k are subscripts, respectively representing the jth word and the kth word in the m words, and exp is the exponential function.

[0027] Further, the vector of each entity pair is obtained by multiplying the actual mention set corresponding to each entity and the hidden state vector corresponding to the document, including:

[0028] The attention about the entity is calculated according to the entity mention set using the following formula:

[0029]

[0030] wherein a is the attention of the ith entity to all the words in the document, is the number of entity mentions of the ith entity, refers to the jth (there are The attention of an entity mentions to all words in the document, where exp is an exponential function; the attention of two entity pairs is obtained by multiplying their attention using the following formula:

[0031] A (s,o) =A s A o

[0032] Among them, A (s,o) For the attention of this entity pair, A s and A o These are the attention to the head entity and the attention to the tail entity, respectively.

[0033] The entity pair attention and hidden state H are multiplied using the following formula to obtain a vector about the entity pair:

[0034] f (s,o) =H T A (s,o)

[0035] Among them, f (s,o) Let H be the feature vector of the entity pair (s,o), and let H be the sequence of hidden states of all words in the document, i.e., H = {h1,h2,…,h...} m}, where h is the hidden state of a single word, T represents the transpose of the matrix, and m is the number of words in the document; A (s,o) For entity pair (s,o), focus on all words in the document.

[0036] Furthermore, based on the entity mention graph, a graph attention mechanism is used to obtain the vectors of the head entity and the tail entity in each entity pair, including:

[0037] Based on the entity mention graph, the attention of the head entity and the tail entity to its mentions is obtained using the graph attention mechanism, as shown in the following formula:

[0038]

[0039]

[0040]

[0041]

[0042] Where T and W are randomized weight matrices; f (s,o) Let (s,o) be the feature vector of the entity pair. This refers to the hidden state mentioned by the j-th entity of entity S. and The number of entity mentions for entity S and entity o, respectively; and are the jth entity mention attention of entity s and entity o on all words in the document after the graph attention mechanism, respectively; are the jth entity mention attention of entity s and entity o on all words in the document after the graph attention mechanism, respectively; is the number of entity mentions of entity o; g s and g o are the attention of entity s and entity o on all words in the document, respectively.

[0043] The head entity and tail entity vectors are obtained using entity mention attention using the following formula:

[0044] h s = g s H

[0045] h o = g o H

[0046] where h s and h o represent the hidden states of entity s and entity o, respectively, g s and g o are the attention of entity s and entity o on all words in the document, respectively, and H is a sequence composed of hidden states of all words in the document.

[0047] Further, the output layer is composed of a layer of feedforward neural network and a layer of Softmax layer.

[0048] Further, the head entity vector and the tail entity vector are defined to pass through the output layer to obtain a probability distribution P(y|x) using the following formula:

[0049] P(y|x) = Softmax(FFN(concat(h s ,h o ))

[0050] where P(y|x) is the probability distribution of the entity relationship corresponding to the head entity and the tail entity.

[0051] Further, the document is sequentially input into the embedding layer and the encoding layer to obtain the hidden state vector corresponding to the document, including:

[0052] A [CLS] label is added at the head of the text of the document, the [CLS] label at the head is used to represent the input sequence of the whole paragraph, a [SEP] label is added between the sentence pairs, the [SEP] label in the middle is used to distinguish the two sentences, and a [SEP] label is added at the end of the text, which is used to represent the end;

[0053] Based on the vocabulary, convert the document into a sequence T = {t1, t2, ..., t}. m}, where m is the sequence length;

[0054] The embedding layer is used to convert the sequence T into a vector matrix V = {v1, v2, ..., v}. m};

[0055] After the matrix vector V passes through the encoding layer, the hidden state H = {h1, h2, ..., h} is obtained. m}

[0056] Secondly, the present invention also provides a document-level relationship extraction apparatus, comprising:

[0057] The entity pair and document acquisition module is used to acquire several entity pairs and documents for which the relations to be extracted are to be acquired.

[0058] The hidden state vector acquisition module is used to sequentially input the entity pair and the document into the embedding layer and the encoding layer to obtain the hidden state vector corresponding to the document, the entity mention set corresponding to each entity in the entity pair, and the hidden state vector of the entity mention set.

[0059] The word attention calculation module is used to construct the inter-entity relationship matrix corresponding to the document based on the inter-entity relationship between each entity in the document and other entities, and to calculate the attention of each word in the document;

[0060] The entity pair vector calculation module is used to multiply the actual mention set corresponding to each entity and the hidden state vector corresponding to the document to obtain the vector of each entity pair;

[0061] The head entity and tail entity vector calculation module is used to construct an entity reference graph by using a graph attention mechanism, taking entity pairs as master nodes and entity references of entities in the entity pair as slave nodes; and using the graph attention mechanism based on the entity reference graph, to obtain the vectors of the head entity and tail entity in each entity pair respectively.

[0062] The entity relationship output module is used to input the head entity vector and the tail entity vector into the output layer for each entity pair, so that the output layer concatenates the head entity vector and the tail entity vector together, and obtains the probability distribution of the entity relationship corresponding to the head entity and the tail entity through the output layer, and selects the entity relationship with the highest probability as the output.

[0063] Thirdly, the present invention also provides an electronic device, comprising:

[0064] At least one memory and at least one processor;

[0065] The memory is configured to store one or more programs.

[0066] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the document-level relation extraction method according to any one of the first aspect of the present application.

[0067] In a fourth aspect, the present application further provides a computer-readable storage medium,

[0068] The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the document-level relation extraction method according to any one of the first aspect of the present application.

[0069] The present application provides a document-level relation extraction method, device, electronic equipment and storage medium, and proposes an entity relationship information attention mechanism and an entity mention selection attention mechanism. The entity relationship information attention mechanism only has one kind of structured information: there is a certain relationship between entities, and the pre-trained model is not retrained, but the vector output is directly obtained from the pre-trained model, so as to avoid the problem that the structured attention network adds multiple structured information to the pre-trained model for retraining, thereby causing a burden to the pre-trained model and producing a negative effect. Moreover, the entity-level attention is not directly obtained by transferring the attention head from the pre-trained language model, but the entity-level attention is extracted according to the entity structured information, so as to solve the problem that the local context pool technology cannot well focus on the words that should be focused on in the document-level relation extraction. The entity mention selection attention mechanism constructs the entity mentions of the same entity into an undirected graph, and then uses a graph attention mechanism to reacquire the attention of each entity mention of the same entity in the entity pair, and reconstructs the entity word vector according to the attention. Thus, the problem that all entity mentions are treated equally and some entity mentions may be irrelevant to the relationship is solved.

[0070] In order to better understand and implement, the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 A step schematic diagram of the document-level relation extraction method provided by the present application;

[0072] Figure 2 An exemplary entity relationship matrix schematic diagram;

[0073] Figure 3 A structure schematic diagram of the document-level relation extraction model used in an embodiment of the present application;

[0074] Figure 4A structural schematic diagram of a document-level relation extraction device provided by the present application. DETAILED DESCRIPTION

[0075] For the purpose, technical solutions and advantages of the present application to be clearer, the following will further describe the embodiments of the present application in conjunction with the drawings.

[0076] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0077] The terms used in the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more associated listed items.

[0078] The following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application, as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not necessarily describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0079] In addition, in the description of the present application, "multiple" means two or more, unless otherwise specified. The association between the associated objects is described as "and / or", which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents a "or" relationship between the associated objects before and after it.

[0080] To solve the problems in the background art, the application provides an inter-entity relationship information attention mechanism and an entity mention selection attention mechanism. The inter-entity relationship information attention mechanism only has one kind of structured information: whether there is a certain relationship between entities, and instead of retraining a pre-trained model, a vector output is directly obtained from the pre-trained model. Instead of directly transferring an attention head from a pre-trained language model to obtain entity-level attention, entity-level attention is re-extracted according to inter-entity structured information. The entity mention selection attention mechanism constructs entity mentions of the same entity into an undirected graph, and then uses a graph attention mechanism to re-obtain the attention of each entity mention of the same entity in the entity pair, and reconstructs an entity word vector according to the attention.

[0081] Based on the above inter-entity relationship information attention mechanism and entity mention selection attention mechanism, the application provides a document-level relationship extraction method, as shown in Figure 1 The method comprises the following steps:

[0082] S01: Obtain a plurality of entity pairs and a document to be extracted.

[0083] In one specific embodiment, the document-level relationship extraction method provided by the application is a classification task, which classifies the relationship of the input entity pair (s, o). The document-level extraction model used by the application is a combination of a pre-trained language model and a graph neural network-based model. The application defines the relationship set of the entity pair as Y={y1,y2,…,y k}, where k is the number of categories. Then define the entity pair input as X={x1,x2,…,x d}, X is an entity pair, and d is the number of entity pairs. The purpose of the semantic matching task is to predict the category Y according to the input X.

[0084] S02: sequentially input the entity pairs and the document into an embedding layer and an encoding layer to obtain a hidden state vector corresponding to the document, an entity mention set corresponding to each of the entity pairs, and a hidden state vector of the entity mention set.

[0085] In a preferred embodiment, the application uses a BERT pre-trained language model as a sentence encoder. BERT includes an embedding layer and an encoding layer. First, the input document text is preprocessed, a [CLS] label is added at the head of the text, the [CLS] label at the head is used to represent the input sequence, a [SEP] label is added between the sentence pairs, the [SEP] label in the middle is used to distinguish the two sentences, and a [SEP] label is added at the end of the text to indicate the end; then the text is converted into T={t1,t2,…,t m}, where m is the sequence length. Finally, the embedding layer converts the sequence T into a matrix of vectors V = {v1, v2, …, v m}.

[0086] The structure of the encoding layer is composed of 12 identical Transformer encoder blocks stacked together, each of which contains multi-head self-attention mechanism and feed-forward neural network, and applies residual connection and layer normalization to the two sub-structures, and the output of each layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is multi-head self-attention mechanism or feed-forward neural network. The matrix vector V is converted into hidden state H = {h1, h2, …, h m} after the encoding layer. The output of BERT is defined as:

[0087] (h1, h2, …, h m ) = BERT(x) (1)

[0088] Then, the entity mention set of entity z i is defined as: and the hidden state of the entity mention set is defined as: The number of entity mentions of entity z i is defined as: The meaning of the i-th entity mention of the j-th entity is the hidden state vector.

[0089] The entity mention refers to the word in which the entity appears in the text. For example, if there is a document: Beijing is the capital of China. The population of Beijing is about 20 million. Then the entity mention of the entity Beijing is the word “Beijing” in the first sentence and the word “Beijing” in the second sentence.

[0090] Finally, the entity pair is defined as (z s , z o ). z s represents the head entity in the entity pair, and z o represents the tail entity in the entity pair.

[0091] S03: According to the inter-entity relationship between each entity and other entities in the document, an inter-entity relationship matrix corresponding to the document is constructed, and the attention of each word in the document is calculated.

[0092] Preferably, the following sub-steps are included:

[0093] S031: According to the following relationship, an inter-entity relationship matrix C corresponding to the document is constructed:

[0094] The present application first defines a kind of structured information about entities: inter-entity relationship. If there is a certain relationship between a certain entity and another entity, the present application considers that there is an inter-entity relationship between the two entities. Thus, the text constructs an inter-entity relationship matrix C according to the input document text. As shown in Figure 2 If two entities have a relationship, their entity mentions have a value of 1 at the intersection of the matrix, and if there is no relationship, the value is 0.

[0095] S032: Construct the query / key / value vector values of the inter-entity relationship information attention mechanism using the following formula:

[0096]

[0097]

[0098]

[0099] where W Q ,W K ,W V are randomly initialized weight values, and i is the position of the word in the text.

[0100] S033: Calculate the score of each word in the word using the following formula:

[0101]

[0102] where s ij represents the score of the jth word in the ith word, A ij represents the structured information hidden state value, and if the value of C ij is 1, the initialized value of A ij is a random value, otherwise the initialized value of A ij is 0.

[0103] S034: Calculate the attention of each word according to the score of each word using the following formula:

[0104]

[0105] where a i is the attention of the ith word, m is the number of words, j and k are indices representing the jth word and the kth word in the m words, respectively, and exp is the exponential function.

[0106] S04: Multiply the actual mention set corresponding to each entity pair by the hidden state vector corresponding to the document to obtain the vector of each entity pair.

[0107] Preferably, the following sub-steps are included:

[0108] S041: Using the following formula, calculate the attention for an entity based on the entity mention set:

[0109]

[0110] in, For the i-th entity, pay attention to all words in the document. Let i be the number of entity mentions for the i-th entity. It refers to the j-th entity among the i-th entities (out of a total of...). (a) entity mentions attention to all words in the document, where exp is an exponential function.

[0111] S042: Use the following formula to multiply the attention of two entities to obtain the attention for that entity pair:

[0112] A (s,o) =A s A o

[0113] Among them, A (s,o) For the attention of this entity pair, A s and A o These are the attention to the head entity and the attention to the tail entity in the entity pair, respectively.

[0114] S043: Use the following formula to multiply the entity pair attention and hidden state H to obtain a vector about the entity pair:

[0115] f (s,o) =H T A (s,o)

[0116] Among them, f (s,o) Let H be the feature vector of the entity pair (s,o), and let H be the sequence of hidden states of all words in the document, i.e., H = {h1,h2,…,h...} m}, where h is the hidden state of a single word, T represents the transpose of the matrix, and m is the number of words in the document; A (s,o) For entity pair (s,o), focus on all words in the document.

[0117] S05: Using a graph attention mechanism, an entity mention graph is constructed by taking entity pairs as master nodes and entity mentions of entities in the entity pair as slave nodes; and based on the entity mention graph, the vectors of the head entity and tail entity in each entity pair are obtained by using the graph attention mechanism.

[0118] Preferably, it includes the following sub-steps:

[0119] S051: According to the entity mention graph, the attention of the head entity and the tail entity to its entity mention is obtained respectively using the graph attention mechanism, and the formula is as follows:

[0120]

[0121]

[0122]

[0123]

[0124] wherein T and W are randomized weight matrices; f (s,o) is the feature vector of the entity pair (s, o), refers to the hidden state of the jth entity mention of entity S, and are the number of entity mentions of entity S and entity o, respectively; and are the attention of the jth entity mention of entity s and entity o to all words in the document after the graph attention mechanism, respectively; is the attention of the jth entity mention of entity o to all words in the document after the graph attention mechanism; is the number of entity mentions of entity o; g s and g o are the attention of entity S and entity o to all words in the document, respectively.

[0125] S052: The vectors of the head entity and the tail entity are obtained using the entity mention attention using the following formula:

[0126] h s =g s H

[0127] h o =g o H

[0128] wherein h s and h o represent the hidden state of entity s and entity o, respectively, g s and g o are the attention of entity S and entity o to all words in the document, respectively, and H is a sequence composed of the hidden states of all words in the document.

[0129] S06: for each entity pair, input the head entity vector and the tail entity vector into an output layer, so that the output layer concatenates the head entity vector and the tail entity vector together, obtains the probability distribution of the entity relationship corresponding to the head entity and the tail entity through the output layer, and selects the entity relationship with the highest probability as the output.

[0130] Preferably, the output layer is composed of a layer of forward feedback neural network FFN and a Softmax function.

[0131] The probability distribution P(y|x) obtained by the head entity vector and the tail entity vector through the output layer is defined using the following formula:

[0132] P(y|x)=Softmax(FFN(concat(h s ,h o ))

[0133] Wherein, P(y|x) is the probability distribution of the entity relationship corresponding to the head entity and the tail entity.

[0134] In a preferred embodiment, the document-level relationship extraction method provided by the application uses a document-level extraction model as shown in Figure 3 Preferably, the loss algorithm of the document-level extraction model is:

[0135] Given the training data Where x i represents the i-th input entity pair, y i represents the label corresponding to the i-th input entity pair, and the training set size is n. The main learning goal of the document-level relationship extraction model is to minimize the cross-entropy loss function, and the formula is as follows:

[0136]

[0137] The application uses an attention mechanism of the relationship structure between entities, which focuses attention on the document extraction task, thereby enhancing the word vector expression of the entity. An attention mechanism of relationship mention selection is proposed, so that the model pays attention to more important entity extraction when performing relationship extraction.

[0138] The application also provides a document-level relationship extraction device, as shown in Figure 4 The document-level relationship extraction device 400 comprises:

[0139] An entity pair and document acquisition module 401 is configured to acquire a plurality of entity pairs and a document to be extracted.

[0140] The hidden state vector acquisition module 402 is configured to sequentially input the entity pair and the document into an embedding layer and an encoding layer to obtain a hidden state vector corresponding to the document, an entity mention set corresponding to each of the entity pairs, and a hidden state vector of the entity mention set.

[0141] The word attention calculation module 403 is configured to construct an inter-entity relationship matrix corresponding to the document according to inter-entity relationships between each entity in the document and other entities, and calculate attention of each word in the document.

[0142] The entity pair vector calculation module 404 is configured to multiply the actual mention set corresponding to each entity and the hidden state vector corresponding to the document to obtain a vector of each entity pair.

[0143] The head entity and tail entity vector calculation module 405 is configured to use a graph attention mechanism to construct an entity mention graph by taking the entity pair as a master node and the entity mentions of the entities in the entity pair as slave nodes, and to use the graph attention mechanism to obtain a vector of a head entity and a vector of a tail entity in each entity pair according to the entity mention graph.

[0144] The entity relationship output module 406 is configured to input the vector of the head entity and the vector of the tail entity into an output layer for each entity pair, so that the output layer splices the vector of the head entity and the vector of the tail entity together, obtains a probability distribution of an entity relationship corresponding to the head entity and the tail entity through the output layer, and selects an entity relationship with the highest probability as an output.

[0145] Preferably, the word attention calculation module comprises:

[0146] The inter-entity relationship matrix construction unit is configured to construct an inter-entity relationship matrix C corresponding to the document according to the following relationship: if two entities have a relationship, the values of their entity mentions at the intersection positions of the matrix are 1, and if there is no relationship, the values are 0.

[0147] The attention mechanism vector value construction unit is configured to construct query / key / value vector values of the inter-entity relationship information attention mechanism using the following formula:

[0148]

[0149]

[0150]

[0151] wherein W Q ,W K ,W V are randomly initialized weight values, and i is the position of the word in the text.

[0152] a word score calculation unit configured to calculate a score of each word in the word using the following formula:

[0153]

[0154] wherein s ij represents the score of the jth word in the ith word, A ij represents the structured information hidden state value, if C ij is equal to 1, then A ij is initialized to a random value, otherwise A ij is initialized to 0;

[0155] a word attention calculation unit configured to calculate the attention of each word according to the score of each word using the following formula:

[0156]

[0157] wherein a i is the attention of the ith word, m is the number of words, j and k are indices representing the jth word and the kth word in the m words, respectively, and exp is the exponential function.

[0158] Preferably, the entity pair vector calculation module comprises:

[0159] an entity attention calculation unit configured to calculate the attention of an entity according to the entity mention set using the following formula:

[0160]

[0161] wherein, is the attention of the ith entity to all words in the document, is the number of entity mentions of the ith entity, is the attention of the jth (there are entity mentions in the ith entity) entity mention of the ith entity to all words in the document, and exp is the exponential function.

[0162] an entity pair attention calculation unit configured to multiply the attention of two entities to obtain the attention of the entity pair using the following formula:

[0163] A (s,o) = A s A o

[0164] wherein A (s,o) is the attention of the entity pair, A s and A o are the attention of the head entity and the tail entity in the entity pair, respectively.

[0165] The entity pair vector computation unit is used to multiply the entity pair attention and the hidden state H using the following formula to obtain a vector for that entity pair:

[0166] f (s,o) =H T A (s,o)

[0167] Among them, f (s,o) Let H be the feature vector of the entity pair (s,o), and let H be the sequence of hidden states of all words in the document, i.e., H = {h1,h2,…,h...} m}, where h is the hidden state of a single word, T represents the transpose of the matrix, and m is the number of words in the document; A (s,o) For entity pair (s,o), focus on all words in the document.

[0168] Preferably, the head and tail entity vector calculation module includes:

[0169] The head entity and tail entity attention calculation unit is used to obtain the attention of the head entity and tail entity to the entity's mentions using a graph attention mechanism, based on the entity mention graph. The formula is as follows:

[0170]

[0171]

[0172]

[0173]

[0174] Where T and W are randomized weight matrices; f (s,o) Let (s,o) be the feature vector of the entity pair. This refers to the hidden state mentioned by the j-th entity of entity S. and The number of entity mentions for entity S and entity o, respectively; and These represent the attention paid by the j-th entity mentions of entities s and o to all words in the document after the graph attention mechanism. After the graph attention mechanism, the j-th entity mention of entity o pays attention to all words in the document; g is the number of entity mentions for entity o. s and g o Let S and o represent the attention of entities S and o to all words in the document, respectively.

[0175] a head entity and tail entity vector calculation unit, configured to obtain vectors of the head entity and the tail entity using entity mention attention using the following formula:

[0176] h s =g s H

[0177] h o =g o H

[0178] where h s and h o represent hidden states of the entity s and the entity o respectively, g s and g o are attentions of all words in the document for the entity s and the entity o respectively, and H is a sequence composed of hidden states of all words in the document.

[0179] Preferably, the output layer is composed of a layer of feedforward neural network and a layer of Softmax layer.

[0180] Preferably, the head entity vector and the tail entity vector are defined to obtain a probability distribution P(y|x) after passing through the output layer using the following formula:

[0181] P(y|x)=Softmax(FFN(concat(h s ,h o ))

[0182] where P(y|x) is a probability distribution of the entity relationship corresponding to the head entity and the tail entity.

[0183] Preferably, the hidden state vector acquisition module comprises:

[0184] a document label adding unit, configured to add a [CLS] label at the head of the text of the document, the [CLS] label at the head is used to represent the whole input sequence, add a [SEP] label between sentence pairs, the [SEP] label in the middle is used to distinguish two sentences, and add a [SEP] label at the end of the text, which is used to represent the end;

[0185] a document conversion unit, configured to convert the document into a sequence T={t1,t2,…,t m} according to a word table, where m is the length of the sequence;

[0186] a sequence conversion unit, configured to convert the sequence T into a vector matrix V={v1,v2,…,v m} using the embedding layer;

[0187] a hidden state extraction unit, configured to obtain hidden states H={h1,h2,…,h m}.

[0188] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The device embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0189] The embodiment of the present application further provides an electronic device, comprising:

[0190] at least one memory and at least one processor;

[0191] The memory is configured to store one or more programs.

[0192] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the document-level relation extraction method as described above.

[0193] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The device embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present disclosure. Those skilled in the art can understand and implement without creative labor.

[0194] The embodiment of the present application further provides a computer readable storage medium,

[0195] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the document-level relation extraction method as described above.

[0196] Computer-usable storage media include permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0197] The application provides a document-level relation extraction method and device, electronic equipment and storage medium, and proposes an entity relationship information attention mechanism and an entity mention selection attention mechanism. The entity relationship information attention mechanism only has one kind of structured information: there is a certain relationship between entities, and the pre-trained model is not retrained, but the vector output is directly obtained from the pre-trained model, so as to avoid the problem that the structured attention network adds multiple structured information to the pre-trained model for retraining, thereby causing a burden to the pre-trained model and producing a negative effect. Moreover, the entity-level attention is not directly obtained by transferring the attention head from the pre-trained language model, but is extracted according to the entity structured information, so as to solve the problem that the local context pool technology cannot well focus on the words that should be focused on in document-level relation extraction. The entity mention selection attention mechanism constructs the entity mentions of the same entity into an undirected graph, then uses a graph attention mechanism to reacquire the attention of each entity mention of the same entity in the entity pair, and reconstructs the entity word vector according to the attention. Thus, the problem that all entity mentions are treated equally and some entity mentions may be irrelevant to the relationship is solved.

[0198] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A document-level relation extraction method, characterized in that, The method comprises the following steps: Obtaining a plurality of entity pairs and a document to be extracted; Inputting the entity pairs and the document into an embedding layer and an encoding layer in sequence to obtain a hidden state vector corresponding to the document and an entity mention set corresponding to each of the entity pairs and a hidden state vector of the entity mention set; Constructing an inter-entity relationship matrix corresponding to the document according to inter-entity relationships between each entity in the document and other entities, and calculating the attention of each word in the document; Multiplying the actual mention set corresponding to each entity and the hidden state vector corresponding to the document to obtain a vector of each entity pair; Using a graph attention mechanism, taking an entity pair as a main node and the entity mentions of the entities in the entity pair as slave nodes to construct an entity mention graph, and using the graph attention mechanism to obtain a vector of a head entity and a tail entity in each entity pair according to the entity mention graph; For each entity pair, inputting the head entity vector and the tail entity vector into an output layer to make the output layer splice the head entity vector and the tail entity vector together, obtaining a probability distribution of an entity relationship corresponding to the head entity and the tail entity through the output layer, and selecting an entity relationship with the highest probability as an output.

2. The document-level relation extraction method of claim 1, wherein, According to the inter-entity relationships between each entity in the document and other entities, the inter-entity relationship matrix corresponding to the document is constructed, and the attention of each word in the document is calculated, including: According to the following relationship, the inter-entity relationship matrix C corresponding to the document is constructed: if two entities have a relationship, the value of their entity mentions at the intersection of the matrix is 1, and if there is no relationship, the value is 0; Using the following formula, the query / key / value vector values of the inter-entity relationship information attention mechanism are constructed: where W Q , W K , W V are randomly initialized weight values, and i is the position of the word in the text. Using the following formula, the score of each word in the word is calculated: where s ij represents the score of the jth word in the ith word, A ij represents the structured information hidden state value, if C ij is 1, the initialized value of A ij is a random value, otherwise the initialized value of A ij is 0; Using the following formula, the attention of each word is calculated according to the score of each word: where a i is the attention for the ith word, m is the number of words, j and k are subscripts representing the jth and kth words in the m words, respectively, and exp is the exponential function.

3. The document-level relation extraction method of claim 1, wherein, According to the actual mention set corresponding to each entity and the hidden state vector corresponding to the document, the vector of each entity pair is obtained, including: Using the following formula, the attention of the entity is calculated according to the entity mention set: where, is the attention of the ith entity pair to all words in the document, is the number of entity mentions of the ith entity, refers to the attention of the jth (out of a total of ) entity mention in the ith entity to all words in the document, exp is the exponential function; Using the following formula, the attention of the two entities is multiplied to obtain the attention of the entity pair: A (s，o) = A s A o wherein A (s，o) is the attention of the entity pair, A s and A o are the attention of the head entity and the tail entity in the entity pair, respectively; Using the following formula, the vector of the entity pair is obtained by multiplying the entity pair attention and the hidden state H: f (s，o) = H T A (s，o) where f (s，o) is the feature vector for entity pair (s, o), H is a sequence of hidden states of all words in the document, i.e., H = {h1, h2,..., h m}, h is the hidden state of a single word, T represents the transpose of a matrix, and m is the number of words in the document; A (s，o) is the attention of entity pair (s, o) on all words in the document.

4. The document-level relation extraction method of claim 1, wherein, According to the entity mention graph, the vector of the head entity and the tail entity in each entity pair is obtained using the graph attention mechanism, including: According to the entity mention graph, the attention of the head entity and the tail entity of the entity mention is obtained using the graph attention mechanism, and the formula is as follows: where T and W are randomized weight matrices; f (s，o) is the feature vector for entity pair (s, o), denotes the hidden state of the jth entity mention of entity S, and are the number of entity mentions of entity S and entity o, respectively; and are the attention of the jth entity mention of entity s and entity o on all words in the document after the graph attention mechanism, respectively; is the attention of the jth entity mention of entity o on all words in the document after the graph attention mechanism; is the number of entity mentions of entity o; g s and g o are the attention of entity S and entity o on all words in the document, respectively; Using the following formula, the vector of the head entity and the tail entity is obtained using the entity mention attention: h s = g s H h o = g o H where h s and h o represent the hidden states of entity s and entity o respectively, g s and g o are the attention of entity S and entity o on all words in the document respectively, and H is the sequence of hidden states of all words in the document.

5. The document-level relationship extraction method according to claim 1, wherein: The output layer is composed of a feedforward neural network and a Softmax layer.

6. The document-level relationship extraction method according to claim 5, wherein: Using the following formula, the probability distribution P(y|x) obtained by the head entity vector and the tail entity vector through the output layer is defined: P(y | x) = Softmax(FFN(concat(h s , h o ))) Wherein, P(y|x) is the probability distribution of the entity relationship corresponding to the head entity and the tail entity.

7. The document-level relation extraction method of claim 1, wherein, The document is sequentially input into an embedding layer and an encoding layer to obtain a hidden state vector corresponding to the document, including: Adding a [CLS] label at the head of the text of the document, the [CLS] label at the head is used to represent the entire input sequence, adding a [SEP] label between the sentence pairs, the [SEP] label in the middle is used to distinguish the two sentence pairs, and adding a [SEP] label at the end of the text, which is used to indicate the end; The document is converted into a sequence T = {t1, t2,..., tm} according to a vocabulary, where m is the sequence length. m}, where m is the sequence length. using the embedding layer to convert the sequence T into a vector matrix V = {v1, v2,..., vn} where n is the length of the sequence T. m} ; The matrix vector V is obtained after the encoding layer to get the hidden state H = {h1, h2, …, h m}.

8. A document-level relation extraction apparatus characterized by comprising: Including: An entity pair and document acquisition module is configured to acquire a plurality of entity pairs and a document to be extracted; A hidden state vector acquisition module is configured to sequentially input the entity pairs and the document into an embedding layer and an encoding layer to obtain a hidden state vector corresponding to the document, an entity mention set corresponding to each of the entity pairs, and a hidden state vector of the entity mention set. A word attention calculation module is configured to construct an inter-entity relationship matrix corresponding to the document according to the inter-entity relationship between each entity in the document and other entities, and calculate the attention of each word in the document. An entity pair vector calculation module is configured to multiply the actual mention set corresponding to each entity and the hidden state vector corresponding to the document to obtain a vector of each entity pair. A head entity and tail entity vector calculation module is configured to use a graph attention mechanism to construct an entity mention graph by taking an entity pair as a master node and the entity mentions of the entity pair as slave nodes, and to use the graph attention mechanism to obtain a vector of the head entity and the tail entity in each entity pair according to the entity mention graph. An entity relationship output module is configured to input the head entity vector and the tail entity vector into an output layer for each entity pair, so that the output layer concatenates the head entity vector and the tail entity vector together, and obtains the probability distribution of the entity relationship corresponding to the head entity and the tail entity through the output layer, and selects the entity relationship with the highest probability as the output.

9. An electronic device, comprising: Including: At least one memory and at least one processor; The memory is configured to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the document-level relationship extraction method according to any one of claims 1-7.

10. A computer readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the document-level relationship extraction method according to any one of claims 1-7.

Images