Remote supervision relation extraction method based on multilevel classification hierarchical attention

Abstract

The invention discloses a remote supervision relation extraction method based on multilevel classification hierarchical attention. A remote supervision relation extraction (MCHRE) model based on multilevel classification hierarchical attention is used. According to the model, a hierarchical structure of a relationship is modeled by utilizing a graph attention network (GATS), an implicit semantic relationship between relationship categories is deeply mined, and relationship embedding for perceiving a global hierarchical structure is generated; in a classification strategy part, a hierarchical attention mechanism is taken as a basis, a multi-level classification structure is introduced, a classification process with finer granularity is realized, knowledge is guided to migrate from a head relation to a tail relation, so that data noise reduction is realized, and the extraction performance of a long-tail relation is effectively improved. The method has strong anti-noise interference capability, is excellent in relation extraction performance, is comprehensively superior to a baseline model in long-tail extraction performance, and is an effective method for solving the data long-tail problem in remote supervision relation extraction.

Description

Technical Field

[0001] The invention relates to a remote supervision relationship extraction method based on multi-level classification hierarchical attention, belonging to the technical field of information extraction. Background Art

[0002] In recent years, the mobile Internet has developed tremendously, which has further lowered the threshold for people to enter the Internet. A large number of Internet users are generating a large amount of data every moment, including pictures, text, voice and video. How to obtain real and effective information from these data has become a current research hotspot, among which text, the oldest form of human data, is the top priority. Therefore, researchers have proposed information extraction (IE) technology, which aims to quickly process low-structured text and form concise and easy-to-store structured data. Information extraction is an important research content in the field of natural language processing (NLP). It plays an important role in the construction of various natural language applications, including knowledge graphs, search engines, question-answering systems and recommendation systems. Information extraction mainly includes three subtasks: named entity recognition (NER), relation extraction (RE) and event extraction (EE), which correspond to the three factual information of entity, relation and event in the text respectively. Relation extraction is an important subtask of IE. It predicts the semantic relationship between entities based on the entities marked by NER. For relationship extraction tasks, given two entities and the text containing the entity pair, it is necessary to predict the relationship between the entity pair based on the text semantics. For example, given the two entities "Purple Mountain" and "Nanjing City", and the sentence "Purple Mountain is located in Xuanwu District, Nanjing City.", by analyzing the semantics of the sentence, it can be obtained that "Purple Mountain" and "Nanjing City" have a "located in" relationship. If the predicted relationship type is predefined, relationship extraction can be regarded as a relationship classification task.

[0003] Remote supervision relationship extraction skips the manual construction of data sets in supervised learning, which significantly reduces time and labor costs, and further lowers the threshold for the application of relationship extraction technology in various vertical fields. While remote supervision algorithms bring huge benefits, they are also inevitably accompanied by label noise problems and data long-tail problems. How to use imperfect remote supervision data sets for effective relationship extraction training has become the focus and difficulty of remote supervision relationship extraction. The remote supervision relationship extraction technology of the present invention aims to train a robust and balanced relationship extractor on a noisy and unbalanced remote supervision data set. Summary of the invention

[0004] Purpose of the invention: In view of the problems and deficiencies in the prior art, the present invention provides a remote supervised relationship extraction method based on multi-level classification hierarchical attention, which integrates the attention mechanism with the relationship extraction model, and makes full use of the ability of the attention mechanism to dynamically allocate "information attention", so that the information of each sample can be used reasonably to the greatest extent. In-depth research on remote supervised relationship extraction will help to promote relationship extraction to a wider range of application scenarios, and has extraordinary research and practical value.

[0005] Technical solution: A remote supervised relationship extraction method based on multi-level classification hierarchical attention. The present invention proposes a hierarchical structure encoder based on a graph attention network, which deeply mines the associated information in the hierarchical structure, provides more dependent paths for knowledge dissemination, and outputs the relationship embedding that perceives the global hierarchical structure; introduces a multi-level classification framework, combined with hierarchical attention, to achieve multi-granularity relationship classification, which can effectively utilize the feature information of sentence packages and the associated information in the hierarchical structure, achieve soft data denoising, and improve the extraction performance of long-tail relationships. The method includes the following steps:

[0006] Step 1: Get the sentence bag, use the sentence encoder to extract the features of a single sentence, and get the sentence feature vector.

[0007] Step 2: Perform a hierarchical attention mechanism to obtain bag representations of sentence bags at different levels using the hierarchical structure of relations.

[0008] Step 3: The hierarchical encoder takes the hierarchy of relations as input, adopts graph attention networks (GATs) to model the hierarchy of relations, and outputs global structure-aware relation embeddings.

[0009] Step 4: Take the package representations and relation embedding matrices at each level as input and input them into the multi-level classification framework. Execute the local classification strategy multiple times inside it, calculate the matching score, and then input softmax to obtain the relation classification probability distribution of the current layer.

[0010] Step 5: According to the classification probability distribution of each level of relationship and the hierarchical structure of the relationship, construct the expected probability distribution for the previous level. The smaller the distribution difference between the expected probability distribution and the actual probability distribution, the more reasonable the classification results of the adjacent levels, thereby achieving the unification of multi-level classification results.

[0011] Step 6: Calculate the final loss by combining the hierarchical loss function and the loss function of the local probability constraint.

[0012] In step 1, the feature extraction of the sentence can be completed by a sentence encoder based on deep learning, such as CNN, PCNN, etc.

[0013] In step 2, hierarchical attention is based on sentence-level selective attention, and the hierarchical structure of relations is used to improve the extraction effect of long-tail relations. Sentence-level selective attention is used once at each level corresponding to the true relation r, and the bag representation obtained at each level is concatenated and input into the softmax classifier for classification. A normalized weight distribution is generated for the sentence bag through the attention query mechanism, and then the bag representation of the entire bag is obtained by weighted summation, and finally the bag representation is used to predict a relationship label for the corresponding entity pair.

[0014] Given a sentence bag B = {s 1 ,s 2 ,…,s n},s i represents the feature vector of the i-th sentence in the bag, s i The attention weight value is:

[0015]

[0016] Among them, e i It represents the relevance value of the ith sentence to the true relation r, and its value is obtained by performing a dot product operation with the attention query vector:

[0017]

[0018] Among them, q r The attention query vector representing the correspondence r.

[0019] Finally, the obtained attention weight distribution is weighted and summed with the sentence features in the bag to obtain the bag representation of the bag:

[0020]

[0021] For ease of representation, the process of obtaining bag representation using the sentence-level attention mechanism is simplified to the following formula:

[0022] b h,t =ATT(q r ,{s 1 ,s 2 ,…,s n})

[0023] The relationship label corresponding to the sentence bag is r. According to the hierarchy of relationships, r is at the bottom of the hierarchy. From this point, we trace back to its ancestor relationship node until the root node. The relationship nodes on the path form a relationship chain (r 0 ,r1 ,r 2 ,…,r k ), where r = r k , r 0 is a virtual node, which is the root node of the entire hierarchy. Then, multiple selective attentions are performed using the relationship nodes on the relationship chain to extract the bag representation of the sentence bag at each relationship level, where the bag representation of the i-th layer is calculated as follows:

[0024]

[0025] in, Represents the attention query vector of relation r at layer i. In the same way, we obtain the bag representation set of sentence bags at all levels, i.e.

[0026] In step 3, the hierarchical encoder regards the hierarchical structure as an undirected graph, uses GATs to model the hierarchical structure of the relationship, and outputs a relation embedding that perceives the global structure.

[0027] The hierarchical encoder is formed by stacking multiple layers of GATs. Each layer of GATs takes the embeddings of all relation nodes in the relation hierarchy as input. Assuming there are n z nodes, then the input set of relation embedding can be expressed as And satisfy The input of the first layer of GATs is randomly initialized, and the input of the remaining GATs is the output of the previous layer. The core principle of GATs is to calculate the attention weights between nodes. The weight calculation between node i and node j is as follows:

[0028] α i,j =softmax j (LeakyRelu(W att [Wh i Wh j ]+b att ))

[0029] Among them, LeakyRelu(·) is a nonlinear activation function, b att is a learnable weight parameter, is the attention vector, and softmax() is used to obtain a normalized weight value.

[0030] After calculating the attention weight matrix, the following neighbor aggregation operation is used to obtain the updated relational embedding of node i:

[0031]

[0032] Among them, N irepresents the set of nodes that have an adjacent relationship with node i, is the learnable weight parameter matrix, is the bias term, and σ(·) represents the nonlinear activation function.

[0033] Get all updated relationship embeddings In order to make the above process stable and explore more subspaces, a multi-head attention mechanism is adopted. The new relationship embedding update process can be expressed as follows:

[0034]

[0035] Among them, H represents the number of heads of multi-head attention. After obtaining the embeddings of all relationship nodes, the relationship embeddings of the same level are stacked to form k-level embedding matrices {R (1) ,R (2) ,…,R (k)}, input into the multi-level classification hierarchical attention for matching classification.

[0036] In step 4, the multi-level classification framework is connected after the hierarchical attention, and takes the packet representation of each level as input to perform multiple local classifications. The multi-level classification framework consists of multiple local classifiers, which internally implement the local classification strategy, use the packet representation and the relationship embedding matrix to calculate the matching score, and then input the softmax to obtain the relationship classification probability distribution of the current layer. Assuming that local classification is performed at level j, is the embedding matrix formed by stacking the embedding vectors of all relationship nodes at level j, N j represents the number of j-level relationship nodes, d g is the dimension of the relation embedding vector. First, the bag is represented as Input into the perceptron to complete the dimensional transformation and obtain the feature vector s required for classification j , the conversion process is as follows:

[0037]

[0038] in, is the trainable parameter matrix, d c represents the dimension of the package representation, b l represents the bias term, and ReLU is the rectified linear unit.

[0039] The j-th layer relation embedding matrix and classification feature vector s j Multiply them together, and then use the softmax normalization operation to obtain the probability distribution of the j-th layer relationship classification. The calculation process is as follows:

[0040] α (j) =softmax(R(j) s j )

[0041] in, Contains the classification probability values ​​of all relations in the jth layer, so relation r is in relation r in the jth layer j Probability You can query to obtain:

[0042]

[0043] In step 5, after multi-level classification, each level will generate a probability distribution of classification, α (i) Represents the classification probability distribution of all relations at level i. The prediction probability vector of level i is defined as N i Represents the number of relationship nodes at the i-th level in the relationship hierarchy. (i) The hierarchical structure of the relationship can construct the expected probability distribution for the previous level The specific construction method is: if there is a relationship node c in the relationship hierarchy, the node has n c child nodes, then the classification probability value of relation c is n c In this way, the probability distribution of the next level can be used to construct the expected probability distribution of the previous level. When expected With the real α (i-1) The smaller the distribution difference between them, the more reasonable the classification results of adjacent levels. Through this distribution constraint, independent local classifications are connected, making the multi-level classification process a whole. In order to meet the local probability constraint, the following objective function is defined for the model:

[0044]

[0045] Where B is a set of sentence bags in a training batch, KL(·) represents relative entropy, which is an asymmetric measure of the difference between two probability distributions, and α (l) is the true probability distribution, is the fitted probability distribution.

[0046] Given a training batch of bag sets The corresponding true relation label is {r 1 ,r 2 ,…}. In step 6, since the model adopts a multi-level classification framework, the traditional cross entropy loss function is improved to form a hierarchical loss function, which is defined as follows:

[0047]

[0048] Combined with the above local probability constraint loss function, the loss function of the model is finally defined as follows:

[0049]

[0050] Among them, λ∈[0,1] is the proportional coefficient of the local probability constraint term, ||θ|| 2 represents the L2 regularization term.

[0051] Beneficial effects: Compared with the prior art, the remote supervised relationship extraction method based on multi-level classification hierarchical attention provided by the present invention has the following advantages:

[0052] (1) GATs are used to model the hierarchical structure of relations, deeply explore the implicit semantic connections between relation categories, and generate relation embeddings that perceive the global hierarchical structure.

[0053] (2) Based on the hierarchical attention mechanism, a multi-level classification structure is introduced to achieve a more fine-grained classification process, guiding the migration of knowledge from head relations to tail relations, thereby achieving data denoising and effectively improving the extraction performance of long-tail relations.

[0054] (3) Based on the fact that the classification results of adjacent levels are highly dependent, a novel local probability constraint is introduced as the objective function.

[0055] (4) The present invention has excellent effectiveness in noise reduction and long-tail extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the hierarchical attention mechanism structure of an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of a multi-level classification framework structure of an embodiment of the present invention;

[0059] Figure 4 The following is a brief description of the local probability constraints of an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0061] like Figure 1As shown, the remote supervision relation extraction method based on multi-level classification hierarchical attention includes the following steps:

[0062] Step 1: Obtain a sentence bag, use a sentence encoder to extract the features of a single sentence, and obtain a sentence feature vector. The NYT dataset is the most widely used dataset in the DSRE field. Each training sample contains all the information required for relation extraction. A complete example contains four types of information: sentence text, belonging relation category, head entity, and tail entity. Both the head and tail entities consist of three parts of information: id, name, and location.

[0063] Select example sentence bags [{Ernst Haefliger, a Swiss tenor who... roles, died on Saturday in Davos, Switzerland, where he maintained a second home.}, {Ernst Haefliger was born in Davos on July 6, 1919, and studied at the Wettinger Seminary...}]. Use PCNN as the sentence encoder, use the word2vec model to complete the conversion of words to vectors. The dimension of the corresponding word embedding is 50. The convolution window size of the sentence encoder PCNN is 3, the position feature dimension representing the relative distance is 5, and the PCNN hidden layer size is set to 230.

[0064] The PCNN input layer realizes the conversion of text to real-valued vectors, and the convolution layer processes the corresponding one-dimensional convolution operation on the text. The window size of the convolution kernel is 3, and all 3-gram features in the sentence are extracted. Given the input word vector sequence as X = {x 1 , x 2 , …, x l}, and m different convolution kernels {w 1 , w 2 , …, w m}, where is the vector representation of the i-th word, the dimension of the convolution kernel is related to the size of the sliding window w, is the i-th convolution kernel, and the symbol x i: j (i < j) represents the concatenated vector from the i-th word vector to the j-th word vector. The single operation of one-dimensional convolution can be expressed by the following formula

[0065] c ij = w i x j-w+1:j 1 ≤ i ≤ m, 1 ≤ j ≤ l + w - 1

[0066] Word vectors x outside the index rangei (i<1 or i>l) are defined as all zero vectors, c ij It represents the result of the jth convolution operation of the i-th convolution kernel. The output of the final convolution layer can be expressed as a matrix

[0067] In the pooling layer, PCNN uses the positions of the two entities in the sentence to divide the sentence into three parts, and then performs maximum pooling on each part. i The entity is divided into {q i1 ,q i2 ,q i3}, then the process of segmented maximum pooling can be expressed as p ij =max(q ij )1≤j≤3,p i ={p i1 ,p i2 ,p i3}, pi is the pooled output of the i-th convolution kernel, which is a 3D vector. The output of the entire pooling layer is p 1:m .

[0068] Step 2: Implement a hierarchical attention mechanism, whose structure is as follows Figure 2 As shown in the figure, the hierarchical structure of relations is used to obtain the package representation of sentence packages at different levels. Given a sentence package B = {s 1 ,s 2 ,…,s n},s i Represents the feature vector of the i-th sentence, s i The attention weight value is Finally, the obtained attention weight distribution is weighted and summed with the sentence features in the bag to obtain the bag representation of the bag. The above process of obtaining bag representation using sentence-level attention mechanism is simplified to

[0069] b h,t =ATT(q r ,{s 1 ,s 2 ,…,s n}).

[0070] The relationship label corresponding to the sentence bag is r. According to the hierarchy of relationships, r is at the bottom of the hierarchy. From this point, we trace back to its ancestor relationship node until the root node. The relationship nodes on the path form a relationship chain (r 0 ,r 1 ,r 2 ,…,r k ), where r = r k , r 0is a virtual node, which is the root node of the entire hierarchy. Then, multiple selective attentions are performed using the relationship nodes on the relationship chain to extract the bag representation of the sentence bag at each relationship level, where the bag representation of the i-th layer is obtained by formula Get, and then get the bag representation set of the sentence bag at all levels in the same way

[0071] Step 3: The hierarchical encoder takes the hierarchical structure of the relation as input, uses graph attention networks (GATs) to model the hierarchical structure of the relation, and outputs the relation embedding that perceives the global structure. The hierarchical encoder is formed by stacking multiple layers of GATs, each of which takes the embedding of all relation nodes in the relation hierarchy as input. Assume that there are n z nodes, then the input set of relation embedding can be expressed as {h 1 ,h 2 ,…,hn z}, and satisfy The input of the first layer of GATs is randomly initialized, and the input of the remaining GATs is the output of the previous layer. The core principle of GATs is to calculate the attention weights between nodes. The weight calculation between node i and node j is as follows:

[0072] α i,j =softmax j (LeakyRelu(W att [Wh i Wh j ]+b att ))

[0073] After calculating the attention weight matrix, use The neighbor aggregation operation obtains the updated relation embedding of node i. Get all updated relation embeddings Later, in order to make the above process stable and explore more subspaces, a multi-head attention mechanism was adopted, and the new relationship embedding update process was used After obtaining the embeddings of all relationship nodes, the relationship embeddings of the same level are stacked to form k-level embedding matrices {R (1) ,R (2) ,…,R (k)}, input into the multi-level classification hierarchical attention for matching classification.

[0074] Step 4: Take the package representation and relation embedding matrix of each level as input, input them into the multi-level classification framework, execute the local classification strategy multiple times inside it, calculate the matching score, and then input softmax to obtain the relation classification probability distribution of the current layer. The multi-level classification framework is connected after the hierarchical attention, and its structure is as follows Figure 3As shown in the figure, the package representations at each level are used as input to perform multiple local classifications. The multi-level classification framework consists of multiple local classifiers, which internally implement the local classification strategy, use the package representation and the relationship embedding matrix to calculate the matching score, and then input the softmax to obtain the relationship classification probability distribution of the current layer. Assuming that local classification is performed at level j, is the embedding matrix formed by stacking the embedding vectors of all relationship nodes at level j, N j represents the number of j-level relationship nodes, d g is the dimension of the relation embedding vector. First, the bag is represented as Input into the perceptron to complete the dimensional transformation Get the feature vector s required for classification j .

[0075] The j-th layer relation embedding matrix and classification feature vector s j Multiply them together, and then use the softmax normalization operation to obtain the probability distribution of the j-th layer relationship classification. The calculation process is as follows:

[0076] α (j) =softmax(R (j) s j )

[0077] Relation r is in level j j Probability You can query to obtain

[0078] Step 5: The classification results of adjacent levels of the hierarchy of relations are highly dependent. Based on this phenomenon, a local probability constraint is designed. A simple explanation of the local probability constraint can be seen in Figure 4 According to the classification probability distribution of each level of relationship and the hierarchical structure of the relationship, the expected probability distribution is constructed for the previous level. The smaller the distribution difference between the expected probability distribution and the actual probability distribution, the more reasonable the classification results of the adjacent levels are, thus achieving the unification of multi-level classification results. After multi-level classification, each level will generate a classification probability distribution, α (i) Represents the classification probability distribution of all relations at level i. The prediction probability vector of level i is defined as N i Represents the number of relationship nodes at the i-th level in the relationship hierarchy. (i) The hierarchical structure of the relationship can construct the expected probability distribution for the previous level In this way, the probability distribution of the next level can be used to construct the expected probability distribution of the previous level. When expected With the real α (i-1)The smaller the distribution difference between them, the more reasonable the classification results of adjacent levels. Through this distribution constraint, the independent local classifications are connected, making the multi-level classification process a whole. In order to meet the local probability constraint, the objective function is defined for the model

[0079] Step 6: Calculate the final loss by combining the hierarchical loss function and the local probability constraint loss function. Given a set of packages in a training batch, the corresponding true relationship label is {r 1 ,r 2 ,…}. Since the model adopts a multi-level classification framework, the traditional cross entropy loss function is improved to form a hierarchical loss function Combined with the loss function of the local probability constraint, the final loss function of the model is L = L hier +λL lpc +||θ 2 .

[0080] The present invention uses a graph attention network to model the hierarchical structure of relations, deeply explores the semantic dependency paths between relation labels, outputs global relation embeddings that perceive the hierarchical structure, and then proposes to combine the multi-level classification framework with hierarchical attention to improve the information utilization of sentence bag features and hierarchical structures, realize multi-granularity relation classification, and finally use local probability constraints to combine the multi-level classification process into a whole. The present invention is very excellent in relation extraction performance, outperforms all benchmark models in multiple evaluation indicators, has a strong ability to resist noise interference, and its performance in long-tail extraction is comprehensively better than the baseline model, which is an effective method to solve the long-tail problem of data in DSRE.

[0081] Obviously, those skilled in the art should understand that the various steps of the remote supervision relationship extraction method based on multi-level classification hierarchical attention of the above-mentioned embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A remote supervised relation extraction method based on multi-level classification hierarchical attention, characterized in that: The steps include: Step 1: Get the sentence bag, use the sentence encoder to extract the features of a single sentence, and get the sentence feature vector; Step 2: Perform a hierarchical attention mechanism to obtain bag representations of sentence bags at different levels using the hierarchical structure of relations; Step 3: The hierarchical encoder takes the hierarchy of relations as input, adopts graph attention networks (GATs) to model the hierarchy of relations, and outputs relation embeddings that perceive the global structure; Step 4: Take the package representations and relation embedding matrices at each level as input and input them into the multi-level classification framework. Perform the local classification strategy multiple times inside it, calculate the matching score, and then input the softmax to obtain the relation classification probability distribution of the current layer. Step 5: According to the classification probability distribution of each level of relationship and the hierarchical structure of the relationship, construct the expected probability distribution for the previous level. The smaller the distribution difference between the expected probability distribution and the actual probability distribution, the more reasonable the classification results of the adjacent levels, thereby achieving the unification of multi-level classification results; Step 6: Calculate the final loss by combining the hierarchical loss function and the loss function of the local probability constraint.

2. The remote supervised relationship extraction method based on multi-level classification hierarchical attention according to claim 1 is characterized in that: In step 1, the feature extraction of the sentence can be completed by a sentence encoder based on deep learning, such as CNN, PCNN, etc.

3. The remote supervised relationship extraction method based on multi-level classification hierarchical attention according to claim 1 is characterized in that: In step 2, hierarchical attention is based on sentence-level selective attention, and the hierarchical structure of relations is used to improve the extraction effect of long-tail relations; sentence-level selective attention is used once at each level corresponding to the true relation r, and the bag representation obtained at each level is concatenated and input into the softmax classifier for classification; a normalized weight distribution is generated for the sentence bag through the attention query mechanism, and then the bag representation of the entire bag is obtained by weighted summation, and finally the bag representation is used to predict a relationship label for the corresponding entity pair; Given a sentence bag B = {s1, s2, ..., s n },s i represents the feature vector of the i-th sentence in the bag, s i The attention weight value is: Among them, e i It represents the relevance value of the ith sentence to the true relation r, and its value is obtained by performing a dot product operation with the attention query vector: Among them, q r The attention query vector representing the corresponding relationship r; Finally, the obtained attention weight distribution is weighted and summed with the sentence features in the bag to obtain the bag representation of the bag: For ease of representation, the process of obtaining bag representation using the sentence-level attention mechanism is simplified to the following formula: b h,t =AT(q r ,{s1,s2,…,s n }) The relationship label corresponding to the sentence bag is r. According to the hierarchy of relationships, r is at the bottom of the hierarchy. From this point, we trace back to its ancestor relationship node until the root node. The relationship nodes on the path form a relationship chain (r 0 ,r 1 ,r 2 ,…,r k ), where r = r k , r 0 is a virtual node, which is the root node of the entire hierarchy. Then, multiple selective attentions are performed using the relationship nodes on the relationship chain to extract the bag representation of the sentence bag at each relationship level, where the bag representation of the i-th layer is calculated as follows: in, Represents the attention query vector of relation r at layer i; in the same way, obtain the bag representation set of sentence bags at all levels, that is, 4. The remote supervised relationship extraction method based on multi-level classification hierarchical attention according to claim 1 is characterized in that: In step 3, the hierarchical encoder regards the hierarchical structure as an undirected graph, uses GATs to model the hierarchical structure of the relationship, and outputs the relationship embedding that perceives the global structure; The hierarchical encoder is formed by stacking multiple layers of GATs. Each layer of GATs takes the embeddings of all relation nodes in the relation hierarchy as input. Assuming there are n z nodes, then the input set of relation embedding can be expressed as And satisfy The input of the first layer of GATs is randomly initialized, and the input of the remaining GATs is the output of the previous layer. The core principle of GATs is to calculate the attention weights between nodes. The weight calculation between node i and node j is as follows: α i,j =softmax j (LeakyRelu(W att [Wh i ;Wh j ]+b att )) Among them, LeakyRelu(·) is a nonlinear activation function, b att is a learnable weight parameter, is the attention vector, and a normalized weight value is obtained using softmax(); After calculating the attention weight matrix, the following neighbor aggregation operation is used to obtain the updated relational embedding of node i: Among them, N i represents the set of nodes that have an adjacent relationship with node i, is the learnable weight parameter matrix, is the bias term, σ(·) represents the nonlinear activation function; Get all updated relationship embeddings In order to make the above process stable and explore more subspaces, a multi-head attention mechanism is adopted. The new relationship embedding update process can be expressed as follows: Among them, H represents the number of heads of multi-head attention. After obtaining the embeddings of all relationship nodes, the relationship embeddings of the same level are stacked to form k-level embedding matrices {R (1) ,R (2) ,…,R (k) }, input into the multi-level classification hierarchical attention for matching classification.

5. The remote supervised relationship extraction method based on multi-level classification hierarchical attention according to claim 1 is characterized in that: In step 4, the multi-level classification framework is connected after the hierarchical attention, and the bag representations at each level are used as input to perform multiple local classifications; The multi-level classification framework consists of multiple localclassifiers, which internally implement local classification strategies, use bag representation and relation embedding matrix to calculate matching scores, and then input softmax to obtain the relation classification probability distribution of the current layer; assuming that local classification is performed at level j, is the embedding matrix formed by stacking the embedding vectors of all relationship nodes at level j, N j represents the number of j-level relationship nodes, d g is the dimension of the relation embedding vector; first, the bag is represented Input into the perceptron to complete the dimensional transformation and obtain the feature vector s required for classification j , the conversion process is as follows: in, is the trainable parameter matrix, d c represents the dimension of the package representation, b l Represents the bias term, ReLU is the rectified linear unit; The j-th layer relation embedding matrix and classification feature vector s j Multiply them together, and then use the softmax normalization operation to obtain the probability distribution of the j-th layer relationship classification. The calculation process is as follows: a (j) =softmax(R (j) s j ) in, Contains the classification probability values ​​of all relations in the jth layer, so relation r is in relation r in the jth layer j Probability You can query to obtain:

6. The remote supervised relationship extraction method based on multi-level classification hierarchical attention according to claim 1 is characterized in that: In step 5, after multi-level classification, each level will generate a probability distribution of classification, α (i) Represents the classification probability distribution of all relations at level i; the prediction probability vector of level i is defined as N i Represents the number of relationship nodes at the i-th level in the relationship hierarchy; according to α (i) The hierarchical structure of the relationship can construct the expected probability distribution for the previous level The specific construction method is: if there is a relationship node c in the relationship hierarchy, the node has n c child nodes, then the classification probability value of relation c is n c The sum of the probability values ​​of the sub-relationships; in this way, the probability distribution of the next level can be used to construct the expected probability distribution of the previous level When expected With the real α (i-1) The smaller the distribution difference between them, the more reasonable the classification results of adjacent levels are. Through this distribution constraint, independent local classifications are connected, making the multi-level classification process a whole. In order to meet the local probability constraint, the following objective function is defined for the model: Where B is a set of sentence bags in a training batch, KL(·) represents relative entropy, which is an asymmetric measure of the difference between two probability distributions, and α (l) is the true probability distribution, is the fitted probability distribution.

7. The remote supervised relationship extraction method based on multi-level classification hierarchical attention according to claim 1 is characterized in that: In step 6, the traditional cross entropy loss function is improved to form a hierarchical loss function, which is defined as follows: Combined with the above local probability constraint loss function, the loss function of the model is finally defined as follows: L=L hier +λL lpc +||θ|| 2 Among them, λ∈[0,1] is the proportional coefficient of the local probability constraint term, ||θ|| 2 represents the L2 regularization term.

Images