A document-level relation extraction method based on multi-graph neural network inference

Abstract

The application relates to a document-level relation extraction method based on multi-graph neural network reasoning, which comprises the following steps: step 1: obtaining a document data set and a given control entity pair; step 2: constructing a pre-document-level relation extraction model based on multi-graph neural network reasoning, inputting the data of step 1 into the pre-document-level relation extraction model for training, and obtaining a trained post-document-level relation extraction model; and step 3: inputting the data of step 1 into the trained post-document-level relation extraction model in step 2 to obtain corresponding entity pair relation prediction probability. The application constructs a multi-graph neural network reasoning component, the component enables the model of the application to reason document graph features from multiple angles, improves the relation prediction performance, and constructs a global feature for a document sequence, so that the model retains the graph features and adds sequence features to assist relation prediction.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular relates to a document-level relation extraction method based on multi-graph neural network reasoning. Background Technology

[0002] Currently, there are two main directions for document-level relation extraction methods based on deep learning: (1) methods based on document sequence encoding; and (2) methods based on document graph reasoning. Most methods are based on document graph models, mainly because document graphs fully consider the mutual information between entities and use graph structures to connect entities scattered in different sentences, thus avoiding the long dependency problem that exists when directly using document sequence encoding.

[0003] Although document graphs have shown excellent performance in many methods, they still have the following shortcomings: (1) Existing methods usually use only a single graph neural network when reasoning about document graphs. However, the node relationships in a document graph are very complex. In particular, many methods use multi-path relationships during the construction of a document graph, which makes it difficult for a single-structure graph neural network to fully reason about document graph information, thus affecting the final performance. (2) Many document graph-based methods do not consider using document sequence features, resulting in features that only reflect graph information and ignore the original sequence information of the document, making it difficult to improve performance. (3) Due to insufficient labeled samples during training, most methods usually use custom negative samples to assist training. However, using these negative samples also introduces a certain degree of noise.

[0004] For example, a document-level relation extraction method based on associated sentence selection and relation graph reasoning (Publication No.: CN113468325A, Publication Date: 2021-10-01) includes the following steps: obtaining a first set of sentences containing head entities and a second set of sentences containing tail entities in the document; selecting several sentences in the first and second sets of sentences that are close in position within the document as associated sentences; text encoding each character in the associated sentences; fusing the entity encodings in all associated sentences as input to the entity nodes of the relation graph model; and obtaining the shortest semantic dependency characters related to the entities as input to the dependency nodes of the relation graph model.

[0005] This extraction method can select and retain only sentences that are helpful for determining entity relationships by using related sentences. Then, it uses graph convolutional neural network-based relational graph reasoning to improve accuracy. However, this extraction method currently only uses a single graph neural network, which makes it difficult to fully reason about the document graph information, thus affecting the final performance. At the same time, it does not consider using document sequence features, so the obtained features can only reflect graph information, thus ignoring the original sequence information of the document, making it difficult to improve performance. Summary of the Invention

[0006] To address the technical deficiencies in the background technology, this invention proposes a document-level relation extraction method based on multi-graph neural network reasoning, which solves the aforementioned technical problems and meets practical needs. The specific technical solution is as follows:

[0007] This invention relates to a document-level relation extraction method based on multi-graph neural network reasoning, comprising the following steps:

[0008] Step 1: Obtain the document dataset and the given control entity pairs. The document dataset includes the training document dataset and the test document dataset. The control entity pairs include a first control entity pair and a second control entity pair.

[0009] Step 2: Construct a pre-document level relation extraction model based on multi-graph neural network reasoning, and divide the training document dataset from Step 1 into several groups of training document sequences and the first control entity pairs into the pre-document level relation extraction model for training. Output the relation prediction probability of the first document entity pairs in the training document sequences, and obtain the trained post-document level relation extraction model.

[0010] Step 3: Input the test document dataset and the given second reference entity pair into the post-document level relation extraction model trained in Step 2 to obtain the second document entity pair relation prediction probability corresponding to the test document dataset.

[0011] As a further technical solution of the present invention, the training document dataset and the test document dataset in step 1 above are divided into different batches. Each batch has a set of document sequences, namely a training document sequence and a test document sequence. The training document sequence has a first document entity pair, and the test document sequence has a second document entity pair.

[0012] As a further technical solution of the present invention, step 2 above includes the following steps:

[0013] Step 2.1: Input, input the training document sequence from Step 1 and the first control entity pair;

[0014] Step 2.2: Feature extraction, calculate and reason from the training document sequence in Step 2.1 to obtain graph convolution features, graph attention features and global attention features;

[0015] Step 2.3: Based on the position information of the first reference entity pair in Step 2.1, find the corresponding vector representation in the graph convolution feature, graph attention feature and global attention feature in Step 2.2, concatenate them to obtain the final relation feature, and calculate the relation prediction value of the first document entity pair in the training document sequence based on this.

[0016] Step 2.4: Based on step 2.3, a balanced cross-entropy loss function is introduced to improve the noise problem, and the balanced cross-entropy loss function is used to train the pre-document-level relation extraction model.

[0017] As a further technical solution of the present invention, step 2.2 above includes,

[0018] Step 2.2.1: Encoding. The pre-document level relation extraction model encodes the input training document sequence by inputting the documents in the training document sequence from Step 2.1 into the encoder to obtain the sequence encoding.

[0019] Step 2.2.2: Construct a document graph. For any given document in the training document sequence of Step 2.1, construct a document graph. The document graph includes a set of nodes and a set of path edges. The set of nodes includes a set of sentence nodes and a set of entity nodes in the document. The set of path edges includes a set of intra-sentence paths, a set of inter-sentence paths, and a set of logical paths. Construct the adjacency matrix corresponding to the document graph.

[0020] Step 2.2.3: Graph Neural Network Inference. The sequence encoding from Step 2.1 and the node set from Step 2.2.2 are used to generate graph node encoding, and the graph node encoding matrix is ​​obtained using the logsumexp function. Then, the graph node encoding matrix and the adjacency matrix are input into the multi-graph neural network component for inference. The multi-graph neural network component includes a graph convolutional neural network (GCN) and a graph attention network (GAT) to obtain graph convolutional features and graph attention features, respectively.

[0021] Step 2.2.4: Generate global attention features, calculate the multi-head self-attention features obtained by sequence encoding in step 2.2.1, and concatenate the multi-head self-attention features to obtain context features. Then, use the graph node encoding matrix from step 2.2.3 and the context features to calculate the global attention features of the document sequence with graph node encoding information.

[0022] As a further technical solution of the present invention, the intra-sentence path, inter-sentence path and logical path will be retained if different types of paths appear between two entities. For multiple paths of the same type, only the first path is retained. The intra-sentence path and inter-sentence path are actual paths, and the logical path is a virtual path.

[0023] As a further technical solution of the present invention, the graph convolutional neural network (GCN) constructs a network with several layers, performs layered calculations on the graph node encoding matrix and adjacency matrix to obtain the graph convolutional encoding matrix of the document graph, wherein the graph convolutional encoding matrix is ​​a graph convolutional feature.

[0024] As a further technical solution of the present invention, the graph attention network (GAT) first calculates the attention value between two nodes in the node set, then aggregates nodes through the attention value and updates the node encoding of the nodes, calculates the multi-head attention encoding of each node in the node set, and concatenates them into a multi-head attention matrix, which is a graph attention feature.

[0025] As a further technical solution of the present invention, the calculation direction of the attention value is unidirectional.

[0026] The beneficial effects of this invention are as follows:

[0027] Unlike most previous methods that only used a single graph neural network structure, this invention constructs a multi-graph neural network inference component, which enables the model of this invention to infer document graph features from multiple perspectives, thereby improving the performance of relation prediction.

[0028] In addition to using document graph features, this invention also constructs a global feature for the document sequence, which integrates the node information of the document graph into the document sequence. This allows the model to retain graph features while adding sequence features to assist in relation prediction.

[0029] To address the noise problem introduced by using negative samples during training, this invention employs a positive-negative sample balancing loss function to reduce the noise caused by negative samples during training.

[0030] Experiments were conducted on multiple public datasets, and the results show that the relation extraction performance of this invention is superior to the current mainstream models on these two benchmark datasets. Attached Figure Description

[0031] Figure 1 This is a diagram of the overall architecture of the present invention.

[0032] Figure 2 This is a general framework diagram of the present invention.

[0033] Figure 3 This is a diagram illustrating the construction of the document diagram of this invention.

[0034] Figure 4 This is a document diagram of the present invention.

[0035] Figure 5 This is a diagram of the global feature construction component based on document sequence according to the present invention.

[0036] Figure 6 This is the pseudocode used for training in this invention.

[0037] Figure 7This is a performance comparison chart of the present invention with 12 existing mainstream methods based on BiLSTM encoders.

[0038] Figure 8 This is a performance comparison chart of the present invention and 12 existing mainstream methods based on BERT encoders.

[0039] Figure 9 This figure shows a performance comparison between the present invention and mainstream document graph methods at different entity distances.

[0040] Figure 10 This is a performance comparison chart between the present invention and mainstream document graph methods.

[0041] Figure 11 This is a performance comparison chart of the present invention and its variants.

[0042] Figure 12 This is a performance comparison chart of the present invention and its two graphical variants.

[0043] Figure 13 This is a graph showing the performance comparison of different BERT encoders on the DocRED dataset used in the experiments of this invention.

[0044] Figure 14 This is a graph showing the impact of the experimental hyperparameters α and β on the relation prediction performance of this invention.

[0045] Figure 15 This is a feature representation diagram in two-dimensional space obtained by the t-SNE dimensionality reduction technique of this invention. Detailed Implementation

[0046] The following is in conjunction with the appendix Figures 1-15 The embodiments of the present invention will be described in conjunction with related examples. The embodiments of the present invention are not limited to the following examples, and the relevant necessary components of the present invention should be regarded as well-known technology in the field of this art, which can be known and mastered by those skilled in the art.

[0047] With the rapid development and in-depth application of the Internet, text data is growing exponentially, and information extraction technology is becoming increasingly important in text information processing. Among them, relation extraction is an important task of information extraction, which aims to automatically identify the relationships between given entities in text and store them in the form of triples (subject, relation, object) to provide entity relationships for downstream tasks such as knowledge graphs and automatic question answering.

[0048] Currently, document-level relation extraction methods based on deep learning mainly fall into two categories: 1. methods based on document sequence encoding; 2. methods based on document graph reasoning. Most of these methods are implemented based on document graph models. This is mainly because document graphs fully consider the mutual information between entities and use graph structures to connect entities scattered in different sentences, thus avoiding the long dependency problem that exists when directly using document sequence encoding to a certain extent.

[0049] Although document graphs have demonstrated excellent performance across many methods, they still have the following shortcomings:

[0050] 1) Existing methods typically use only a single graph neural network when reasoning about document graphs. However, the node relationships in a document graph are very complex. In particular, many methods use multi-path relationships during the construction of the document graph. This makes it difficult for graph neural networks with a single structure to fully reason about document graph information, thus affecting the final performance.

[0051] 2) Many document graph-based methods do not consider using document sequence features, resulting in features that only reflect graph information and ignore the original sequence information of the document, making it difficult to improve performance.

[0052] 3) During training, due to insufficient labeled samples, most methods usually use custom negative samples to assist training. However, using these negative samples can also introduce a certain degree of noise.

[0053] To address the above problems, this invention provides a document-level relation extraction method, MGR (Multi-Graph Neural Network Reasoning), based on multi-graph neural network reasoning, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0054] Step 1: Obtain the document dataset and the given control entity pairs. The document dataset includes the training document dataset and the test document dataset. The control entity pairs include the first control entity pair and the second control entity pair.

[0055] Step 2: Construct a pre-document level relation extraction model based on multi-graph neural network reasoning, and divide the training document dataset from Step 1 into several groups of training document sequences and the first control entity pairs into the input pre-document level relation extraction model for training. Output the relation prediction probability of the first document entity pairs in the training document sequence, and obtain the trained post-document level relation extraction model.

[0056] Step 3: Input the test document dataset and the given second control entity pair into the post-document level relation extraction model trained in Step 2 to obtain the second document entity pair relation prediction probability corresponding to the test document dataset.

[0057] To facilitate the distinction between given entity pairs and entity pairs in document sequences, given entity pairs are referred to as "comparison entity pairs," and entity pairs in document sequences are referred to as "document entity pairs." Additionally, to differentiate between documents and models before and after training, terms such as "before," "after," "first," and "second" have been added above.

[0058] The above is a general outline of the steps of this invention. The following will provide a more detailed explanation of the invention using data and formulas.

[0059] The data comes from a document dataset and given entity pairs. The document dataset is divided into several sets of document sequences, each set of document sequences includes several documents, and each document contains document entity pairs. This also means that the training document dataset contains training document sequences, the training document sequences contain training documents, and the training documents contain training entity pairs. The same applies to the test document dataset.

[0060] Suppose that the given document contains a total of t sentences and n entities, then the sets formed by them are represented as D = {s1, s2, ..., sn}. t}and (Document entity pair), where the i-th sentence is defined as x j The words that represent the sentence, and This indicates the length of the sentence. At this point, given the entity set... The two entities e s and e o (Compare the entity pairs) and the predefined set of relations R = {r1, ..., r} s}∪{NA}, where NA represents an entity relation that is not in set R.

[0061] Therefore, the task of document-level relation extraction is to predict entity pairs (e) based on a given document D. s e o ) s≠o (Comparison of entity pairs) The probability value of each relation label in a given relation set R: p(r s，o |(e s e o ), D), r s，o ∈R, and select all relation labels r with p values ​​greater than 0.5. s，o As an entity pair (e s e o The relationship prediction of (reference entity pairs) is as follows, and the more detailed steps of the present invention are as follows.

[0062] For ease of description, the following terms "document sequence" and "training document sequence" refer to "training document sequence"; "document" and "training document" refer to "training document"; "reference entity pair" and "first reference entity pair" refer to "first reference entity pair"; "document entity pair" and "first document entity pair" refer to "first document entity pair".

[0063] Step 1: Obtain the document dataset and the given control entity pairs. The document dataset includes the training document dataset and the test document dataset. The control entity pairs include the first control entity pair and the second control entity pair.

[0064] Step 2: Construct a pre-document level relation extraction model based on multi-graph neural network reasoning, and divide the training document dataset from Step 1 into several groups of training document sequences and the first control entity pairs into the pre-document level relation extraction model for training. Output the relation prediction probability of the first document entity pairs in the training document sequences, and obtain the trained post-document level relation extraction model.

[0065] Step 2.1: Input, input the training document sequence from Step 1 and the first control entity pair.

[0066] Step 2.2: Feature extraction. The training document sequence from Step 2.1 is used to calculate and infer graph convolution features, graph attention features, and global attention features.

[0067] Step 2.2.1: Encoding. Input the documents in the training document sequence from Step 2.1 into the encoder to obtain the sequence encoding.

[0068] During the training phase, the documents in the entire training dataset are divided into different batches, with each batch using a set of document sequences. Training will be conducted, and the next step will be through a single document. Let's demonstrate the training process. As defined in the previous document-level relation extraction definition, each document can be represented as D = {s1, s2, ..., s...}. t Each sentence can be represented as} So, regarding the document During the coding phase, to facilitate processing, the document... The sentences in the sequence are combined into a long sequence for encoding, that is, let x here j This represents the words in the long sequence, while m represents the length of the combined sentences in the document, with a maximum value set to 512.

[0069] Subsequently, the documents in the growth sequence will be merged. Input into the encoder:

[0070] H = [h1, h2, ..., h m]=Encoder([x1,x2,...,x m (1)

[0071] Among them, the current document The sequence encoding representation of H, Document Chinese word x m d-dimensional eigenvectors This represents a sequence matrix.

[0072] Step 2.2.2: For any given document in the training document sequence of Step 2.1 Construct a heterogeneous graph G = (V, E), i.e., a document graph, and construct the adjacency matrix corresponding to the document graph.

[0073] like Figure 4 As shown, it contains a set of nodes V and a set of path edges E, where V = V e ∪V s , and V e ={e1, e2, ..., e n} and V s ={s1, s2, ..., s t} represent documents respectively The set of nodes consisting of sentences and document entities; E = E1∪E2∪E3 represents the set of edges consisting of the set of intra-sentence paths E1, the set of inter-sentence paths E2, and the set of logical paths E3, respectively. The following will use... For example, construct the adjacency matrix of the corresponding document graph G according to the following formula.

[0074]

[0075] Among them, v i v j For any two nodes in the node set V, the merge function merges them according to given rules. i v j Select the appropriate path and make the corresponding update in U.

[0076] like Figure 3 As shown, the path construction rules are as follows:

[0077] 1) Intra-sentence path: If two document entities are located in the same sentence within a document, then an intra-sentence path is constructed for these two document entity nodes and the sentence node. Assuming... It is document entity e s and e o In the document graph G, nodes represent the nodes. Therefore, the node pair (v) input in formula (2) is... i vj That is and Since these two document entities are located in the same sentence, they can be directly associated. After the merge operation, an intra-sentence inference path can be formed in the adjacency matrix. Add it to E1.

[0078] 2) Inter-sentence path: If two document entities are scattered in different sentences and pass through intermediate document entities or sentences (there can be multiple intermediate document entities, usually document entity e) s and e o If the document entities mentioned in all sentences can form a one-way reasoning path, then an inter-sentence path is constructed between the document entity nodes and sentence nodes.

[0079] Specifically, in the process of constructing the path, the sentence nodes containing the two document entities are first identified. Then, starting from one of the nodes (using a graph search algorithm), if the other node can be reached successfully, the path is valid. The intermediate nodes are saved, which is the inter-sentence path.

[0080] Assumption and It is a document graph G and a set of nodes V. e Document entity nodes in This represents an intermediate document entity node, while It is document entity e s and e o The sentence s s and s o The corresponding sentence node, then the node pair (v) input in formula (2) at this time i v j According to as well as and Based on the document's position, a merge operation is performed using a graph search algorithm. After the merge operation is complete, a unidirectional cross-sentence reasoning path can be constructed in the adjacency matrix. And add it to E2.

[0081] 3) Logical Path: If two document entities do not belong to either of the above two path relationships, then there is no actual path between them. A virtual logical path is constructed for these two document entity nodes. For example... It is document entity e s and e o The node representation in the document graph G. Then, the node pair (v) input in formula (2) at this time. i v j )Depend on and In this case, a logical path can be constructed for these two nodes using the merge operation. And add it to E3.

[0082] Since a document entity usually corresponds to multiple mentions, there may be situations where there are both "intra-sentence paths" and "inter-sentence paths" between two document entities. For different types of paths, they will be retained. However, if there are multiple paths of the same type, only the first path will be retained.

[0083] Meanwhile, the directions between nodes are random, and different types of paths are tended to be retained during selection to ensure the diversity of paths between nodes, which will benefit downstream inference tasks. Thus, the construction of the document graph G = (V, E) is completed, and the adjacency matrix U of the document graph G is obtained.

[0084] Step 2.2.3: The graph neural network component uses the node set V and adjacency matrix U of the document graph G, along with the document... The sequence encoding representation H is used as input to infer document graph features.

[0085] Before performing inference, a graph node encoding representation needs to be generated for each node in V, using the previously obtained document sequence representation H = [h1, h2, ..., h m To generate graph node codes, the following steps are taken, starting with the documentation. Extract a single node v from the given information. j Position vector ∈V in It is an indicator vector containing only 0s and 1s, whose dimension is the length m of the document sequence, used to represent v. j The position in the document sequence is then determined, and the position vectors of each node are merged into a position matrix:

[0086] P = concat([p1, ..., p2]) n (3)

[0087] in, n represents the number of nodes, m is the sequence length, and concat(·) represents the concatenation function. Since each graph node may correspond to multiple document entity references in the document sequence, the influence of different document entity references on node encoding needs to be considered during the construction of graph node encoding. To obtain a better node representation, the logsumexp function is used to calculate the node encoding:

[0088]

[0089] C = concat([c1, ..., c...)n (5)

[0090] in, Encoding representation for a single node. This indicates the number of document entities referred to by the current node. Let n be the node encoding matrix of the graph, where n is the number of nodes.

[0091] The multi-graph neural network component comprises two distinct graph neural network architectures: the Graph Convolutional Neural Network (GCN) and the Graph Attention Network (GAT). In the GCN, a network of depth k is constructed, where for a given node v... i The graph node representation of its (l+1)th layer can be calculated by the following formula:

[0092]

[0093] in, Encoding the graph nodes of layer l, b (l) For the bias term of the l-th layer, Let N be the learnable parameters of the l-th layer, and let σ(·) represent the nonlinear activation function ReLU. E This represents the type of edge, where N is the number of edges. E =3 corresponds to the three paths constructed earlier, each using a different weight matrix. Mapping, This represents a vector addition operation. After obtaining the graph node representation of the (l+1)th layer, With the 0th layer Add them together to get node v i The output vector representation of the (l+1)th layer

[0094] Next, the graph node vectors of each layer are aggregated to obtain v. i Graph convolution represents vector

[0095]

[0096]

[0097] Here, w0 represents the learnable parameters of layer 0. Based on this, the graph convolutional encoding matrix of the document graph G can be obtained, which is the concatenation of each vector in the node set V:

[0098]

[0099] 2) In GAT, the attention value between two nodes needs to be calculated first, and then the nodes are aggregated based on the attention value. For a given node v i and v jTheir attention values The calculation formula is as follows:

[0100]

[0101]

[0102] in, and They are nodes v i and v j The graph node encoding, w1, w2, and w3 are shared learnable parameters, e ij Indicates v i For v j Attention value, N i Representative node v i The number of all adjacent nodes in the document graph G, where "||" is the single vector concatenation operator, exp(·) is the exponential function, and LeakyReLU(·) is the non-linear activation function. It's worth noting that this attention calculation is unidirectional, i.e., v i For v j The attention value is not equal to v j For v i Attention value.

[0103] Next, the formula with multiple parameters is used to update node v. i Node encoding

[0104]

[0105] Where K represents the number of long positions, N i For node v i The number of all adjacent nodes in the graph, c j This indicates a node v i Adjacent node v j The graph represents the encoding. It is the learnable parameter matrix of the k-th head. The multi-head attention code of each node in the node set V is calculated and concatenated into a multi-head attention matrix.

[0106] A gat =concat([α1, ..., α) n (13)

[0107] Thus, the two document graph features G are calculated separately using the multi-graph neural network component. gcn and A gat These are graph convolution features and graph attention features.

[0108] Step 2.2.4: Construct a global attention feature.

[0109] In this invention, such as Figure 5 As shown, the document's sequence encoding H = [h1, h2, ..., h...] is used. m A global attention feature is constructed that, in addition to using sequence encoding, also incorporates node encoding of the document graph to enhance the feature representation, such as... Figure 2 As shown in (b) above. Specifically, h is represented by the i-th sequence. i For example, first calculate h i Self-attention score Then, the weights are obtained by normalization. Based on this, use weights Multiply by h i We can get h i Multi-head self-attention characteristics

[0110]

[0111]

[0112]

[0113] in, w4 and W5 represent the learnable parameter matrices, K represents the number of multi-heads, and d i h i Dimensions.

[0114] Subsequently, the multi-head self-attention features of each vector in the sequence code H are calculated and concatenated to obtain the context features of the entire sequence code H.

[0115]

[0116] Where W6 is the learnable parameter matrix.

[0117] Based on this, consider the graph node encoding matrix. and context features obtained from sequences The interrelationships. Similarly, first calculate the i-th context feature z. i The multi-head attention score ω for matrix C i Then normalize to obtain the weights. Next, the attention score will be calculated. Multiply by z i Get z i The multi-head attention feature q of the graph node encoding matrix C i :

[0118]

[0119]

[0120]

[0121] in, w7 and W8 represent learnable parameters, d′ i For z i The dimension. Then, a global attention feature of a document sequence with graph node encoding information can be obtained through formula (21).

[0122]

[0123] Where W9 is the learnable parameter matrix.

[0124] Step 2.3: Based on the position information of the first reference entity pair in Step 2.1, find the corresponding vector representation in the graph convolution feature, graph attention feature and global attention feature in Step 2.2, concatenate them to obtain the final relation feature, and calculate the relation prediction value of the first document entity pair in the training document sequence based on this.

[0125] Specifically, assuming the input comparison entity pair is (e s e o First, based on the positional information of the corresponding entity pair, in feature G... gcn A gat and Q seq Find the corresponding vector representation and concatenate them to obtain the final relation feature.

[0126]

[0127] in, From this, we can deduce... Based on this, the relationship prediction value of the control entity pair is calculated. The details are as follows: The features are processed by a fully connected layer to calculate the probability vector of each type of relationship, and then the sigmoid(·) function is used to map the probability to [0, 1].

[0128] For each probability vector, the maximum value is selected to represent the probability of this type of relationship, as shown in the following formula:

[0129]

[0130] Among them, W r b is a learnable parameter r This is a bias term.

[0131] Step 2.4: Based on step 2.3, a balanced cross-entropy loss function is introduced to improve the noise problem, and the balanced cross-entropy loss function is used to train the pre-document-level relation extraction model.

[0132] The following loss function is used to train and learn this invention:

[0133]

[0134]

[0135] In (24) and (25), p i and Representing the contrasting entity pair (e) s e o The true value (one-hot encoded) and predicted probability value of the relation, r s，o =1 indicates a positive sample label, while r s，o =0 indicates a negative sample label. Specifically, inspired by the balanced cross-entropy loss function, the binary cross-entropy loss function is improved, and a new loss function is constructed. That is, in Add a positive and negative sample balance factor β to make it more suitable for cases where positive and negative samples are imbalanced.

[0136] For document entity pairs that actually exist in the dataset, β will reduce the loss value, while for artificially constructed negative samples, it will amplify the loss value. This will guide the pre-document level relation extraction model based on multi-graph neural network inference to pay more attention to the actual document entity pairs.

[0137] The following describes the training of the model before training in this invention and the optimal algorithm therein. All "steps" mentioned below refer to the steps described below. Figure 6 The algorithm referred to in this text is not directly related to the "steps" mentioned above.

[0138] like Figure 6 As shown, the pseudocode for training according to this invention (Algorithm 1) and its time and space complexity analysis are given. Let The time and space complexity analysis of Algorithm 1 is as follows.

[0139] 1) Time complexity: The time cost of step 1 is... The time cost of step 2 is The time cost of steps 3-8 is The time cost of step 9 is The time cost of steps 10-23 is The time cost of step 24 is Therefore, the time complexity of this invention is

[0140] 2) Space complexity: The space overhead of step 1 is The space cost of steps 2 and 5-6 is O(|V|). 2 The space cost of steps 9 and 12 is O(d), where d is the feature dimension; the space cost of step 13 is O(|V|); the space cost of step 14 is O(d|V|); the space cost of steps 15 and 16 is also O(d|V|). Where n is the number of nodes in the largest document graph in the minimum batch; the space cost of steps 17 and 18 is... Where m is the sequence length; the space overhead of step 19 is equal to Typically, d 2 Far greater than n and |V| 2 Therefore, the space complexity of this invention is O(d). 2 ).

[0141] Step 3: Input the test document dataset and the given second control entity pair into the post-document level relation extraction model trained in Step 2 to obtain the second document entity pair relation prediction probability corresponding to the test document dataset.

[0142] The following are the experimental data of this invention, and the results show that the relation extraction performance of this invention on these two benchmark datasets is superior to the current mainstream models.

[0143] 1) Data Sources: The experiments primarily used two popular public datasets: The first is DocRED, a document-level relation extraction dataset based on large-scale annotations from Wikipedia. DocRED contains 3053 training documents, 1000 development documents, and 1000 test documents, with a total of 96 relation types and an average of 19.5 document entities per document. The second is CDR, a relation extraction dataset in the biomedical field, mainly used for extracting disease and drug relations from medical documents. CDR has 500 documents in each of its training, development, and test sets, with an average of 6.8 document entities per document.

[0144] 2) Comparison Method: To facilitate comparison, the experiment implemented this invention using BiLSTM and BERT-based encoders respectively. Simultaneously, the performance of 12 mainstream document-level relation extraction methods was evaluated (the hyperparameter settings of the existing methods remained consistent with the original text):

[0145] ①EOG: Proposes a graph neural network that uses edge-oriented rather than node-oriented reasoning for document-level entity relation extraction.

[0146] ②LSR: Constructs a document graph that does not rely on coreference and uses graph neural networks for multi-hop reasoning of entity relationships.

[0147] ③GLRE: Based on graph neural network inference, it constructs global entity features, local entity features, and text features to predict entity relationships.

[0148] ④GAIN: Two document graph structures were constructed at the granularity of entity reference and entity to predict entity relationships in a document.

[0149] ⑤HeterGSAN: Uses heuristic rules to transform input documents into heterogeneous graphs and uses a graph attention network to predict entity relationships.

[0150] ⑥SIRE: Two different granularity coding layers are used to process intra-sentence paths and inter-sentence paths in the document graph.

[0151] ⑦DRN: Constructs a document graph using intra-sentence paths, logical paths, and coreference paths, and develops separate reasoning tasks for these three paths to predict entity relationships.

[0152] ⑧ALTOP: Instead of using a document graph, it uses a local text pooling method to process entity representations and proposes an adaptive threshold loss function.

[0153] ⑨DocuNet: Models the task of extracting document-level relations as a semantic segmentation task and improves U-net for extracting document-level relations.

[0154] ⑩KD-DocRE: Proposes a multi-axis attention to capture logical dependencies between entities and proposes an adaptive Focal Loss to solve the problem of imbalanced relation labels.

[0155] 11 EIDER: For the first time, Evidence labels were added to the DocRE task, and new features were trained using Evidence labels to enhance the RE task.

[0156] 12. SAIS: Building upon training aided by Evidence labels, it further expands the data by dividing Evidence features into PooledEvidence and Fine-grained Evidence. Simultaneously, it saves computational costs by performing ensemble learning on subsets of triples.

[0157] Of the 12 methods compared above, the first 7 methods (EOG, LSR, GLRE, GAIN, HeterGSAN, SIRE, DRN) are all based on document graphs, which are currently the mainstream methods for document-level relation extraction. The middle 3 methods (ALTOP, DocuNet, KD-DocRE) are non-document graph methods. They are mainly based on the transformer architecture for sequence modeling and predict entity relationships by constructing attention matrices between entities. The last 2 methods (EIDER, SAIS) expand the data by introducing evidence labels to train new features, thereby further improving the performance of document-level relation extraction.

[0158] In the experimental work, the main focus is on the relation extraction task, not on evidence extraction. Therefore, in the comparison between EIDER and SAIS, the evidence features will be removed for comparison.

[0159] 3) Evaluation metrics and hyperparameter settings: All experiments were performed in the following environment: Python 3.7, PyTorch 1.6.0, GeForce RTX3080Ti GPU. In order to maintain the same evaluation settings as existing work, the experimental results will be given on both the development set and the test set.

[0160] In addition, two evaluation metrics consistent with existing work were adopted: F1 and IgnF1.

[0161]

[0162] Where Precision represents the accuracy of relation prediction, and Recall represents the number of times the true labeled relation is predicted. To avoid evaluation bias caused by the model over-memorizing document entity pairs from the training set before training, IgnF1 removes overlapping document entity pairs from the development / test set and the training set during calculation.

[0163]

[0164] Among them, E T E represents a document entity pair representing the development / test set. overlap This represents document entity pairs that overlap between the training set and the development / test set, therefore |E T |-|E overlap | indicates that overlapping document entity pairs are removed before calculating the F1 score. The higher the IgnF1 and F1 index, the more accurate the document entity relationship prediction.

[0165] The hyperparameter settings of this invention are as follows:

[0166] ① The model parameters were optimized using the minimum batch-based ADAMW optimizer, with a minimum batch size of 12 (BiLSTM) and 8 (BERT).

[0167] ②The document sequence encoding dimensions are 128 (BiLSTM) and 768 (BERT).

[0168] ③ The global learning rate is 1e-3.

[0169] ④ Set the BERT learning rate in {5e-5, 4e-5, ..., 1e-5}.

[0170] ⑤ The learning rate decay coefficient is selected from {1e-3, 1e-4, 1e-5}.

[0171] ⑥ The proportion of negative samples α takes values ​​in {2, 3, 4, 5, 6}, that is, training is performed using negative samples that are α times the number of positive samples.

[0172] ⑦ Set the maximum number of epochs during model training to 200.

[0173] 4) Experiment and results:

[0174] ① Comparative experiment

[0175] To minimize the impact of word embeddings, encoders, and other components on the overall experiment, this invention was implemented using both BiLSTM and BERT. On the DocRED dataset, the BERT-Base model was selected, and on the CDR dataset, SciBERT was chosen. For the BiLSTM-based method, PubMed word embeddings were used on CDR, and GloVe word embeddings were used on DocRED.

[0176] Figure 7 and Figure 8 Their performance is compared separately, from which we can see that:

[0177] A) When using the same encoder, the performance of this invention is superior to mainstream document-level relation extraction methods in most cases. For example... Figure 7 In the DocRED test set, the IgnF1 and F1 scores of this invention are 54.86% and 57.05%, respectively, both higher than other mainstream methods. On the CDR dataset, the IgnF1 and F1 scores are 73.4% and 73.7%, respectively, which also achieve the best performance.

[0178] This is mainly because: a) compared to using a single graph neural network, this invention uses two graph neural networks with different structures to reason about the document graph, which helps to improve the accuracy of graph feature reasoning; b) this invention also constructs global attention features based on document sequences and introduces graph node information, which also makes the feature expression richer; c) this invention makes the model pay more attention to real sample data by changing the weight of negative samples in the loss function during training.

[0179] B) When using a BiLSTM encoder, document graph-based methods outperform the 12 existing mainstream methods. For example, Figure 7 Document graph-based models such as DRN and GAIN achieved F1 scores of 56.33% and 55.08% respectively on the DocRED dataset, and 71.9% and 68.3% respectively on the CDR dataset. Furthermore, most document graph-based methods showed some improvement after replacing the BiLSTM network with BERT pre-trained encoders. Because document graphs can capture the relationships between different entities in a document and are less affected by the encoder, document graph models exhibit excellent stability and maintain a relative performance advantage across different encoder architectures.

[0180] C) Non-document graph methods such as ALTOP, DocuNet, and KD-DocRE exhibit excellent performance when using BERT encoding, outperforming most document graph methods. This is because they all utilize self-attention computation between entities, and the attention mechanism functionally replaces some of the functions of a document graph. When non-document graph methods compute attention scores between each entity representation, they are similar to document graphs constructing path associations for entities. For example... Figure 8 In the DocRED dataset, KD-DocRE achieves an F1 score of 62.08%, while DocuNet achieves 61.86%. However, when using BiLSTM encoding, these methods outperform document graph-based methods. For example... Figure 7 In the DocRED dataset, DocuNet's F1 score is only 52.77%, while KD-DocRE's F1 score is only 53.87%. This is because non-document graph methods mainly learn the relationships between entities through attention mechanisms, which are highly dependent on the BERT model. Therefore, the performance drops significantly after replacing it with a BiLSTM encoder.

[0181] D) Clearly, methods based on BERT encoders generally outperform those using BiLSTM encoders. This is because BERT encoders utilize a large amount of Wikipedia-based corpus during pre-training and can fine-tune word embedding weights in downstream tasks, thus possessing stronger expressive power compared to fixed-word embeddings like GloVe and PubMed, and extracting better entity representations from document encoding. Furthermore, when using the same BERT encoder, non-document graph methods outperform most document graph methods, indicating that pre-trained models offer significant performance gains for non-graph structures. Improving pre-trained models may be another approach to enhance document-level relation extraction performance.

[0182] E) On the DocRED test set, the IgnF1 score of this invention, when using the BERT encoder, is 59.60%, lower than the 60.18% of the SIRE method. This may be related to the BERT encoder overfitting the document entity pairs in the training set. Especially in graph neural network inference, a node representation is often aggregated from other nodes. Although document entity pairs that overlap with the training set are removed when calculating IgnF1, the remaining document entity pairs may still be affected by overlapping document entity pairs during training, thus affecting the final IgnF1 score.

[0183] To further verify the performance of this invention, all entities in the document were divided into intra-sentence entities and cross-sentence entities, and their F1 scores were calculated separately. Intra-sentence entities are represented by an entity distance d = 0 in the table, while cross-sentence entities are represented by an entity distance d >= 1. In the experiments, several mainstream document graph models, including LSR, GAIN, and SIRE, were mainly selected as the alignment methods of this invention. The comparison results are as follows: Figure 10 As shown.

[0184] As can be seen, this invention outperforms other methods, whether based on a BiLSTM encoder or a BERT encoder. Structurally, these mainstream document graph methods all use a single graph neural network, while this invention uses two different graph neural networks, indicating that using multiple graph neural networks can improve relation extraction performance. To make the results more intuitive, the experimental results are presented in a bar chart, such as... Figure 9 The results are shown. Similarly, it can be seen that the present invention (black in the figure) outperforms other mainstream document graph methods in the comparison, both with BiLSTM and BERT encoders.

[0185] ② Ablation test

[0186] The effectiveness of the main components of this invention will be verified through ablation experiments.

[0187] Since the DocRED dataset obtains test set scores through score submissions and does not provide a test set itself, this ablation experiment was primarily conducted on the DocRED Dev set and the CDR Test set. This experiment mainly constructed the following four variant methods:

[0188] A) MGR-M1: In this invention, only the GCN network is retained in the graph neural network inference part;

[0189] B) MGR-M2: In this invention, only the GAT network is retained in the graph neural network inference part;

[0190] C) MGR-M3: In this invention, the multi-graph neural network inference component is removed, while the context features are retained;

[0191] D)MGR-M4: In this invention, only graph features are used to predict document entity pair relationships, and global features of document sequences are not used.

[0192] Figure 11 A performance comparison of the present invention with four variant methods is presented.

[0193] from Figure 11 As can be seen, on the DocRED dataset, when based on the BiLSTM encoder, MGR-M1 and MGR-M2 achieve F1 scores 2.22% and 1.67% lower than the present invention, respectively, and IgnF1 scores 2.36% and 1.65% lower, respectively. This is because the MGR-M1 and MGR-M2 variants only use a single graph neural network structure to infer document graphs, making it difficult to infer document graph information from multiple perspectives, thus affecting the relation prediction results.

[0194] The performance of MGR-M3 degrades significantly after the graph neural network component is removed. This is because the graph neural network is the core of this invention. After removing the component, the structure is similar to the BiLSTM method, except that there is an additional context feature component. Therefore, the performance is also similar to the BiLSTM method.

[0195] Furthermore, MGR-M4 retains the multi-graph neural network portion, but due to the removal of global attention features, its IgnF1 and F1 scores are slightly lower than those of this invention. This indicates that using document global attention features enhances the final relation prediction.

[0196] Although using better encoding models and models learned from graph networks can achieve better relation extraction results, ablation experiments ( Figure 11As can be seen from MGR-M4, using only the encoder and graph neural network is insufficient to further improve performance. Without using sequence features, the F1 scores are only 55.95 and 61.14, respectively, which differ from the final results by 1.28 and 1.21. The performance variation of the BERT-based encoder is basically consistent with that of BiLSTM.

[0197] Since this invention uses two graph neural networks with different structures, the following two variations of this invention were also constructed in the experiment to verify the performance of the structures:

[0198] E)MGR-G1: In this invention, two GCN networks are used in the graph neural network inference part;

[0199] F)MGR-G2: In this invention, two GAT networks are used in the graph neural network inference part.

[0200] Figure 12 The performance comparison between the present invention and two graph structure variants is presented.

[0201] Overall, the present invention outperforms its variants in most cases, demonstrating that using two different graph neural network structures can achieve better performance when the data scale is large and there is a certain degree of distinction between entities.

[0202] ③ Hyperparameter performance experiment

[0203] This experiment discusses the performance impact of the BERT encoder and its variants RoBERTa and DeBERTa.

[0204] Based on RoBERTa base andDeBERTa base This invention was implemented on the DocRED dataset, and the experimental results are as follows: Figure 13 As shown, from Figure 13 It can be seen that the present invention, which uses RoBERTa and DeBERTa as encoders, outperforms the native BERT on the DocRED dataset. However, the performance difference between RoBERTa and DeBERTa is not significant, indicating that the improved BERT encoder has a certain improvement effect on the present invention, but the magnitude of this improvement is decreasing.

[0205] This experiment primarily examines the impact of two important hyperparameters in this invention on relation prediction performance: the negative sample ratio α and the positive-negative sample balancing factor β. The negative sample ratio α mainly controls the number of negative samples used in the experiment. For example, α = 2 indicates that twice the number of positive samples are used for training with negative samples. The value range of α is {2, 3, 4, 5, 6}. The positive-negative sample balancing factor β is mainly used to adjust the loss value between positive and negative samples, making the cross-entropy loss focus more on positive samples. Since β > 0.5 will make the loss function focus more on negative samples, the value range of β is {0.1, 0.2, 0.3, 0.4, 0.5}. When β = 0.5, the loss function is the traditional cross-entropy loss function.

[0206] Experimental results are as follows Figure 14 As shown.

[0207] The experimental results show that on the DocRED dataset, when α < 4, the effect of β is not significant. This is because the number of negative samples used is too small, resulting in insufficient model fitting ability and thus poor performance. When α >= 4, the overall performance of the invention recovers and remains in a relatively stable range. In particular, when β = 0.2, the invention achieves optimal performance in this range. On the CDR dataset, when α = 3, the invention performs well. Specifically, when β = 0.2 or 0.3, the invention achieves optimal performance in the experimental range. However, as the proportion of negative samples α increases, the performance of the invention on the CDR dataset does not improve synchronously but instead decreases.

[0208] This may be because the CDR dataset is too small, with too few positive samples. Introducing too many negative samples would lead to a decrease in the model's learning ability. Moreover, due to the small number of positive and negative samples used, the effect of the positive and negative sample balancing factor β is not obvious on the CDR dataset.

[0209] ④t-SNE visualization

[0210] This experiment will use t-SNE dimensionality reduction technology to visualize the effectiveness of the feature representation of the present invention.

[0211] For simplicity without sacrificing generality, this experiment selected DRN and ALTOP as comparison methods, where DRN is a document graph-based method and ALTOP is a document sequence encoding-based method. The five most frequently occurring relation labels in the DocRED development set were statistically analyzed. For each relation label, 100 entity pairs (excluding multi-label samples) were selected, and the last-layer output features were visualized using t-SNE dimensionality reduction (normalized to the interval [-1, 1]). Figure 15 As shown.

[0212] from Figure 15 It can be seen that all three methods can effectively distinguish these five types of relation samples, but they differ in region density. The ALTOP method only clearly distinguishes one type of sample, but the boundary delineation of the remaining four types of relation samples is not clear enough. Specifically, see... Figure 15 In (a), the four types of relationship samples at the bottom are clustered together, and a small number of samples exhibit drift. The DRN method clearly distinguishes two of these types of samples, but its classification effect on the remaining three types of relationship samples is slightly worse. Figure 15 (b) The top three types of relation samples are clustered together. For example... Figure 15 As shown in (c), the boundary division of all 5 types of samples in this invention is relatively clear.

[0213] The above experimental comparisons demonstrate that the present invention possesses a strong relation classification capability.

[0214] The above data results show that the relation extraction performance of this invention on these two benchmark datasets is superior to that of the current mainstream models.

[0215] It needs to be specifically noted that Figure 7 and Figure 8 The dataset was trained on the DocRED development set and CDR using five different random seeds. The mean and standard deviation of the IgnF1 and F1 scores are given. Results marked with an asterisk (*) are obtained by reproducing the results on the same dataset. The result of this step is that the encoder portion of these methods' source code has been reimplemented and reproduced on the dataset; The index indicated by the symbol represents the method that did not perform this experiment. All data under this index were re-performed in this experiment, and the remaining results use the data provided in these methods.

[0216] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A document-level relation extraction method based on multi-graph neural network inference, characterized in that, Includes the following steps, Step 1: Obtain the document dataset and the given control entity pairs. The document dataset includes the training document dataset and the test document dataset. The control entity pairs include a first control entity pair and a second control entity pair. Step 2: Construct a pre-document level relation extraction model based on multi-graph neural network reasoning, and divide the training document dataset from Step 1 into several groups of training document sequences and the first control entity pairs into the pre-document level relation extraction model for training. Output the relation prediction probability of the first document entity pairs in the training document sequences, and obtain the trained post-document level relation extraction model. Step 3: Input the test document dataset and the given second reference entity pair into the post-document level relation extraction model trained in Step 2 to obtain the second document entity pair relation prediction probability corresponding to the test document dataset; Step 2 above includes the following steps: Step 2.1: Input, input the training document sequence from Step 1 and the first control entity pair; Step 2.2: Feature extraction, calculate and reason from the training document sequence in Step 2.1 to obtain graph convolution features, graph attention features and global attention features; Step 2.3: Based on the position information of the first reference entity pair in Step 2.1, find the corresponding vector representation in the graph convolution feature, graph attention feature and global attention feature in Step 2.2, concatenate them to obtain the final relation feature, and calculate the relation prediction value of the first document entity pair in the training document sequence based on this. Step 2.4: Based on step 2.3, a balanced cross-entropy loss function is introduced to improve the noise problem, and the balanced cross-entropy loss function is used to train the pre-document-level relation extraction model; Step 2.2 above includes, Step 2.2.1: Encoding. The pre-document level relation extraction model encodes the input training document sequence by inputting the documents in the training document sequence from Step 2.1 into the encoder to obtain the sequence encoding. Step 2.2.2: Construct a document graph. For any given document in the training document sequence of Step 2.1, construct a document graph. The document graph includes a set of nodes and a set of path edges. The set of nodes includes a set of sentence nodes and a set of entity nodes in the document. The set of path edges includes a set of intra-sentence paths, a set of inter-sentence paths, and a set of logical paths. Construct the adjacency matrix corresponding to the document graph. Step 2.2.3: Graph Neural Network Inference. The sequence encoding from Step 2.1 and the node set from Step 2.2.2 are used to generate graph node encoding, and the graph node encoding matrix is ​​obtained using the logsumexp function. Then, the graph node encoding matrix and the adjacency matrix are input into the multi-graph neural network component for inference. The multi-graph neural network component includes a graph convolutional neural network (GCN) and a graph attention network (GAT) to obtain graph convolutional features and graph attention features, respectively. Step 2.2.4: Generate global attention features, calculate the multi-head self-attention features obtained by sequence encoding in step 2.2.1, and concatenate the multi-head self-attention features to obtain context features. Then, use the graph node encoding matrix from step 2.2.3 and the context features to calculate the global attention features of the document sequence with graph node encoding information.

2. The method of claim 1, wherein the multi-graph neural network-based inference is performed on a document level. In step 1 above, the training document dataset and the test document dataset are divided into different batches. Each batch has a set of document sequences, namely the training document sequence and the test document sequence. The training document sequence has a first document entity pair, and the test document sequence has a second document entity pair.

3. The method of claim 1, wherein, The intra-sentence paths, inter-sentence paths, and logical paths will be retained when different types of paths appear in the document. For multiple paths of the same type, only the first path that is used will be retained. The intra-sentence paths and inter-sentence paths are actual paths, while the logical paths are virtual paths.

4. The method of claim 1, wherein, The Graph Convolutional Neural Network (GCN) constructs a network with several layers, performs layered calculations on the graph node encoding matrix and adjacency matrix to obtain the graph convolutional encoding matrix of the document graph, and the graph convolutional encoding matrix is ​​a graph convolutional feature.

5. The method of claim 1, wherein, The graph attention network (GAT) first calculates the attention value between two nodes in the node set, then aggregates nodes through the attention value and updates the node encoding of the nodes. It calculates the multi-head attention encoding of each node in the node set and concatenates them into a multi-head attention matrix, which is a graph attention feature.

6. The method of claim 5, wherein, The calculation direction of the attention value is unidirectional.

Images