Document-level biomedical relationship extraction method based on converter

Abstract

The invention discloses a document-level biomedical relationship extraction method based on a converter, and the method comprises the steps: designing a multi-dimensional information guide mention integration module, constructing a mention graph to obtain the structural information of mentions, and capturing the mentioned context sensing information through a context sensing attention matrix, the guidance mention information is integrated to obtain more flexible entity representation, so that the determination of entity-level relation facts is more facilitated; a multi-dimensional reasoning method comprising a multi-mode logical reasoning module and a U-Net reasoning module is designed, the reasoning module is constructed by utilizing three logical modes and a U-Net network, logical reasoning is performed from a multi-mode logical relationship angle and a global dependency interaction angle in combination with reasoning information, an implicit relationship between entities is better deduced, and the reasoning efficiency is improved. The two reasoning modes are different in emphasis and complement each other, more sufficient and effective reasoning information is obtained, and the logical reasoning ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of biomedical natural language processing, and in particular to a document-level biomedical relationship extraction method based on a converter. Background Art

[0002] In recent years, with the continuous deepening of biomedical research, the number of biomedical literature has grown exponentially. These documents contain rich biomedical knowledge and are of great significance for promoting research and application in the medical field. However, it is extremely difficult for researchers to read such a large number of documents. Therefore, automatically extracting structured knowledge from biomedical literature has become a crucial task. Among them, the task of biomedical relationship extraction has been widely used in the field of biomedical natural language processing and plays a very important role.

[0003] The goal of biomedical relation extraction is to determine the relations between biomedical entity pairs from unstructured biomedical text. Most previous methods focus on predicting the relations between entities in a single sentence, i.e., sentence-level relation extraction. However, from a practical perspective, relational facts in biomedical text are mostly expressed jointly by multiple sentences. Therefore, many recent studies have shifted their attention to biomedical document-level relation extraction, i.e., using the entire input document to extract intra-sentence entity pair relations and cross-sentence entity pair relations. These studies can be divided into sequence-based, graph-based, and transformer-based models.

[0004] Sequence-based methods usually use convolutional neural networks (CNNs) or long short-term memory (LSTMs) to extract relations in documents. These methods perform well when processing sequence data, but they are limited in capturing long-distance dependencies in documents. To overcome this limitation and improve the performance of models on complex tasks, researchers began to explore graph-based methods, which capture the relationship between entities in documents by constructing document graphs. Then, with the development of pre-trained models, transformer-based methods have shown great potential in document-level biomedical relation extraction tasks due to their excellent ability to capture contextual information.

[0005] However, with the continuous deepening of research, document-level biomedical relationship extraction based on transformers faces some challenges. First, most methods focus on the mining of entity-level information, but do not fully mine the information at the mention level, ignoring the rich structural information and association information between mentions. In addition, existing research rarely studies the integration of mention information. It only performs simple operations such as average pooling and logsumexp pooling on the mention-level information to obtain entity representation. All mentions are treated equally, resulting in some irrelevant mentions interfering with the determination of subsequent entity-level relationships. In addition, since many relationships are implicitly expressed in documents and involve complex interactions between entities across sentences, it is necessary to use existing relationships to judge, so the document-level biomedical relationship extraction model should take into account the logical association between different relationship triplets in the document. Although some existing models contain reasoning modules, these methods often perform logical reasoning based on only one perspective, ignoring the complementarity of reasoning information from different perspectives, and the model reasoning ability is limited. Summary of the invention

[0006] The present invention aims to solve the above problems existing in the prior art and provides a document-level biomedical relationship extraction method based on a converter.

[0007] The technical solution of the present invention is: a document-level biomedical relationship extraction method based on a converter, which is extracted based on a switch network model, wherein the switch network model is composed of a document encoding module, a multi-dimensional information guided mention integration module, a multi-mode logic reasoning module, a U-Net reasoning module and a classifier module, and is performed in the following steps:

[0008] Step 1. Through the document encoding module, the document is mapped to an embedded representation containing rich contextual semantic information, as follows:

[0009] Step 1.1 Input the document and insert a special marker symbol at the beginning and end of the mention to mark the scope of the mention;

[0010] Step 1.2 uses BioBERT as a document encoder to map each token and mention tag to an embedding representation containing rich contextual semantic information;

[0011] Step 2. Guide the mention integration module through multi-dimensional information to fuse the specific context-aware information of the head and tail entity mentions with the structural information to obtain the head and tail entity representations, and then construct the entity pair feature matrix as follows:

[0012] Step 2.1 For each mention, the embedding of the start tag is used as the initial embedding of the mention, and then the complex interaction information between mentions is modeled to construct a heterogeneous mention level graph MG. The GCN graph convolutional network is applied to the graph MG to obtain the mention representation m with structural information of the head and tail entities. s ;

[0013] Step 2.2 Construct entity e from scratch h The attention matrix of the ith mention to all tokens in the document Head Entity h All mentioned A m Take the average to get the head entity perception context matrix A h ; For the tail entity e t Perform the same operation to obtain the tail entity perception context matrix A t ; According to formulas (1) and (2), we obtain the entity pair (e h ,e t )'s context-specific embedding c (h,t) ;

[0014]

[0015] c (h,t) =Ha (h,t) (2)

[0016] Step 2.3 Given an entity pair (e h ,e t ) and the mention representation sequence of the head entity or the tail entity, and calculate the head entity representation according to formulas (3)-(5) and tail entity representation

[0017]

[0018] Among them, W Q ∈R d×d ,W K ∈R d×d represents the query and key transition matrix, d is the dimension of the hidden state, and Z is the number of mentions;

[0019] Step 2.4: Through the feedforward neural network, the feature vector F is obtained as shown in formula (6): h,t ;

[0020]

[0021] Step 2.5: Merge the feature vectors of all entity pairs in the document to form the entity pair feature matrix M (0) =[F h,t ]P×P , where each row Corresponding to a head entity e h , each column Corresponding to a tail entity e t , P is the number of entities;

[0022] Step 3. Through the multi-modal logic reasoning module composed of l reasoning layers, the model's logic reasoning ability is trained from three modes: reasoning with the same head entity pair, reasoning with the same tail entity pair, and reasoning with the same head and tail entity pair. Three attention heads are used to simulate three common reasoning modes respectively, as follows:

[0023] Step 3.1 The entity pair feature matrix M obtained by the multi-dimensional information guided mention integration module (0) Input to the feature processing unit, which consists of a 2D convolution with a kernel size of 5×5 and a ReLU activation function, the feature information M of the entity pair can be obtained c , M c The feature vectors in are concatenated and then linearly transformed through a single linear layer to obtain

[0024] Step 3.2 Obtain the output vector of the first head of the entity pair through the multi-head self-attention unit Similarly, calculate the output vectors of the other two heads

[0025] The definition of multi-head self-attention unit is as shown in formula (7):

[0026]

[0027] in,

[0028] Step 3.3: Add the outputs of all heads and feed them into the layer normalization unit to get the inference output M of the lth layer. l , and then calculate the output M of the (l+1)th inference layer (l+1) ;

[0029] Step 3.4 Repeat the above process l times to obtain a more expressive entity feature matrix M (l) , and then get the entity relationship reasoning representation r e , r e From M (l) ;

[0030] Step 4. Use the U-Net reasoning module to model the relationship between entity pairs and use the global dependency between entity pairs to infer the implicit relationship between entities, as follows:

[0031] Step 4.1 The entity pair feature matrix M obtained by the multi-dimensional information guided mention integration module (0) Input into U-Net, perform downsampling and upsampling operations, and obtain an entity-level graph Y of the same size as the input;

[0032] Step 4.2: Generate head and tail entity representations after integrating entity-level graph Y and mentions According to formula (8)-(10)), the fusion is performed to obtain the fused feature representation u h 、u t , and then obtain the fused entity relationship reasoning representation r u ;

[0033]

[0034] P(r|e h ,e t )=σ(u h W p u t +b p ) (10)

[0035] Among them, W h ∈R d×d ,W t ∈R d×d and W p ∈R d×d is a learnable parameter, tanh(),σ is an activation function;

[0036] Step 5. Obtain the relation instance r=(e h ,e t ) represents r, and classifies the relationship as follows:

[0037] Step 5.1 For each entity pair (e h ,e t ), concatenates the following representations: (1) the head and tail entity representations generated by integrating the specific context-aware information of the mention with the structural information f ; (2) The entity relationship reasoning representation r obtained by the multi-modal logic reasoning module e ; (3) The entity relationship reasoning representation r obtained by the U-Net reasoning module u ;

[0038] The r f The calculation process of is shown in (11):

[0039]

[0040] Step 5.2 uses adaptive threshold loss as the classification loss and uses a feedforward neural network to calculate the probability of each relationship instance according to formula (12):

[0041] P(r|e h ,e t )=δ(W 2 σ(W 1 r+b 1 )+b 2 ) (12)

[0042] Where W * , b * is a learnable parameter, σ() is the activation function, and δ() is the Sigmoid function.

[0043] Compared with the prior art, the present invention designs a multi-dimensional information guided mention integration module, constructs a mention graph to obtain structural information between mentions, and uses context-aware attention matrix to capture context-aware information of mentions, and then guides the mention information to integrate to obtain more flexible entity representation, avoid the interference of irrelevant information, and capture more information at the mention level, which is more conducive to the determination of entity-level relationship facts; on the other hand, a multi-dimensional reasoning method including a multi-modal logic reasoning module and a U-Net reasoning module is designed, and the reasoning module is constructed using three logic modes and U-Net reasoning module, and combined with reasoning information, logical reasoning is performed from the perspective of multi-modal logical relationship and global dependency interaction to better infer the implicit relationship between entities. The two reasoning methods have different focuses and can complement each other, thereby obtaining more sufficient and effective reasoning information and improving the logical reasoning ability of the model. Experimental results on two widely used biomedical datasets CDR and GDA show that the present invention has excellent competitive performance and effectively improves the overall performance of document-level biomedical relationship extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the overall architecture of an embodiment of the present invention.

[0045] Figure 2 It is a schematic diagram of the architecture of the multi-mode logic reasoning module of an embodiment of the present invention.

[0046] Figure 3 Schematic diagram of the architecture of the U-Net reasoning module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The invention discloses a document-level biomedical relationship extraction method based on a transformer, which is based on a transformer network model. Figure 1As shown, it consists of a document encoding module, a multi-dimensional information guided mention integration module, a multi-modal logic reasoning module, a U-Net reasoning module and a classifier module, and proceeds in the following steps:

[0048] Step 1. Through the document encoding module, the document is mapped to an embedded representation containing rich contextual semantic information, as follows:

[0049] Step 1.1 Input the document and insert a special marking symbol "*" at the beginning and end of the mention to mark the range of the mention;

[0050] Step 1.2 uses BioBERT as a document encoder to map each token and mention tag to an embedding representation containing rich contextual semantic information;

[0051] Step 2. Guide the mention integration module through multi-dimensional information to fuse the specific context-aware information of the head and tail entity mentions with the structural information to obtain the head and tail entity representations, and then construct the entity pair feature matrix as follows:

[0052] Step 2.1 For each mention, the embedding of the start tag is used as the initial embedding of the mention, and then the complex interaction information between mentions is modeled to construct a heterogeneous mention level graph MG. The GCN graph convolutional network is applied to the graph MG to obtain the mention representation m with structural information of the head and tail entities. s ;

[0053] Step 2.2 Construct entity e from scratch h The attention matrix of the ith mention to all tokens in the document Head Entity h All mentioned A m Take the average to get the head entity perception context matrix A h ; For the tail entity e t Perform the same operation to obtain the tail entity perception context matrix A t ; According to formulas (1) and (2), we obtain the entity pair (e h ,e t )'s context-specific embedding c (h,r) ;

[0054]

[0055] c (h,t) =Ha (h,t) (2)

[0056] Step 2.3 Given an entity pair (e h ,e t ) and the mention representation sequence of the head entity or the tail entity, and calculate the head entity representation according to formulas (3)-(5) and tail entity representation

[0057]

[0058] Among them, W Q ∈R d×d ,W K ∈R d×d represents the query and key transition matrix, d is the dimension of the hidden state, and Z is the number of mentions;

[0059] Step 2.4: Through the feedforward neural network, the feature vector F is obtained as shown in formula (6): h,t ;

[0060]

[0061] Step 2.5: Merge the feature vectors of all entity pairs in the document to form the entity pair feature matrix M (0) =[F h,t ] P×P , where each row Corresponding to a head entity e h , each column Corresponding to a tail entity e t , P is the number of entities;

[0062] Step 3. Figure 2 The multi-mode logic reasoning module composed of l reasoning layers is shown in the figure. It trains the model's logic reasoning ability from three modes: reasoning with same-head entity pairs, reasoning with same-tail entity pairs, and reasoning with same-head and same-tail entity pairs. It uses three attention heads to simulate three common reasoning modes, as follows:

[0063] Step 3.1 The entity pair feature matrix M obtained by the multi-dimensional information guided mention integration module (0) Input to the feature processing unit, which consists of a 2D convolution with a kernel size of 5×5 and a ReLU activation function, the feature information M of the entity pair can be obtained c , M c The feature vectors in are concatenated and then linearly transformed through a single linear layer to obtain

[0064] Step 3.2 Obtain the output vector of the first head of the entity pair through the multi-head self-attention unit Similarly, calculate the output vectors of the other two heads

[0065] The definition of multi-head self-attention unit is as shown in formula (7):

[0066]

[0067] in,

[0068] Step 3.3: Add the outputs of all heads and feed them into the layer normalization unit to get the inference output M of the lth layer. l , and then calculate the output M of the (l+1)th inference layer (l+1) ;

[0069] Step 3.4 Repeat the above process l times to obtain a more expressive entity feature matrix M (l) , and then get the entity relationship reasoning representation r e , r e From M (l) ;

[0070] Step 4. Figure 3 The U-Net reasoning module shown in Figure 1 models the relationship between entity pairs and uses the global dependency between entity pairs to infer the implicit relationship between entities, as follows:

[0071] Step 4.1 The entity pair feature matrix M obtained by the multi-dimensional information guided mention integration module (0) Input into U-Net, perform downsampling and upsampling operations, and obtain an entity-level graph Y of the same size as the input;

[0072] Step 4.2: Generate head and tail entity representations after integrating entity-level graph Y and mentions According to formula (8)-(10)), the fusion is performed to obtain the fused feature representation u h 、u t , and then obtain the fused entity relationship reasoning representation r u ;

[0073]

[0074] P(r|e h ,e t )=σ(u h W p u t +b p ) (10)

[0075] Among them, W h ∈R d×d ,W t ∈R d×d and W p ∈R d×d is a learnable parameter, tanh(),σ is an activation function;

[0076] Step 5. Obtain the relation instance r=(e h ,e t ) represents r, and classifies the relationship as follows:

[0077] Step 5.1 For each entity pair (e h ,e t ), concatenates the following representations: (1) the head and tail entity representations generated by integrating the specific context-aware information of the mention with the structural information f ; (2) The entity relationship reasoning representation r obtained by the multi-modal logic reasoning module e ; (3) The entity relationship reasoning representation r obtained by the U-Net reasoning module u ;

[0078] The r f The calculation process of is shown in (11):

[0079]

[0080] Step 5.2 uses adaptive threshold loss as the classification loss and uses a feedforward neural network to calculate the probability of each relationship instance according to formula (12):

[0081] P(r|e h ,e t )=δ(W 2 σ(W 1 r+b 1 )+b 2 ) (12)

[0082] Where W * , b * is a learnable parameter, σ() is the activation function, and δ() is the Sigmoid function.

[0083] In order to verify the effectiveness of the present invention, experiments were conducted on two widely used document-level biomedical relationship extraction datasets, CDR and GDA, based on the method provided by the present invention. The Biocreative VChemical DiseaseRelationbenchmark (CDR) is a manually annotated dataset consisting of 1,500 PubMed articles (divided equally into 500 articles for training, validation, and testing, respectively), which aims to determine the relationship between chemical entities and disease entities in text. The Gene Disease Associations (GDA) is a large-scale relationship extraction dataset in the biomedical field, which is constructed through remote supervision of 30,192 abstracts from MEDLINE (the test set is 1,000, and the rest are allocated between the training set and the validation set in a ratio of 80:20), which aims to determine the association between gene entities and disease entities in text. The statistics of the two datasets are shown in Table 1.

[0084] Table 1 Statistics of CDR and GDA datasets

[0085]

[0086] The evaluation standard used in the experiment is the F1 value, which is a comprehensive evaluation standard for evaluating general classification problems. In addition, the Intra-F1 and Inter-F1 metrics are also used to evaluate the performance of the model on intra-sentence relations and inter-sentence relations.

[0087] The performance comparison results between the proposed method and the existing methods are shown in Tables 2 and 3. On the CDR dataset, the proposed method performed excellently, with F1, Intra-F1, and Intre-F1 values ​​reaching 86.7%, 89.2%, and 80.7%, respectively, showing better performance than the existing methods. Compared with the most advanced baseline model FILR model, the proposed method improved the overall F1 value by 1.0%, and improved Intra-F1 and Inter-F1 by 0.1% and 3.5%, respectively. This significant performance improvement is attributed to the effective use of multi-dimensional information-guided mention integration and multiple reasoning methods by the proposed method. Through multi-dimensional information-guided mention integration, the context-aware information and structural information of the head and tail entity mentions are dynamically integrated, providing richer mention feature information for generating head and tail entity representations, enabling the model to focus on the mentions that are most helpful for relationship prediction, reducing the interference of other irrelevant information, and thus generating more effective entity representations. At the same time, a multi-factor reasoning method including a multi-modal logic reasoning module and a U-Net reasoning module is utilized to model the interactions between different relationship pairs from the three reasoning modes and the global dependencies between entities, thus realizing relational reasoning more comprehensively and effectively.

[0088] Table 2 Experimental results of the present invention on CDR dataset

[0089]

[0090] Note: The “-” in this table refers to data not given in the original paper.

[0091] The method of the present invention performs well on the GDA dataset, with F1, Intra-F1, and Intre-F1 values ​​reaching 84.3%, 87.5%, and 63.9%, respectively, which is better than most baseline models on GDA and reaches an advanced level. It is worth noting that compared with the current leading CGM2IR model, the Intra-F1 value of the method of the present invention is 4.9% higher than that of CGM2IR, and the Intre-F1 value is 0.8% lower than that of CGM2IR, which indicates that the method of the present invention mainly improves the performance of inter-sentence relations. However, since the number of inter-sentence relations only accounts for 16% of the total relations shown in Table 1, the increase in the Intre-F1 value fails to bring the same gain in the overall F1 value. And the scale of the GDA dataset is nearly 50 times larger than that of CDR, so the improvement in all evaluation indicators is not significant. Although the MIMR model did not obtain the current optimal results on the GDA dataset, compared with the current leading CGM2IR model, the model of the present invention has unique advantages in extracting inter-sentence relations and obtains comparable F1.

[0092] Table 3 Experimental results of the present invention on the GDA dataset

[0093]

[0094]

[0095] Note: The “-” in this table refers to data not given in the original paper.

[0096] All test results verify the effectiveness and advancement of the method in the document-level biomedical relationship extraction task.

Claims

1. A document-level biomedical relationship extraction method based on a transformer is based on a transformer network model, characterized in that: The switch network model is composed of a document encoding module, a multi-dimensional information guided mention integration module, a multi-mode logic reasoning module, a U-Net reasoning module and a classifier module, and is performed in the following steps: Step 1. Through the document encoding module, the document is mapped to an embedded representation containing rich contextual semantic information, as follows: Step 1.1 Input the document and insert a special marker symbol at the beginning and end of the mention to mark the scope of the mention; Step 1.2 uses BioBERT as a document encoder to map each token and mention tag to an embedding representation containing rich contextual semantic information; Step 2. Guide the mention integration module through multi-dimensional information to fuse the specific context-aware information of the head and tail entity mentions with the structural information to obtain the head and tail entity representations, and then construct the entity pair feature matrix as follows: Step 2.1 For each mention, the embedding of the start tag is used as the initial embedding of the mention, and then the complex interaction information between mentions is modeled to construct a heterogeneous mention level graph MG. The GCN graph convolutional network is applied to the graph MG to obtain the mention representation m with structural information of the head and tail entities. s ; Step 2.2 Construct entity e from scratch h The attention matrix of the ith mention to all tokens in the document Head Entity h All mentioned A m Take the average to get the head entity perception context matrix A h ; For the tail entity e t Perform the same operation to obtain the tail entity perception context matrix A t ; According to formulas (1) and (2), we obtain the entity pair (e h ,e t )'s context-specific embedding c (h,t) ; Step 2.3 Given an entity pair (e h ,e t ) and the mention representation sequence of the head entity or the tail entity, and calculate the head entity representation according to formulas (3)-(5) and tail entity representation Among them, W Q ∈R d×d ,W K ∈R d×d represents the query and key transition matrix, d is the dimension of the hidden state, and Z is the number of mentions; Step 2.4: Through the feedforward neural network, the feature vector F is obtained as shown in formula (6): h,t ; Step 2.5: Merge the feature vectors of all entity pairs in the document to form the entity pair feature matrix M (0) =[F h,t ] P×P , where each row Corresponding to a head entity e h , each column Corresponding to a tail entity e t , P is the number of entities; Step 3. Through the multi-modal logic reasoning module composed of l reasoning layers, the model's logic reasoning ability is trained from three modes: reasoning with the same head entity pair, reasoning with the same tail entity pair, and reasoning with the same head and tail entity pair. Three attention heads are used to simulate three common reasoning modes respectively, as follows: Step 3.1 The entity pair feature matrix M obtained by the multi-dimensional information guided mention integration module (0) Input to the feature processing unit, which consists of a 2D convolution with a kernel size of 5×5 and a ReLU activation function, the feature information M of the entity pair can be obtained c , M c The feature vectors in are concatenated and then linearly transformed through a single linear layer to obtain Step 3.2 Obtain the output vector of the first head of the entity pair through the multi-head self-attention unit Similarly, calculate the output vectors of the other two heads The definition of multi-head self-attention unit is as shown in formula (7): in, Step 3.3: Add the outputs of all heads and feed them into the layer normalization unit to get the inference output M of the lth layer. l , and then calculate the output M of the (l+1)th inference layer (l+1) ; Step 3.4 Repeat the above process l times to obtain a more expressive entity feature matrix M (l) , and then get the entity relationship reasoning representation r e , r e From M (l) ; Step 4. Use the U-Net reasoning module to model the relationship between entity pairs and use the global dependency between entity pairs to infer the implicit relationship between entities, as follows: Step 4.1 The entity pair feature matrix M obtained by the multi-dimensional information guided mention integration module (0) Input into U-Net, perform downsampling and upsampling operations, and obtain an entity-level graph Y of the same size as the input; Step 4.2: Generate head and tail entity representations after integrating entity-level graph Y and mentions According to formula (8)-(10)), the fusion is performed to obtain the fused feature representation u h 、u t , and then obtain the fused entity relationship reasoning representation r u ; P(r|e h ,e t )zσ(u h W p u t +b p ) (10) Among them, W h ∈R d×d ,W t ∈R d×d and W p ∈R d×d is a learnable parameter, tanh(),σ is an activation function; Step 5. Obtain the relation instance r=(e h ,e t ) represents r, and classifies the relationship as follows: Step 5.1 For each entity pair (e h ,e t ), concatenates the following representations: (1) the head and tail entity representations generated by integrating the specific context-aware information of the mention with the structural information f ; (2) The entity relationship reasoning representation r obtained by the multi-modal logic reasoning module e ; (3) The entity relationship reasoning representation r obtained by the U-Net reasoning module u ; The r f The calculation process of is shown in (11): Step 5.2 uses adaptive threshold loss as the classification loss and uses a feedforward neural network to calculate the probability of each relationship instance according to formula (12): P(r|e h ,and t )=δ(W2σ(W1r+b1)+b2) (12) Where W * , b * is a learnable parameter, σ() is the activation function, and δ() is the Sigmoid function.

Images