A drug relationship extraction method based on drug entity word masking and Insert-BERT structure

Abstract

The present invention relates to the field of biomedical relationship extraction, and specifically to a drug relationship extraction method based on drug entity word masking and Insert‑BERT structure. The present invention has comprehensive improvements in drug relationship multi-classification tasks and drug relationship detection binary classification tasks. The results of model visualization and ablation experiments show that the proposed method can extract semantic information of different levels and granularities in a continuous space, and can effectively restrict the influence of negative samples.

Description

Technical Field

[0001] The present invention relates to the technical field of biomedical relationship extraction, and in particular to a drug relationship extraction method based on drug entity word masking and Insert-BERT structure. Background Art

[0002] Drug-drug-interaction extraction (DDI) aims to extract the relationship between two or more drugs in the human body and their interactions. It is the most typical task in the field of biomedical relationship extraction. Statistical studies have shown that more and more people need to take multiple drugs at the same time, and the interactions between multiple drugs will have a serious impact on the healthy human body. Therefore, further understanding of drug interactions and designing and developing drug relationship classification systems are of great significance to reducing drug abuse accidents. In view of this, DDI-related work has attracted the attention of more and more researchers, and a variety of data sets reflecting drug action relationships have emerged. However, with the rapid development of biomedicine, the literature describing the mechanism of biological drugs has exploded, and it is no longer possible to analyze and discover drug relationships manually. Therefore, based on the massive literature data describing drug mechanisms, exploring automatic and accurate drug relationship extraction methods has become a problem to be solved in the field of DDI.

[0003] As more and more people take multiple drugs at the same time, causing human harm and drug abuse accidents, how to explore drug interactions from various medical logs or records has attracted great attention from researchers. In the early stage, drug relationship extraction methods were mainly based on traditional machine learning techniques represented by feature engineering. Such methods are limited by sparse data and the relationship extraction effect is not good; in addition, the non-reusability of feature extraction methods also makes the promotion of such methods a big problem. In recent years, methods represented by deep learning have rapidly expanded to the field of drug relationship extraction, especially pre-training models represented by BERT have attracted widespread attention in the field of relationship extraction. However, how to improve the pre-training effect of BERT and better adapt to downstream drug relationship classification tasks still needs further exploration. In addition, although there is a huge amount of data describing drug mechanisms, drug relationship annotation requires a lot of medical professional knowledge, resulting in manually annotated supervised samples only maintaining at the level of 100,000, which is far lower than the data scale in natural language processing fields such as text classification and entity recognition. Relevant studies have shown that it is difficult to train deep neural network models such as CNN or RNN with only a small amount of supervised data, and its accuracy and generalization ability are difficult to guarantee.

[0004] In summary, developing a drug relationship extraction method based on drug entity word masking and Insert-BERT structure is still a key issue that needs to be urgently solved in the field of drug relationship extraction technology. Summary of the invention

[0005] In view of the above shortcomings of the prior art, the present invention firstly improves the completely random masking method in the masked language model under the guidance of the drug entity library, and proposes a pre-training strategy based on drug entity word masking (Entity-Mask-Bert) in order to address the weakness of the BERT pre-training task in expressing the co-existing entity information. Secondly, in the fine-tuning stage of drug relationship classification, four signs are introduced to distinguish different entity combinations of the same input sequence in order to address the problem of poor adaptation of downstream tasks. Based on the Entity-Mask-Bert pre-training model, a fine-tuning scheme of the Insert-BERT structure is proposed. The experimental results of DDI-Extraction 2013 show that the proposed unsupervised + supervised drug relationship extraction method achieves F=0.81. Compared with the existing methods, the proposed method has comprehensive improvements in drug relationship multi-classification tasks and drug relationship detection binary classification tasks. The results of model visualization and ablation experiments show that the proposed method can extract semantic information of different levels and granularities in continuous space, and can effectively restrict the influence of negative samples.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A drug relationship extraction method based on drug entity word masking and Insert-BERT structure, including:

[0008] S1. In the word sequence masking stage of the input layer, drug entity detection is achieved by string matching based on the constructed drug name entity library;

[0009] S2. In the input stage, only one of the drug entities is randomly selected for masking, and for the remaining non-entity words in the sentence, the conventional BERT masking strategy is followed;

[0010] S3. Based on the pre-trained Entity-Mask-BERT, the drug relationship classification of the Insert-BERT structure is constructed by introducing a mark to achieve the purpose of extracting different drug entity relationships in the same input sentence sequence;

[0011] S4. To distinguish the problem that the same input sentence contains different entity combinations, four new symbols, START1, END1, STRAT2 and END2, are added, so that the same input sentence sequence has different representations at the input layer;

[0012] S5. In the early stage of Insert-BERT model training, an optimization strategy based on warmup + triangular learning rate is adopted. In the later stage of training, as the model gradually converges, the learning rate is continuously reduced to fine-tune the model.

[0013] The present invention is further configured as follows: in step S2, when there are multiple entities in the input sentence, a strategy of masking one of the drug entities and using other drug entities to predict the current masked entity can realize unsupervised high-frequency drug entity relationship modeling and complete the pre-training task based on massive data describing drug mechanisms.

[0014] The present invention is further configured as follows: in step S3, for the input sentence sequence X = [x1, x2, e3, x4, e5...e 10 ,···x M ];

[0015] Where M represents the total number of words contained in the input sentence;

[0016] e3, e5 and e 10 Represent 3 drug entities respectively;

[0017] x1,x2,x4,··x M represents a non-drug entity;

[0018] The downstream classification task needs to judge [e3, e5], [e3, e 10 ] and [e3,e 10 ], the relationship between these three drug entity combinations, and the entity combinations share the same input sequence X.

[0019] The present invention is further configured as follows: in step S4, four symbols START1, END1, STRAT2 and END2 are added to determine the entity combination [e n , e m ]The relationship between them.

[0020] The present invention is further configured to: determine the entity combination [e n , e m The relationship between ] is:

[0021] S40, in e n Insert symbols START1 and END1 before and after to mark the location of the first entity;

[0022] S41, in e m Insert symbols START2 and END2 before and after to mark the location of the second entity;

[0023] S42, Similarly, entity combination [e n , ev ] and [e m , e v ] are similarly operated, an input sentence containing N entities will generate Thereby, the same input sentence sequence has different representations in the input layer.

[0024] The present invention is further configured as follows: in step S5, an Insert-BERT model is constructed, in which the main encoding unit is a Transformer, and the parameters of the encoding unit are initialized with the Transformer parameters obtained by Entity-MASK-Bert pre-training.

[0025] The present invention is further configured as follows: in the training phase of the model, assuming that the output of the [CLS] node is represented by r∈R k ;

[0026] S50, define the transfer matrix W∈R k×c Realize the mapping from output representation space to category probability space:

[0027] o=rw;

[0028] Where o=(o1,···,o j ,···,o c )∈R c represents the unnormalized probability of each category;

[0029] k and c represent the output vector dimension and the number of categories of drug entity relations, respectively;

[0030] S51. Use the Softmax function to transform the unnormalized probability space o into the normalized probability space s, as shown in the following formula:

[0031]

[0032] in, represents the p-th drug entity pair in the l-th input sentence;

[0033] express Belongs to category c j probability;

[0034] S52: Complete Insert-BERT model training using the formula in step S51 as the optimization target:

[0035]

[0036] Where L represents the total number of samples in the training set;

[0037] nl Indicates the number of drug entity pairs contained in the lth sample;

[0038] They represent the true value and predicted value of the p-th drug entity pair in the l-th sample respectively.

[0039] The present invention is further configured as follows: during the training and fine-tuning of the Insert-BERT model, the batch-size and the learning rate are set to 16 and 2e-5 respectively.

[0040] Beneficial Effects

[0041] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects:

[0042] The present invention proposes a drug relationship extraction method based on drug entity word masking and Insert-BERT structure. Firstly, the method aims at the weakness of expressing common entity information in BERT pre-training task, improves the completely random masking mode in the mask language model under the guidance of drug entity library, and proposes a pre-training strategy based on drug entity word masking (Entity-Mask-Bert); secondly, in the fine-tuning stage of drug relationship classification, four signs are introduced to distinguish different entity combinations of the same input sequence in order to solve the problem of poor adaptation of downstream tasks, and a fine-tuning scheme of Insert-BERT structure is proposed without destroying the Entity-Mask-Bert pre-training model. The experimental results of DDI-Extraction 2013 show that the proposed unsupervised + supervised drug relationship extraction method achieves F=0.81. Compared with the existing methods, the proposed method has comprehensive improvements in drug relationship multi-classification tasks and drug relationship detection binary classification tasks; the results of model visualization and ablation experiments show that the proposed method can extract semantic information of different levels and granularities in continuous space, and can effectively restrict the influence of negative samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the pre-trained BERT model based on the Transformer structure, which consists of an input layer, an encoding layer, and a decoding layer;

[0044] Figure 2 Schematic diagram of the pre-training process of Entity-Mask-BERT;

[0045] Figure 3 Schematic diagram built for the Insert-BERT model;

[0046] Figure 4 Schematic diagram for visualizing Insert-BERT attention weights. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The present invention will be further described below in conjunction with the embodiments.

[0049] Example 1

[0050] A drug relationship extraction method based on drug entity word masking and Insert-BERT structure is proposed. Aiming at the disadvantage that the completely random masking method of the pre-trained BERT model lacks entity tendency, the masked language model is improved on the basis of introducing drug entity recognition, and a pre-training strategy based on drug entity word masking (Entity-Mask-Bert) is proposed; secondly, based on the pre-trained Entity-Mask-BERT, for the drug relationship extraction task, this paper further designs the Insert-BERT structure to realize the classification of drug relationships; finally, the training process of Insert-BERT is briefly introduced.

[0051] From the training strategy of the masked language prediction task, we can see that BERT uses a completely random method to mask the input word sequence. Due to the lack of prior knowledge, this non-biased random method makes words with less information still masked, or the randomly masked words have less relevance and attention to the relationship extraction task. Figure 1 In the example above, the words "of" or "the" have relatively low information content, but they are still masked and predicted. These masked input sequences containing redundant information not only restrict the representation ability of the encoding layer, but also increase the difficulty of model training and affect the model's fitting ability. In order to eliminate the invalid masks generated by the completely random masking method and consider the pre-trained BERT model to better adapt to the downstream drug relationship classification task, this paper proposes a pre-training strategy based on drug entity word masking (Entity-Mask-BERT). The pre-training process is as follows: Figure 2 shown.

[0052] First, in the word sequence mask stage of the input layer, drug entity detection is achieved by string matching based on the constructed drug name entity library. Figure 2In the input sentence, “terbinafine” and “cyclosporine” marked in red are drug entity names. Then, in the input stage, different from the masking strategy in the pre-trained BERT model, only one of the drug entities is randomly selected for masking. Figure 2 In the sentence, there are two drug entities "terbinafine" and "cyclosporine", and "cyclosporine" is selected and masked at the input. For the remaining non-entity words in the sentence, this article follows the conventional BERT masking strategy. The masking strategy in this article mainly draws on the idea of ​​remote supervision in relationship extraction, that is, if two entities have a large frequency of co-occurrence in remote supervision learning, then the possibility that the two entities have a relationship is greater. When there are multiple entities in the input sentence, the strategy proposed in this article to mask one of the drug entities and use other drug entities to predict the current masked entity can realize unsupervised high-frequency drug entity relationship modeling and complete the pre-training task based on massive data describing drug mechanisms. In addition, unlike Figure 1 ,This paper discards the sentence pair judgment task during the pre-training process.

[0053] The quality of drug relationship extraction is not only closely related to the entity masking strategy introduced in the pre-trained Bert model, but also closely related to the construction of downstream domain classification task models. For an input sentence sequence, if it contains N entities, it is necessary to judge How to distinguish entities in the same input sentence sequence? There are many combinations and classifying each combination is the key problem that this paper needs to solve.

[0054] Specifically, for the input sentence sequence X = [x1, x2, e3, x4, e5...e 10 ,···x M ];

[0055] Where M represents the total number of words contained in the input sentence;

[0056] e3, e5 and e 10 Represent 3 drug entities respectively;

[0057] x1,x2,x4,··x M represents a non-drug entity;

[0058] The downstream classification task needs to judge [e3, e5], [e3, e 10 ] and [e3,e 10 ], the relationship between these three drug entity combinations, and the entity combinations share the same input sequence X.

[0059] Determine entity combination [e n, e m The relationship between ] is:

[0060] 1) In e n Insert symbols START1 and END1 before and after to mark the location of the first entity;

[0061] 2) In e m Insert symbols START2 and END2 before and after to mark the location of the second entity;

[0062] 3) Similarly, entity combination [e n , e v ] and [e m , e v ] are similarly operated, an input sentence containing N entities will generate Thereby, the same input sentence sequence has different representations in the input layer.

[0063] The specific construction form is as follows Figure 3 As shown in the figure, the Transformer model is pre-trained by the proposed Entity-Mask-Bert and is used to initialize Insert-Bert.

[0064] The Insert-BERT constructed in this paper has the following two advantages:

[0065] 1) Only position symbols are added to the input layer of the input sentence sequence, without changing the internal structure of the encoding layer Transformer and the overall model framework, which ensures that the proposed Entity-Insert-BERT structure can be well migrated to the downstream task of drug relationship classification;

[0066] 2) The insertion of the new position symbol only changes the relative position between the original input words. Since the Transformer's attention mechanism is insensitive to position changes, this makes the improved pre-training strategy and fine-tuning relatively consistent.

[0067] The main encoding unit of the constructed Insert-BERT model is Transformer. The parameters of this encoding unit are initialized with the Transformer parameters obtained by Entity-MASK-Bert pre-training. In the training phase of the model, it is assumed that the output of the [CLS] node is represented as r∈R k ;

[0068] S50, define the transfer matrix W∈R k×c Realize the mapping from output representation space to category probability space:

[0069] o=rw;

[0070] Where o=(o1,···,o j ,···,o c )∈R c represents the unnormalized probability of each category;

[0071] k and c represent the output vector dimension and the number of categories of drug entity relations, respectively.

[0072] S51. Use the Softmax function to transform the unnormalized probability space o into the normalized probability space s, as shown in the following formula:

[0073]

[0074] in, represents the p-th drug entity pair in the l-th input sentence;

[0075] express Belongs to category c j probability;

[0076] S52: Complete Insert-BERT model training using the formula in step S51 as the optimization target:

[0077]

[0078] Where L represents the total number of samples in the training set;

[0079] n l Indicates the number of drug entity pairs contained in the lth sample;

[0080] They represent the true value and predicted value of the p-th drug entity pair in the l-th sample respectively.

[0081] During the training and fine-tuning of the Insert-BERT model, batch-size and learning rate were set to 16 and 2e-5 respectively. In terms of optimization method, an optimization strategy based on warmup + triangular learning rate was adopted, that is, in the initial stage of the model, a smaller learning rate was used to ensure that the model parameters were updated in a smaller range to avoid model oscillation; in the later stage of training, as the model gradually converged, the learning rate was continuously reduced to fine-tune the model.

[0082] In addition, for the newly added positional symbols START1, END1, STRAT2, and END2, if their word vector representations are added as additional parameters to insert-BERT for training, the word vector size of the pre-trained BERT will inevitably be destroyed. Given that BERT reserves some word vector units specifically for new symbols, a compromise solution is adopted without increasing the size of BERT word vectors, and the newly added positional symbols are uniformly mapped to the [unseen] word in the BERT vocabulary. Based on the Entity-mask-BERT pre-training parameters, the proposed Insert-BERT model training will reach convergence in about 20 times.

[0083] Example 2

[0084] In order to better apply the BERT model to the field of drug relationship extraction, the pre-trained model of Google BERT is used as the basis, and the summary data in the PubMed dataset is selected as the pre-training corpus of Entity-Mask-Bert. The DDI-Extraction 2013 dataset provides a relatively large annotated corpus for drug relationship extraction. This dataset annotates drug entity pairs into five drug relationship types: Advice, Effect, Mechanism, Int, and Negative (indicating that there is no relationship between the two entities). Therefore, DDI is essentially a 5-label multi-classification task. A brief description of each label is as follows:

[0085] 1) Advice: Describe the relevant opinions on the simultaneous use of two drugs, such as: Interaction may be expected, and UROXATRAL should not be used in combination with other alpha-blockers;

[0086] 2) effect: describes the interaction of drug effects, such as: Methionine may protect against theototoxic effects of gentamicin;

[0087] 3) Mechanism: Describe the pharmacokinetic mechanism, e.g. Grepafloxacin, like other quinolones, may inhibit the metabolism of caffeine and theobromine;

[0088] 4) int: describes drug interactions for which no information is available, such as: The interaction of omeprazole and ketoconazole has been established;

[0089] 5) negative: Although there is co-occurrence between two entity pairs, there is no relationship between them, for example: Concomitantly given thiazide diuretics did not interfere with the absorption of a tablet of digoxin.

[0090]

[0091] After preprocessing and filtering rules

[0092]

[0093] Table 1 Data statistics before and after preprocessing of DrugBank and MedLine

[0094] The training set and test set of DDI-Extraction2013 are mainly composed of two aspects: DrugBank and MedLine. In order to eliminate the influence of too many negative samples and improve the model effect, in the data processing stage, the text preprocessing method proposed by Quan is used to filter out some negative sample pairs. Its detailed description is shown in Table 1.

[0095] In Table 1, the upper part represents the statistics of data without preprocessing, and the lower part represents the statistics of data after preprocessing. In addition, in Table 1, Abstract represents summary data, Positive represents data with four relationships of Advise, Effect, Mechanism and Int, and Negative represents data without any relationship. As can be seen from Table 1, through text processing, the negative samples of the training set can be reduced from 22118 to 14445, and the negative samples of the test set can be reduced from 4367 to 2819. This preprocessing operation can save training time and reduce the imbalance of classification.

[0096] The proposed Insert-Bert model is compared with other related works on the DDI-Extraction 2013 dataset for drug relationship classification. In addition to comparing the multi-classification task effects of Advise, Effect, Mechanism, and Int, the binary classification task effects of drug relationship detection are further compared, that is, DEC is introduced to describe whether there is a relationship between drugs, without distinguishing what kind of drug relationship there is.

[0097] Table 2 Comparison with traditional statistical machine learning-based drug relationship classification methods

[0098] Method Type Advise Effect Mechanism Int DEC F-value Kim 72.5 66.2 69.3 48.3 77.5 67.0 FBK-irst 69.2 62.8 67.9 54.0 80.0 65.1 WBI 63.2 61.0 61.8 51.0 75.9 60.9 UTurku 63.0 60.0 58.2 50.7 69.6 59.4 Insert-BERT 83.8 82.3 84.5 58.0 88.0 81.3

[0099] First, the proposed method is compared with the drug relationship classification method based on traditional statistical machine learning, mainly including three baseline models FBK-irst, WBI and UTurku and Kim method, and the comparison results are shown in Table 2. The above four methods all use SVM classifiers to classify the input drug entity pairs, but there are large differences in the selection of models, strategies and features. FBK and Kim methods first use a binary classification model to realize drug relationship detection (i.e., determine whether a pair of entities have a relationship), and then use a multi-classification model to complete the multi-classification task of drug relationships. In the selection of multi-classification strategies based on the SVM model, FBK adopts a one-against-all strategy, while Kim chooses a one-against-one strategy. Unlike Kim and FBK, WBI and UTurku directly use a multi-classification-SVM to implement all tasks.

[0100] Table 2 shows that compared with the drug relationship classification method based on traditional statistical machine learning, the proposed method has obvious improvements in Advise, Effect, Mechanism, Int, DEC and F value. Taking the Kim et al. system, which is currently effective in traditional machine learning, as an example, the F value of the Insert-BERT model proposed in this paper has increased from 67.0% to 81.3%; in terms of single category, the classification effect has also been significantly improved, and the F value of each category has increased by more than 10%. In addition, features still play a relatively important role in the above four methods. Some commonly used features, such as n-gram information, word pair information, part of speech information, etc. are introduced into the model; at the same time, some high-order features, such as syntactic tree information, dependency syntactic information, syntactic path information, etc. are also integrated into the model. These traditional drug relationship classification methods that integrate artificially extracted features have achieved relatively good results to a certain extent, but on the one hand, such methods are difficult to reuse, and on the other hand, the cascade of artificially extracted features is prone to increase error transmission. These factors restrict the application of traditional drug relationship classification methods and the improvement of drug relationship extraction effects.

[0101] Table 3 Comparison with deep learning-based drug relationship classification methods

[0102]

[0103]

[0104] Different from the classification method based on traditional statistical machine learning, the drug relationship classification method based on deep learning directly uses an end-to-end approach to extract drug relationships. In order to analyze the advantages of the drug relationship classification method based on deep learning, this paper further compares the proposed method with the drug relationship classification models related to CNN and RNN. In the CNN-related models, most models first convert the input text into word vector representation, then use the window convolution of CNN to achieve sequence modeling and entity pair feature extraction, and finally connect an external classifier to complete the entity relationship classification. Unlike CNN, the RNN-related model directly models the sequence. Considering that a small amount of drug relationship supervision data is difficult to guarantee the model classification effect, this paper adopts a pre-training strategy based on drug entity word masking and a fine-tuning scheme based on the Insert-BERT model to achieve drug relationship classification. Table 3 shows that compared with the drug relationship classification methods related to CNN and RNN, the overall and single-category F values ​​of the proposed method are significantly improved. Table 3 also shows that the classification effect based on the RNN sequence modeling method is slightly better than the CNN-based method, and the effect of the deep learning-based method is also far better than the SVM-related model.

[0105] The experimental results in Tables 2 and 3 show that the model effect of Insert-BERT far exceeds that of CNN, RNN and SVM related models. The method proposed in this paper has achieved good classification results on the four types of relationships: Advise, Effect, Mechanism and Int. Except for the Int type, the F value of other types has increased by more than 5%. In Tables 2 and 3, the classification effect of Int of each method is poor. The main reason is that the training and test data of Int are relatively small (less than 100), and the model is difficult to fit, resulting in low classification effect. In addition, in terms of F value, the Insert-BERT model proposed in this paper has achieved F = 0.81, which is far better than the MCNN and TM-RNN related models, which further illustrates the effectiveness of the proposed pre-training + fine-tuning scheme. In addition, in the binary DEC task, Insert-BERT also achieved the best results, and its F value F exceeded the MCNN method by 9%. Experiments show that the proposed method has comprehensive improvements in both the multi-classification task of drug relationship extraction and the binary classification task of drug relationship detection.

[0106] Table 4 Comparison of model effects of Google Bert initialization and Entity-mask-Bert initialization

[0107] Model structure F-value Insert-BERT (Google Bert Initialization) 79.0 Insert-BERT (Entity-Mask-Bert Initialization) 81.3

[0108] Table 5 Effect of preprocessing rules on the effects of different methods

[0109] method No pre-processing Preprocessing F-value changes CNN 65.0 69.7 4.7 MCNN 67.8 70.2 2.4 SCNN 64.5 68.4 3.9 RNN 67.2 69.4 2.2 TM-RNN 70.8 72.4 1.6 Insert-BERT 81.0 81.3 0.3

[0110] In order to further verify the effectiveness of the Entity-Mask-Bert and pre-training strategy proposed in this paper, the model effects of Insert-BERT under Google Bert initialization and Entity-mask-BERT initialization are compared. The proposed Entity-Mask-BERT has the same parameters as Google Bert, and both use a 12-layer-768 structure.

[0111] In order to further explore the deep features learned by Insert-BERT and the weights of the words in the sentence that affect them, this paper conducts a visual analysis of the [CLS] node of BERT. Specifically, for each [CLS] node of BERT, this paper calculates the weights of other words and the current [CLS] node and displays them in a visual way, such as Figure 4 For the convenience of analysis and visualization, Figure 4A represents entity 1, B represents entity 2, and S1, E1, S2 and E2 are mapped to the special symbols [unseen1], [unseen2], [unseen3], [unseen4] in the Bert vocabulary respectively. Figure 4 In the figure, the thickness of the line between the [CLS] node and other words on the left represents the weight relationship between the [CLS] node and other nodes. Through visual weight analysis, we can draw the following qualitative conclusions: 1) In shallow feature learning, Insert-BERT mainly focuses on learning some features at the overall level of the sentence. For example, in layer0 to layer7, most of the weights are mainly concentrated on the two nodes [CLS] and [SEP]; 2) In slightly higher-level feature learning, Insert-BERT begins to pay attention to some detailed features that are not at the overall level. Taking layer8-layer11 as an example, most of the weights are concentrated on nodes other than [CLS] and [SEP]; 3) There are differences in the features focused on by each layer. For example, in layer6 and layer7, Insert-BERT focuses on some entity boundary features; while in layer4 and layer5, Insert-BERT pays more attention to some keywords that can contain entity relationships, such as "increases". From these qualitative analyses, it can be seen that the proposed Insert-BERT can extract semantic information of different levels and granularities in continuous space.

[0112] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A drug relationship extraction method based on drug entity word masking and Insert-BERT structure, characterized in that: The following steps are involved: S1. In the word sequence masking stage of the input layer, drug entity detection is achieved by string matching based on the constructed drug name entity library; S2. In the input stage, only one of the drug entities is randomly selected for masking, and for the remaining non-entity words in the sentence, the conventional BERT masking strategy is followed; S3. Based on the pre-trained Entity-Mask-BERT, the drug relationship classification of the Insert-BERT structure is constructed by introducing a mark to achieve the purpose of extracting different drug entity relationships in the same input sentence sequence; S4. To distinguish the problem that the same input sentence contains different entity combinations, four new symbols, START1, END1, STRAT2 and END2, are added to judge the entity combination [e n , e m ], so that the same input sentence sequence has different representations in the input layer; Specifically, determine the entity combination [e n , e m The relationship between ] is: In e n Insert symbols START1 and END1 before and after to mark the location of the first entity; In e m Insert symbols START2 and END2 before and after to mark the location of the second entity; Similarly, entity combination [e n , e v ] and [e m , e v ] are similarly operated, an input sentence containing N entities will generate types of input sequences, so that the same input sentence sequence has different representations in the input layer; S5. In the early stage of Insert-BERT model training, an optimization strategy based on warmup + triangular learning rate is adopted. In the later stage of training, as the model gradually converges, the learning rate is continuously reduced to fine-tune the model; Specifically, during the training phase of the model, assume that the output of the [CLS] node is represented as r∈R k ; S50, define the transfer matrix W∈R k×c Realize the mapping from output representation space to category probability space: o=rw; Where o=(o1,···,o j ,···,o c )∈R c represents the unnormalized probability of each category; k and c represent the output vector dimension and the number of categories of drug entity relations, respectively; S51. Use the Softmax function to transform the unnormalized probability space o into the normalized probability space s, as shown in the following formula: in, represents the p-th drug entity pair in the l-th input sentence; express Belongs to category c j probability; S52: Complete Insert-BERT model training using the formula in step S51 as the optimization target: Where L represents the total number of samples in the training set; n l Indicates the number of drug entity pairs contained in the lth sample; They represent the true value and predicted value of the p-th drug entity pair in the l-th sample respectively.

2. The drug relationship extraction method based on drug entity word masking and Insert-BERT structure according to claim 1 is characterized in that: In step S2, when there are multiple entities in the input sentence, the strategy of masking one of the drug entities and using other drug entities to predict the current masked entity can achieve unsupervised high-frequency drug entity relationship modeling and complete the pre-training task based on massive data describing drug mechanisms.

3. The drug relationship extraction method based on drug entity word masking and Insert-BERT structure according to claim 1 is characterized in that: In step S3, for the input sentence sequence X = [x1, x2, e3, x4, e5... 10 ,···x M ]; Where M represents the total number of words contained in the input sentence; e3, e5 and e 10 Represent 3 drug entities respectively; x1,x2,x4,··x M represents a non-drug entity; The downstream classification task needs to judge [e3, e5], [e3, e 10 ] and [e3,e 10 ], the relationship between these three drug entity combinations, and the entity combinations share the same input sequence X.

4. The drug relationship extraction method based on drug entity word masking and Insert-BERT structure according to claim 1 is characterized in that: In step S5, the Insert-BERT model is constructed, whose main encoding unit is Transformer, and the parameters of the encoding unit are initialized with the Transformer parameters obtained by Entity-MASK-Bert pre-training.

5. The drug relationship extraction method based on drug entity word masking and Insert-BERT structure according to claim 1 is characterized in that: During the training and fine-tuning of the Insert-BERT model, the batch-size and learning rate were set to 16 and 2e-5, respectively.

Images