Protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks
By using Gaussian probability distribution and multi-task learning in protein interaction relationship extraction, the model's processing ability of input text and target entities is improved, the problem of low accuracy in the existing technology is solved, and more accurate protein interactions are achieved Relationship extraction.
Patent Information
- Application Number
- CN202111627067.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-28
AI Technical Summary
Existing deep learning methods fail to effectively utilize the problems of long input text and scattered distribution of target entities in protein interaction relationship extraction, resulting in low model extraction accuracy.
The Gaussian probability distribution is introduced to enhance the weight of the target entity and its adjacent words, and combined with multi-task learning, the feature extraction and relationship prediction capabilities are improved through BioBERT pre-trained models and auxiliary tasks.
The model's extraction accuracy of protein interaction relationships is improved, the input text is too long and the target entity distribution is scattered, the attention to semantic information is enhanced, and the model performance is improved.
Smart Images

Figure CN114373512B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks. Background Art
[0002] In order to help health professionals and researchers conduct precision medicine, studying protein-protein interactions affected by gene mutations is of great clinical significance. Biomedical literature contains a large number of protein-protein interactions affected by gene mutations. Automatically extracting protein interaction relationships described in biomedical literature is an important step towards precision medicine.
[0003] Protein interaction relationship extraction methods can be mainly divided into the following three categories: methods based on handwritten templates, methods based on statistical machine learning, and methods based on neural networks. Template-based methods have high accuracy and simple methods, but they have low recall rates and templates in specific fields need to be constructed by experts, which is difficult, time-consuming and labor-intensive. Methods based on statistical machine learning include feature-based methods and kernel-based methods. They do not require experts to build templates, which solves some of the shortcomings of template-based methods, but require manual design of features or kernel functions to enhance performance, which is still time-consuming and labor-intensive. Deep learning technology uses neural networks to extract feature information, thereby reducing the reliance on feature engineering in statistical machine learning methods. Therefore, most of the existing methods currently use deep learning to extract protein interaction relationships.
[0004] However, the current deep learning methods only improve model performance through general techniques, without considering the characteristics of the protein interaction relationship extraction task itself: the input text is long, the target protein entities are scattered in the data set, and the accuracy of the model in extracting protein interaction relationships is not high. Summary of the invention
[0005] In view of this, the present invention provides a protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks. By introducing Gaussian probability distribution, the weights of the target protein entity and its adjacent words are enhanced, thereby solving the problems of long input text and scattered distribution of target entities in the biomedical field, and enhancing the pre-trained model's attention to the semantic information of the target entity and its adjacent words during the feature extraction process; at the same time, multi-task learning is used to further improve the model's ability to extract protein interaction relationships.
[0006] To this end, the present invention provides the following technical solutions:
[0007] The present invention provides a protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks, and the specific steps are as follows:
[0008] S1. Preprocess the corpus of the protein interaction relationship extraction task to obtain the word segmentation results;
[0009] S2, input the word segmentation results obtained in S1 into the BioBERT pre-trained model to generate high-quality contextual representations of words and obtain the feature vector of the entire protein interaction relationship document;
[0010] S3, sending the feature vector obtained in S2 to the fully connected layer of the auxiliary task for document classification prediction; the auxiliary task is a document classification task, and the document classification task is used to determine relevant literature describing gene mutations that affect protein-protein interactions;
[0011] S4, construct a Gaussian probability list according to the position of the target entity pair in the word segmentation result obtained in S1, assign different probability values to different token positions, multiply the Gaussian probability list storing the probability values and the feature vector of the document to obtain the corresponding enhanced representation of the target entity;
[0012] S5. Connect the document representation with information added by the auxiliary task in S3 with the enhanced representation of the target entity obtained in S4 to form a long vector, and extract key features through a fully connected layer. After extracting the key features through the fully connected layer, input the results into the fully connected neural network again to predict the relationship between the two biomedical entities in the sentence, and finally obtain the probability distribution of the protein interaction relationship, so as to extract the protein interaction relationship.
[0013] Furthermore, S1 specifically includes:
[0014] Use GNormPlus tool to annotate protein interaction relationship entities in the corpus;
[0015] Use data preprocessing procedures to convert raw corpus into input that can be accepted by the deep learning network model;
[0016] The BioBERT tokenizer is used to tokenize the input and construct the document sequence into the form of “[CLS]Title+Abstract[SEP]”.
[0017] Furthermore, the BioBERT pre-trained model consists of a multi-layer bidirectional transformer encoder.
[0018] Furthermore, the final hidden states of all tokens are used in the BioBERT pre-trained model to represent the entire sequence representation.
[0019] In another aspect, the present invention provides a protein interaction relationship extraction device based on Gaussian enhancement and auxiliary tasks, the device comprising:
[0020] A preprocessing unit is used to preprocess the corpus of the protein interaction relationship extraction task to obtain the word segmentation results;
[0021] A feature vector unit, used to input the word segmentation result obtained by the preprocessing unit into the BioBERT pre-training model, generate a high-quality contextual representation of the word, and obtain a feature vector of the entire protein interaction relationship document;
[0022] An auxiliary task unit, used for sending the feature vector obtained by the feature vector unit to the fully connected layer of the auxiliary task for document classification prediction; the auxiliary task is a document classification task, and the document classification task is used to determine relevant literature describing gene mutations that affect protein-protein interactions;
[0023] A Gaussian enhancement unit is used to construct a Gaussian probability list according to the position of the target entity pair in the word segmentation result obtained by the preprocessing unit, assign different probability values to different token positions, and multiply the Gaussian probability list storing the probability values with the feature vector of the document to obtain a corresponding enhanced representation of the target entity;
[0024] The protein interaction relationship extraction unit is used to connect the document representation with information added by the auxiliary task in the auxiliary task unit with the enhanced representation of the target entity obtained in the Gaussian enhancement unit to form a long vector, and extract key features through a fully connected layer; after the key features are extracted through the fully connected layer, the results are input into the fully connected neural network again to predict the relationship between the two biomedical entities in the sentence, and finally the probability distribution of the protein interaction relationship is obtained, so as to extract the protein interaction relationship.
[0025] On the other hand, the present invention also provides a computer-readable storage medium, characterized in that a computer instruction set is stored in the computer-readable storage medium, and when the computer instruction set is executed by a processor, the above-mentioned protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks is implemented.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1) Use the pre-trained model BioBERT to generate high-quality contextual representations of words, thereby representing words with word vectors that are closer to actual semantics, solving the polysemy problem of words in biomedical texts;
[0028] 2) Based on BioBERT, Gaussian probability distribution is used to generate enhanced representations of target entities, which increases the weights of target protein entities and their neighboring words. This effectively solves the problem of too long input text and scattered distribution of target entities in the protein interaction relationship extraction task, and improves the performance of the model.
[0029] 3) Using document classification as an auxiliary task increases the information of document representation, improves the representation of documents obtained from the model, and implicitly increases the data information. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0031] Figure 1 Schematic diagram of a process for extracting protein interaction relationships based on Gaussian enhancement and auxiliary tasks in an embodiment of the present invention;
[0032] Figure 2 This is a schematic diagram of a protein interaction relationship extraction model framework in an embodiment of the present invention;
[0033] Figure 3 Schematic diagram of Gaussian probability distribution in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0036] The present invention provides a protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks. First, the GNormPlus tool is used to annotate the protein entities in the protein interaction relationship task of the corpus, and then the data preprocessing program is used to convert the original corpus into an input acceptable to the deep learning network model, and then the BioBERT word segmenter is used to segment the input; then the segmented result is input into the BioBERT pre-trained model to generate a high-quality context representation of the word, and the feature vector of the entire protein interaction relationship document is obtained; then the model includes two downstream tasks: Task 1 is a document classification task: the feature vector of the protein interaction relationship document obtained is sent to the fully connected layer of the document classification for document classification prediction, and the hidden state of the shared BioBERT layer is improved in the process of performing the document classification task, implicitly increasing the data information represented by the document; Task 2 is a relationship extraction task: first, the Gaussian probability distribution is used to increase the weight of the target protein entity and its neighboring words to obtain an enhanced representation of the target entity; then the enhanced representation of the two target entities enhanced by Gaussian enhancement and the feature vector of the protein interaction relationship document are spliced into a long vector, which is sent to the fully connected layer after dropout, and finally the output layer outputs the probability distribution of the relationship.
[0037] like Figure 1 As shown, a protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks in an embodiment of the present invention uses a protein interaction relationship extraction model based on a deep neural network to extract protein interaction relationships, such as Figure 2 As shown, the protein interaction relationship extraction model includes an input layer, a feature extraction layer, a document representation layer, an enhanced representation layer and a fusion classification layer, wherein the enhanced representation layer includes an auxiliary task module and a Gaussian enhancement module. The method includes the following specific steps:
[0038] S1. Preprocess the corpus for protein interaction extraction task at the input layer:
[0039] First, the GNormPlus tool was used to annotate the protein-protein interaction relationship entities in the corpus. Secondly, the data preprocessing program was used to convert the original corpus into input that could be accepted by the deep learning network model. The BioBERT tokenizer was then used to segment the input and construct the document sequence in the form of "[CLS]Title+Abstract[SEP]".
[0040] S2, in the feature extraction layer, the input is sent to the BioBERT pre-trained model to extract features:
[0041] The S1 preprocessed corpus is input into the BioBERT pre-trained model to extract the features of the entire protein interaction relationship text document and output the feature vector of the document.
[0042] BioBERT consists of a multi-layer bidirectional transformer encoder. For an input sentence consisting of multiple words, BioBERT uses a wordpiece tagger to segment the input sentence into one or more tokens. Tokens are language units such as words, punctuation, numbers, alphanumerics, etc. For example, "Title" is marked as the token "CLS" and "Abstract" is marked as the token "SEP", which alleviates the vocabulary problem to a certain extent. The first word of each sentence is always a special token "CLS", and the final hidden state corresponding to this token is used as the sequence representation for the classification task. BioBERT uses the final hidden state of the token "CLS" to represent the entire sequence representation. However, the final hidden states of the remaining tokens also contain valuable information. Therefore, in an embodiment of the present invention, all final hidden states of all tokens are retained in the sequence.
[0043] S3. Use document classification as an auxiliary task in the enhanced representation layer to increase the information of document representation:
[0044] The feature vector of the document obtained in S2 is sent to the fully connected layer of document classification for document classification prediction.
[0045] In this process, the representation of the document obtained from the model is improved, and the data information is implicitly increased. Intuitively, it is reasonable to assume that the auxiliary task will improve the representation of the text obtained from the model and implicitly increase the data. Therefore, a document classification task is added as an auxiliary task in an embodiment of the present invention to improve the performance of relationship extraction. The purpose of the classification task is to identify relevant literature describing gene mutations that affect protein-protein interactions. The classification task can be considered as an upstream task of relationship extraction, because the dataset of the relationship extraction task is a subset of related documents that have been manually annotated with related gene pairs. In this architecture, parameter sharing is used for multi-task learning, and all tasks share BioBERT as an embedding layer.
[0046] S4. Use Gaussian probability distribution to form an enhanced representation of the target entity in the enhanced representation layer:
[0047] A Gaussian probability list is constructed according to the position of the target entity pair in the result after S1 word segmentation, and different probability values are assigned to different token positions. The Gaussian probability list storing the probability values is multiplied by the feature vector of the document to obtain the corresponding enhanced representation of the target entity.
[0048] In the embodiment of the present invention, Gaussian probability distribution is introduced to enhance the weights of the target protein entity and its adjacent words, so that the model can learn the local structure of the instance.
[0049] The Gaussian probability density function is as follows:
[0050]
[0051] The Gaussian cumulative distribution function is as follows:
[0052]
[0053] The Gaussian probability distribution function is as follows:
[0054] P(x)=F(x)-F(xw) (3)
[0055] Where x is the actual number, μ is the mean of the distribution, δ is the standard deviation, and w is the token window; in the embodiment of the present invention, the token window w is set to 1 to represent the distance of each token itself, and the optimal values of μ and δ are set to 0 and 2.5 respectively (in order to represent the distance of each token itself, the token window w is set to 1, and the optimal μ and δ values are set to 0 and 2.5 respectively). First, the tokens are numbered according to the two target entities in the example, so as to obtain a list of relative distances between the two tokens ( Figure 3 The probability of each token in the instance is then calculated using the Gaussian probability distribution function. These token probabilities are then element-wise multiplied with the instance token representation to obtain the first and second target entity enhanced representations, respectively, as follows:
[0056]
[0057]
[0058] Where N is the sequence length, x1 and x2 are token numbers, Rtar1 and Rtar2 represent the first and second target entity enhanced representations, respectively;
[0059] S5. Vector fusion and protein interaction prediction:
[0060] The document representation with information added by the auxiliary task in S3 is connected with the enhanced representation of the target entity obtained in S4 to form a long vector, and then the key features are extracted through a fully connected layer. After the key features are extracted through the fully connected layer, the results are input into the fully connected neural network again to predict the relationship between the two biomedical entities in the sentence, and finally the probability distribution of the protein interaction relationship is obtained, thereby extracting the protein interaction relationship.
[0061] In the fusion classification layer, the related representations are concatenated into a fused representation h(h = [R ins ; R tar1 ; R tar2]). A fully connected neural network is then used to learn the representation h; finally, a Softmax function is used to calculate the probability type r of the protein interaction relationship:
[0062] P(r|h)=softmax(W0h+b0) (6)
[0063] Where W o and b o are weight parameters and bias parameters. In addition, the model proposed in the present invention uses the cross entropy function as the loss function and uses ADAM to optimize the parameters.
[0064] In the protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks provided in the embodiment of the present invention, Gaussian probability distribution and auxiliary tasks enhance the feature representation capability of the input document from two different perspectives, thereby better extracting the relationship between entities. This method solves the problems of too long input text and scattered distribution of target protein entities in the data set in the current protein interaction relationship extraction, which not only improves the accuracy of the performance on the task, but also has reference significance for other biomedical relationship extraction tasks with similar problems.
[0065] Corresponding to the protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks in the present invention, the present invention also provides a protein interaction relationship extraction device based on Gaussian enhancement and auxiliary tasks, the device comprising:
[0066] A preprocessing unit is used to preprocess the corpus of the protein interaction relationship extraction task to obtain the word segmentation results;
[0067] A feature vector unit, used to input the word segmentation result obtained by the preprocessing unit into the BioBERT pre-training model, generate a high-quality contextual representation of the word, and obtain a feature vector of the entire protein interaction relationship document;
[0068] An auxiliary task unit, used for sending the feature vector obtained by the feature vector unit to the fully connected layer of the auxiliary task for document classification prediction; the auxiliary task is a document classification task, and the document classification task is used to determine relevant literature describing gene mutations that affect protein-protein interactions;
[0069] A Gaussian enhancement unit is used to construct a Gaussian probability list according to the position of the target entity pair in the word segmentation result obtained by the preprocessing unit, assign different probability values to different token positions, and multiply the Gaussian probability list storing the probability values with the feature vector of the document to obtain a corresponding enhanced representation of the target entity;
[0070] The protein interaction relationship extraction unit is used to connect the document representation with information added by the auxiliary task in the auxiliary task unit with the enhanced representation of the target entity obtained in the Gaussian enhancement unit to form a long vector, and extract key features through a fully connected layer; after the key features are extracted through the fully connected layer, the results are input into the fully connected neural network again to predict the relationship between the two biomedical entities in the sentence, and finally the probability distribution of the protein interaction relationship is obtained, so as to extract the protein interaction relationship.
[0071] For the protein interaction relationship extraction device based on Gaussian enhancement and auxiliary tasks in the embodiment of the present invention, since it corresponds to the protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks in the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks in the above embodiment, which will not be described in detail here.
[0072] An embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, the protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks as provided in any of the above embodiments is implemented.
[0073] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0075] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks, characterized in that: The method comprises: S1. Preprocess the corpus of the protein interaction relationship extraction task to obtain the word segmentation results; S2, input the word segmentation results obtained in S1 into the BioBERT pre-trained model to generate high-quality contextual representations of words and obtain the feature vector of the entire protein interaction relationship document; S3, sending the feature vector obtained in S2 to the fully connected layer of the auxiliary task for document classification prediction; the auxiliary task is a document classification task, and the document classification task is used to determine relevant literature describing gene mutations that affect protein-protein interactions; S4, construct a Gaussian probability list according to the position of the target entity pair in the word segmentation result obtained in S1, assign different probability values to different token positions, multiply the Gaussian probability list storing the probability values and the feature vector of the document to obtain the corresponding enhanced representation of the target entity; S5, the Gaussian probability distribution and the auxiliary task represent the features of the input document from two different perspectives; the document representation with information added by the auxiliary task in S3 is connected with the enhanced representation of the target entity obtained in S4 to form a long vector, and key features are extracted through a fully connected layer; after the key features are extracted through the fully connected layer, the results are input into the fully connected neural network again to predict the relationship between the two biomedical entities in the sentence, and finally the probability distribution of the protein interaction relationship is obtained, thereby extracting the protein interaction relationship.
2. A protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks according to claim 1, characterized in that: S1 specifically includes: Use GNormPlus tool to annotate protein interaction relationship entities in the corpus; Use data preprocessing procedures to convert raw corpus into input that can be accepted by the deep learning network model; Use the BioBERT tokenizer to tokenize the input and construct the document sequence into the form of "[CLS]Title+Abstract[SEP]".
3. A protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks according to claim 2, characterized in that: The BioBERT pre-trained model consists of a multi-layer bidirectional transformer encoder.
4. A protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks according to claim 3, characterized in that: The final hidden state of all tokens is used in the BioBERT pre-trained model to represent the entire sequence representation.
5. A protein interaction relationship extraction device based on Gaussian enhancement and auxiliary tasks, characterized in that: The device comprises: A preprocessing unit is used to preprocess the corpus of the protein interaction relationship extraction task to obtain the word segmentation results; A feature vector unit, used to input the word segmentation result obtained by the preprocessing unit into the BioBERT pre-training model, generate a high-quality contextual representation of the word, and obtain a feature vector of the entire protein interaction relationship document; An auxiliary task unit, used for sending the feature vector obtained by the feature vector unit to the fully connected layer of the auxiliary task for document classification prediction; the auxiliary task is a document classification task, and the document classification task is used to determine relevant literature describing gene mutations that affect protein-protein interactions; A Gaussian enhancement unit is used to construct a Gaussian probability list according to the position of the target entity pair in the word segmentation result obtained by the preprocessing unit, assign different probability values to different token positions, and multiply the Gaussian probability list storing the probability values with the feature vector of the document to obtain a corresponding enhanced representation of the target entity; The protein interaction relationship extraction unit is used to connect the document representation with information added by the auxiliary task in the auxiliary task unit with the enhanced representation of the target entity obtained in the Gaussian enhancement unit to form a long vector, and extract key features through a fully connected layer; after the key features are extracted through the fully connected layer, the results are input into the fully connected neural network again to predict the relationship between the two biomedical entities in the sentence, and finally the probability distribution of the protein interaction relationship is obtained, so as to extract the protein interaction relationship.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer instruction set, and when the computer instruction set is executed by a processor, the protein interaction relationship extraction method based on Gaussian enhancement and auxiliary tasks as described in any one of claims 1 to 4 is implemented.
