A method for predicting compound properties based on multi-task deep learning model
Through the combination of multi-task deep learning model and GAT model, compound characteristics are extracted and the connection between tasks is modeled, the problem of insufficient performance of single-task model is solved, the compound properties prediction effect is improved, and the drug development cost is reduced.
Patent Information
- Application Number
- CN202111534259.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-15
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2041-12-15
AI Technical Summary
The prior art has limitations in the prediction of compound properties in drug research and development, and fails to fully utilize the data information of multiple tasks, resulting in the problem of degradation of model performance and high R&D costs and long cycles.
The multi-task deep learning model is adopted to extract compound characteristics through expert networks and gated networks, combine GAT models to design the overall network structure, and integrate shared and unique expert networks, and use multi-task weighted loss for joint training to improve the effectiveness of compound properties prediction.
It improves the effectiveness of the compound properties prediction task, can better extract the connection between molecular characteristics and modeling tasks, improves model performance and flexibility, and reduces drug research and development costs.
Smart Images
Figure CN114566230B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of methods for predicting the properties of compounds using a multi-task deep learning model, and specifically refers to a method for predicting the properties of compounds based on a multi-task deep learning model. Background Art
[0002] Compound property prediction plays an important role in drug development. In recent years, artificial intelligence has made many attempts and successful cases in this regard. Most of the research is to improve the prediction effect of a single property by continuously improving the single-task model.
[0003] With the diversity of diseases and the frequent emergence of drug resistance, the demand for drugs is increasing, but the research and development of new drugs has risks such as long research and development cycles, high costs and low success rates. Generally speaking, it takes billions of dollars and decades for an innovative drug to go from research and development to final market launch. Despite the high investment and long time, the success rate of small molecule drugs in the final market launch is still very low, and the risk of failure is high. How to reduce costs and speed up the development of new drugs has become a challenging and urgent issue in the industry. In order for a compound to become a drug that is eventually marketed, it often needs to meet many properties. Therefore, using artificial intelligence to predict the properties of compounds can greatly save the cost of drug research and development.
[0004] To use artificial intelligence to predict the properties of compounds, we first need to represent the compounds as data that can be processed by computers. At present, there are generally two ways to represent compounds, namely the smile sequence form and the two-dimensional molecular graph form. Then use traditional machine learning or deep learning to predict the properties of the corresponding compounds. It is equivalent to training a classification or regression model. For example: Newby et al. predicted the role of compound permeability and solubility in the oral absorption of drugs by building a decision tree model. Others use the naive Bayes model to predict the adverse reactions of drugs. In addition to traditional machine learning, the structure-based deep learning network AtomNet uses CNN to predict the biological activity of small molecules. In addition, due to the rapid development and effectiveness of graph models, more and more graph models are applied to compound property prediction tasks, such as GCN and GAT. Graph models extract two-dimensional structural information of graph nodes and edges. Compared with the smile sequence form, the extracted information is more sufficient and more effective predictions can be made.
[0005] First of all, traditional machine learning methods often require manual feature extraction, which is troublesome. And the feature extraction ability is often weaker than deep learning. Secondly, most of the current deep learning methods are implemented through single-task models. Compared with multi-task, the collected data is not fully utilized. For a batch of compounds, we may have multiple labels. If we use a single-task model for training, other label information will not be used. If we can use the information of other labels and model multiple tasks at the same time, the knowledge learned from multiple tasks can be shared, which may improve the model effect of a single task. The most commonly used structure for multi-task is Hard parameter sharing. This structure allows multiple tasks to share the underlying neural network. The advantage is that there are fewer parameters and a simpler model, but this may cause conflicts during model optimization due to the weak correlation between multiple tasks, resulting in performance degradation. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 It is a two-dimensional structure of a compound in a method for predicting compound properties based on a multi-task deep learning model of the present invention.
[0007] Figure 2 It is the overall structure of a multi-task model of a method for predicting compound properties based on a multi-task deep learning model of the present invention. Summary of the invention
[0008] The technical problem to be solved by the present invention is to overcome the above technical defects and provide a method for predicting the properties of compounds based on a multi-task deep learning model, which can use more data to improve the performance of a single-task model through multi-task learning. This architecture can extract molecular features well, and the connections and differences between modeling tasks make the model very flexible.
[0009] In order to solve the above technical problems, the present invention provides a technical solution: a method for predicting the properties of compounds based on a multi-task deep learning model, comprising the following steps:
[0010] Step 1: Collect a data set containing multiple task labels;
[0011] Step 2: Divide the data set;
[0012] Step 3: Select appropriate expert network to extract compound features;
[0013] Step 4: Multi-task model structure design;
[0014] Step 5: Train the model.
[0015] Preferably, in step 1, the data set includes the smile sequence form of the compound and the labels of multiple tasks. The label type can be a classification type or a regression type, and some samples are allowed to lack labels of some tasks, so that mask processing can be performed when calculating the loss.
[0016] Preferably, in step 2, the data set is divided into a training set, a validation set and a test set. The training set is used to train the model parameters, the validation set is used to select the best model parameters by early stopping to prevent overfitting, and the test set is used to evaluate the model effect.
[0017] Preferably, in step 3, the GAT model is selected as the expert network in the multi-task framework to perform feature extraction on the compound.
[0018] Preferably, the overall network structure in step 4 is mainly composed of three parts, an expert network, a gating network and a prediction network. The expert network is responsible for extracting the features of the input compound and can be divided into two categories according to the different extracted information. One category is shared by all tasks. This type of expert network is responsible for learning the common features between different tasks, responsible for modeling the connection between multiple tasks, and can use the training data of all tasks to update parameters. The other category is a task-specific expert network, which is responsible for learning the specific domain knowledge of each task and can only use the labeled data of the task for parameter update. The feature extraction representation of each task sample consists of two parts:
[0019]
[0020] in represents the output of the k-th task-specific expert network, It represents the output of the expert network shared by all tasks, and then the shared and unique expert networks are fused through a gating network to finally obtain the final representation of each task. Here, the gating network also uses a GAT model with fewer parameters to learn the weights of the fusion of different types of expert networks. The output of the gating network is two neurons, followed by a softmax layer. The specific representation is as follows:
[0021] w k (x) = Softmax(g(x))
[0022] Where g(x) represents the output of the gating network, so the final representation of each task is:
[0023] O k (x) = w k (x)S k (x)
[0024] Finally, the final feature representation of each task is connected to a prediction network to predict different tasks. The prediction network here uses multiple fully connected layers, and the number of neurons in the fully connected hidden layer is [150, 32, 1].
[0025] y k (x) = T k (O k (x)
[0026] Preferably, in step 5, the weighted loss of multiple tasks is used as the final loss to jointly train the shared part and the unique part of the model. The formula is expressed as:
[0027]
[0028] where w k represents the weight of the kth task, L k Represents the loss of the kth task. The loss of each task can be of different loss types, such as rmse or cross entropy. When the sample does not have a label for the corresponding task, the loss needs to be masked. Generally, the loss weight of each task is set to the same value, and different weights can also be tried.
[0029] The advantages of the present invention compared with the prior art are: firstly, the multi-task structure and the GAT model are combined to realize the prediction of compound properties, and the method can improve the effects of multiple compound property prediction tasks; a new multi-task model is proposed to simultaneously predict the properties of compounds, thereby improving the performance of the model. DETAILED DESCRIPTION
[0030] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0031] A method for predicting compound properties based on a multi-task deep learning model comprises the following steps:
[0032] Step 1: Collect a data set containing multiple task labels;
[0033] Step 2: Divide the data set;
[0034] Step 3: Select appropriate expert network to extract compound features;
[0035] Step 4: Multi-task model structure design;
[0036] Step 5: Train the model.
[0037] In step 1, the data set includes the smile sequence form of the compound and the labels of multiple tasks. The label type can be classification type or regression type, and some samples are allowed to lack the labels of some tasks, so that mask processing can be performed when calculating the loss.
[0038] In step 2, the data set is divided into training set, validation set and test set. The training set is used to train the model parameters, the validation set is used to select the best model parameters for early stopping to prevent overfitting, and the test set is used to evaluate the model effect.
[0039] Two methods are compared in step three. One is to first perform one-hot processing on the smile sequence to obtain a two-dimensional matrix of 0 / 1, similar to the field of image processing, and then use the CNN model to extract features from the two-dimensional matrix. The advantage of this method is that the model is simpler and can be trained faster on large data sets. The disadvantage is that it cannot capture the edge and node information in the two-dimensional structure of the compound, and there is a lot of information loss. The other is to first use the rdkit toolkit to extract the two-dimensional structural information of smile, that is, the edge and node information. Then use the GAT graph model to extract features. The advantage of this method is that the attention mechanism and graph model can fully extract the two-dimensional information of the compound. The disadvantage is that feature preprocessing is more troublesome and training is more time-consuming. After comprehensive consideration, the GAT model was selected as the expert network in the multi-task framework to extract features from compounds.
[0040] The overall network structure in step 4 is mainly composed of three parts: expert network, gating network and prediction network. The expert network is responsible for extracting the features of the input compound. It can be divided into two categories according to the extracted information. One category is shared by all tasks. This type of expert network is responsible for learning the common features between different tasks, responsible for modeling the connection between multiple tasks, and can use the training data of all tasks to update parameters. The other category is task-specific expert network, which is responsible for learning the specific domain knowledge of each task and can only use the labeled data of the task for parameter update. The feature extraction representation of each task sample consists of two parts:
[0041]
[0042] in represents the output of the k-th task-specific expert network, It represents the output of the expert network shared by all tasks, and then the shared and unique expert networks are fused through a gating network to finally obtain the final representation of each task. Here, the gating network also uses a GAT model with fewer parameters to learn the weights of the fusion of different types of expert networks. The output of the gating network is two neurons, followed by a softmax layer. The specific representation is as follows:
[0043] w k(x) = Softmax(g(x))
[0044] Where g(x) represents the output of the gating network, so the final representation of each task is:
[0045] O k (x) = w k (x)S k (x)
[0046] Finally, the final feature representation of each task is connected to a prediction network to predict different tasks. The prediction network here uses multiple fully connected layers, and the number of neurons in the fully connected hidden layer is [150, 32, 1].
[0047] y k (x) = T k (O k (x)
[0048] The above introduces the specific model structure. During training, the weighted loss of multiple tasks is used as the final loss to jointly train the shared part and the unique part of the model. The formula is expressed as:
[0049]
[0050] where w k represents the weight of the kth task, L k Represents the loss of the kth task. The loss of each task can be of different loss types, such as rmse or cross entropy. When the sample does not have a label for the corresponding task, the loss needs to be masked. Generally, the loss weight of each task is set to the same value, and different weights can also be tried.
[0051] When the present invention is implemented, the method designs a multi-task learning architecture, which can well extract features of compounds and model the correlation and differences of multiple tasks. In the modeling process, a shared expert network is designed to learn the connection between multiple tasks, and a unique expert network is designed for each task to model the unique information of the task. Then, by designing a gating network, the shared expert network and the unique expert network are fused to obtain the final representation of different tasks, and then a multi-task model is designed to predict different tasks, which can use more data to improve the performance of the single-task model through multi-task learning. This architecture can extract molecular features well, and the connection and difference between modeling tasks make the model very flexible.
[0052] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.
Claims
1. A method for predicting the properties of compounds based on a multi-task deep learning model, characterized in that: The following steps are involved: Step 1: Collect a data set containing multiple task labels. The data set includes the smile sequence form of the compound and the labels of multiple tasks. The label type can be classification type or regression type, and some samples are allowed to lack the labels of some tasks, so that mask processing can be performed when calculating the loss; Step 2: Divide the data set into training set, validation set and test set. The training set is used to train the model parameters, the validation set is used to select the best model parameters early to prevent overfitting, and the test set is used to evaluate the model effect. Step 3: Select the GAT model as the expert network in the multi-task framework to extract features of compounds; Step 4: Multi-task model structure design. The overall network structure is mainly composed of three parts: expert network, gating network and prediction network. The expert network is responsible for extracting the features of the input compound. It can be divided into two categories according to the extracted information. One is shared by all tasks. This type of expert network is responsible for learning the common features between different tasks, responsible for modeling the connection between multiple tasks, and can use the training data of all tasks to update parameters. The other is the task-specific expert network, which is responsible for learning the specific domain knowledge of each task and can only use the labeled data of the task to update parameters. The feature extraction representation of each task sample consists of two parts. composition: in represents the output of the k-th task-specific expert network, It represents the output of the shared expert network for all tasks, and then the shared and unique expert networks are fused through a gating network to finally obtain the final representation of each task. Here, the gating network also uses a GAT model with fewer parameters to learn the weights of the fusion of different types of expert networks. The output of the gating network is two neurons, followed by a softmax layer, which is specifically expressed as follows: w k (x)=Softmax(g(x)) Where g(x) represents the output of the gating network, so the final representation of each task is: O k (x)=w k (x)S k (x) Finally, the final feature representation of each task is connected to a prediction network to predict different tasks. The prediction network here uses multiple fully connected layers, and the number of neurons in the fully connected hidden layer is [150, 32, 1]; y k (x)=T k (O k (x)) Step 5: Train the model. During training, the weighted loss of multiple tasks is used as the final loss to jointly train the shared part and the unique part of the model. The formula is expressed as: where w k represents the weight of the kth task, L k It represents the loss of the kth task. The loss of each task is rmse or cross entropy. When the sample does not have the label of the corresponding task, the loss needs to be masked. Generally, the loss weight of each task is set to the same value, and different weights can also be tried.