Drug molecule generation optimization method based on deep reinforcement learning

Abstract

The invention discloses a drug molecule generation optimization method based on deep reinforcement learning, in order to overcome the problems of unbalanced distribution of existing drug and target point data and the like, a backbone network structure of an LSTM and transformer structure is established to solve the generation optimization of drug molecules, a local module is improved to reinforce performance, then reinforcement learning is added, and the drug molecule generation optimization method based on deep reinforcement learning is realized. The attributes of molecules generated in each round are processed to serve as loss of deep learning and award of reinforcement learning, and the model is dynamically corrected, so that high-quality drugs which can be effectively combined with target spots, are high in water solubility and moderate in stability and can be easily absorbed by human bodies can be designed. The measurement standard adopted by the attribute optimization task is chemical conversion property MMPs, and the hydrophobicity, solubility and clearance rate of molecules are used as evaluation indexes. By adopting the model provided by the invention, on the basis of ensuring the basic stability of a new drug, better target binding activity can be obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of drug research and development, and in particular relates to a drug molecule generation optimization method based on deep reinforcement learning. Background Art

[0002] Predicting the degree of binding between a drug and a target can quickly evaluate the performance of a new drug and accelerate the drug screening process. When it is determined that a new drug has a high degree of binding to a target, but there are key properties that need to be improved, it is necessary to make appropriate structural adjustments to the drug, change the atomic groups at key positions or change their positions, so that the properties of the drug are optimized without affecting the binding to the target. This requirement is called drug optimization. The task of drug optimization is to output a new molecule that is similar to the original molecule and has properties that meet the expected requirements given a drug molecule and a series of property change requirements.

[0003] At present, the selection of models mainly includes traditional sequence-to-sequence models and the latest transformer models that have achieved excellent results. CHEMBL is mostly used as the basic data set in data selection. In subsequent improvements, researchers proposed a multi-resolution and multi-level coupled encoder-decoder structure to solve the task of simultaneous optimization of multiple attributes. With the cross-application of language models and biomedicine, more drug molecule generation and optimization models based on language models have been proposed, which are collectively referred to as sequence-to-sequence Seq2Seq models. In drug optimization, reinforcement learning can also help the continuous generation process and produce more desired results. Studies have found that deep learning and reinforcement learning have something in common and can be effectively combined. Therefore, deep reinforcement networks can improve the learning ability of models and need further research. Summary of the invention

[0004] 1. Train the MolrProtTrans model with the drug molecule and protein binding data set to complete the classification and regression tasks:

[0005] Second, in the drug molecule optimization task, a sequence-to-sequence basic generative optimization model is trained through the performance change data set of the initial drug and the molecule after local changes of the initial drug;

[0006] 3. Add reinforcement learning strategy and dynamically train the model.

[0007] Beneficial effects of the present invention:

[0008] 1. A deep reinforcement network DQN is proposed. Based on reinforcement Q learning, it uses a deep network to capture the optimal strategy, thereby selecting actions that maximize the benefits of the recent reward function. The deep model used has also evolved from the simple fully connected network used in the TD strategy to a deeper and more complex convolutional network CNN. Then, in order to model multi-role learning tasks, various RNN or LSTM networks are used as deep learning architectures, combined with algorithms such as gradient strategy search, to improve learning capabilities.

[0009] 2. In the present invention, we make improvements based on the sequence-to-sequence generation model, select the classic LSTM model and the latest transformer model, add the EAGCN module as an auxiliary feature extractor in the vector generation part of the model encoder and decoder, and add an additional channel attention mechanism in the transformer model, so that the model can fully combine the structural information of the molecule and the proposed property optimization goals from multiple dimensions. Subsequently, the reinforcement learning method is applied to each round of training, which can significantly optimize the key properties round by round while optimizing the basic properties. After sufficient training, the molecules generated by the model can design more suitable local groups without destroying the original backbone carbon chain, optimize the desired properties, and meet the expectations of drug developers. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is the technical roadmap of the present invention.

[0011] Figure 2 This is the LSTM molecule generation model in the present invention.

[0012] Figure 3 This is a molecular generation model for the transformer architecture in the present invention.

[0013] Figure 4 This is a flowchart of deep reinforcement learning in the present invention.

[0014] Figure 5 This is the result of MolGen optimizing aspirin drug in the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0016] Step 1: First, train the Molr molecule representation model and ProtTrans protein model in MolrProtTrans.

[0017] The Transformer architecture is used to connect the feature vectors of drug molecules and proteins, the molecule vector is input into the Encoder module, and the protein vector is sent into the Decoder module.

[0018] Each molecule and protein data in the data set of the classification problem in the task is characterized by a designed representation model to obtain respective feature vectors, which are spliced ​​together, and the true value label of whether the data is actually combined is recorded to train the spliced ​​model.

[0019] Finally, MolrProtTrans is used for regression tasks.

[0020] Before training the overall network of MolrProtTrans, the Molr model is pre-trained to better represent molecular features.

[0021] The described ProtTrans model needs to be pre-trained to represent protein sequences.

[0022] The obtained feature vectors of drugs and targets are transmitted to the Transformer skeleton.

[0023] The described use of various machine learning models is docked into the feature vectors of molecules and proteins generated by MolrProtTrans.

[0024] The output interface of MolrProtTrans is modified to apply it to regression tasks.

[0025] The molecular representation models used include KCL, Grover and Mpnn, and the protein representation models include ESM and CNN.

[0026] Step 2: To solve the drug generation problem, a sequence model in the language model, called Seq2Seq, is used. The molecular property change dataset is used to train the molecular optimization model MolGen.

[0027] In order to effectively solve the translation problem of long sequence sentences, the present invention introduces an attention mechanism.

[0028] After each layer of the decoder receives a word and converts it into a decoded vector, it first receives the hidden vectors translated by the encoder for all words, and then calculates the distance between the word vector calculated by the decoder and the vector of each input word. This is used as the attention score of each input word for the output word, and the final vector of the decoder for the word is calculated.

[0029] The MolGen described is divided into two parts: an RNN encoder and an RNN decoder with an attention mechanism. The specific architecture of the RNN is LSTM.

[0030] The LSTM drug generation model is described as follows Figure 2 .

[0031] The RNN encoder converts the input word into a 128-dimensional vector first, and then passes it through three consecutive bidirectional LSTMs with an intermediate layer dimension of 256. In order to prevent overfitting, the drop probability value is set to 0.2.

[0032] When the model receives a word input, it first encodes it into the embedding space into a 128-dimensional vector representation, and then obtains the hidden representation of each word in sequence from front to back and from back to front, and its dimension becomes 256. The same is true for the Decoder.

[0033] The salient feature of the Seq2seq method is that it outputs in sequence. Each time the decoder outputs the next word vector, it needs to be supported by the previous word vector and all the embedding vectors of the encoder to obtain the result through the model. The result shape is [64, L, 64]. Among them, 64 is the batch size, L is the length of the molecular sequence, and 64 is the feature vector dimension. Finally, the 64-dimensional feature vector is mapped to an atom by comparing the atom dictionary to obtain the output molecular sequence of [64, L, 1].

[0034] The MolGen model is divided into two parts: an RNN encoder (Encoder) and an RNN decoder (Decoder) with an attention mechanism.

[0035] When the model receives a word input, it first encodes it into a vector, then sequentially obtains the hidden representation of each word from front to back and from back to front, and then adds the forward and reverse hidden layer vectors of each input word.

[0036] Each time the Decoder outputs the next word vector, it needs to be supported by the previous word vector and all the embedded vectors of the Encoder to obtain the result through the model.

[0037] In the decoding stage, the decoder outputs the next word vector based on the previous word vector and all the embedding vectors of the encoder.

[0038] The model described above uses batch training. Assume that there are N batches of training sequence data, xi and yi represent the source sequence and target sequence of the i-th batch respectively. θ is the total parameter set of the model. Then the training loss is:

[0039] Formula 1:

[0040] Among them, i traverses all data entries, t traverses all iterative batches of the model, and the model optimizes the t-1th batch of iterative results generated by the i-th batch of data to obtain the t-th round of optimization, and compares it with the target value to calculate the predicted loss.

[0041] The model is run on Windows system, the GPU used is NVIDIA GeForce, the default batch size is 64, the model parameters are updated using Adam optimizer, the initial learning rate is positioned at le-4, and there is a proportional decay of 0.8 every 10 rounds of training.

[0042] In order to achieve better learning results, the MolGen inserts the following two modules (EAGCN and ECA) into the backbone network to enhance the performance of the model in the molecular representation stage and the transmission stage.

[0043] The EAGCN is based on the CEM structure, which can assign different attention weights to different atomic nodes and put them into the representation of atoms. At the same time, it can add strong chemical bond consistency constraints, so that even in different molecules in the data set, the same type of bonds can still be assigned the same attribute encoding.

[0044] For a given initial feature map, the size is H*W*C.

[0045] The method first uses global average pooling GAP to average the H*W plane matrix of each channel to obtain an initial channel attention vector with a shape of 1*1*C. Compared with the fully connected layer processing used in the traditional SE module, ECA (Efficient Channel Attention Network) uses convolution with a kernel size of k to slide on the original features and adds padding to maintain the shape of the vector unchanged. K is an adaptive parameter, which eliminates the step of manual experiment to tune the parameters, and the model can more flexibly cope with features of various channel numbers. The network design based on the transformer structure in the present invention is as follows Figure 3 .

[0046] Step 3: Add reinforcement learning training methods to the molecular optimization model to help complete the drug molecule optimization task. Reinforcement learning can be described by a difference equation, x t+1 =f(x t ,a t ,e t ), x t is the current state of the system, a t is the operation selected in the current stage, e t is a random deviation.

[0047] In reinforcement learning, the reward can be calculated based on the current state and the action taken, denoted as R = R (xt, at). Since the action space of learning is continuous, the optimization target is not necessarily the reward Rt at a certain moment, but the sum of all Rt after a series of time and operations. The reinforcement learning formula is expressed as a system of equations:

[0048] Formula 2: x t+1 =f(x t , a t );

[0049] Formula 3:

[0050] Among them, x t+1 Is the model through x t and a t Calculated, t represents the round of reinforcement learning. Obj represents the goal of the reinforcement learning model, which is to make the reward value R of all rounds t The total is maximum.

[0051] First, the generator is called from a given molecule to obtain a new molecule, and then the strategy module is used to predict the key properties of the new molecule and make a reward evaluation. This part is named the graph convolution strategy module because it uses the feature extraction and gradient strategy of the graph network. After the graph convolution strategy, the reward is converted into the loss function caused by the generation model, and its parameters are derived and updated, and then the next round of iterative generation is entered. Through this process, reinforcement learning is combined with deep learning ideas to dynamically modify the model. As the number of rounds increases, the network will generate molecules with higher and more reasonable expected properties.

[0052] The Molr model is used to predict the properties of molecular graphs, and it has been trained in the previous prediction of binding degree experiment. In addition to the addition of chemical reaction constancy in the training method, the basic network structure is similar to the traditional graph neural network, and the core is composed of stacked GCN modules. All the attributes of the last layer are spliced ​​and expanded into one dimension, and connected to several fully connected networks to obtain prediction indicators. The binding degree with each target is obtained from the experimental results of large-scale databases such as chembl. In order to minimize the errors caused by the experiment, a more robust loss function Cauthy loss is used, and two adaptive parameters α are proposed as robustness parameters of the loss. c focuses on the local range of the loss near the coordinate origin of x=0, and they participate in the calculation of the loss function.

[0053] After the attribute prediction model is fully trained on the molecule-protein binding dataset, it generates the attribute prediction value for the given molecule as the reward for the generation action, and takes the inverse value to get the loss of the generation model. The workflow of the system is as follows: Figure 4 .

[0054] Among them, mol_gen is a molecular generation model. Use the generator of the transformer architecture trained in the previous experiment and the Molr molecular representation model trained in the previous article, connect to the pki property predictor, and fine-tune it on the new dataset to serve as the pki property predictor. At the end of the whole process, a batch of new molecules with different structures from the original molecules and properties that meet expectations will be obtained.

[0055] The model's optimization results for the drug aspirin are as follows: Figure 5 As shown in the figure. The hydrophilicity of the molecules generated in the three rounds is 3.38, 3.57 and 4.06 respectively, showing an upward trend. The generated molecules still retain the main carbon chain structure of the source molecules, but modify the local groups and optimize properties such as water solubility. Moreover, as the number of training rounds increases, the generated molecules are more similar to the main part of the source molecules, which follows the principle of achieving the best optimization from the smallest possible structural modification.

[0056] In summary, the present invention improves the molecular optimization model. A sequence-to-sequence language model is used as the backbone network, an EAGCN module is added as an auxiliary feature extraction method, and a reinforcement learning strategy is used during training. Taking the actual drug aspirin as an example, after multiple rounds of optimization, the new drug finally obtained meets the basic rules of stability, and fully meets the expected optimization goals in terms of logD, solubility and clearance indicators. The model proposed in the present invention has been significantly improved compared to previous models on a variety of public drug target data sets, and can effectively solve the problem of unbalanced distribution of drug and target data. In the drug optimization task, MolGen can achieve better target binding activity on the basis of ensuring the basic stability of the new drug.

[0057] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A drug molecule generation optimization method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Train the MolrProtTrans model with the drug molecule and protein binding data set to complete the classification and regression tasks; Step 2: In the drug molecule optimization task, a sequence-to-sequence basic generative optimization model is trained through the performance change data set of the initial drug and the molecule after local changes of the initial drug; Step 3: Add reinforcement learning strategy and dynamically train the model.

2. The method for optimizing drug molecule generation based on deep reinforcement learning according to claim 1, characterized in that: In the step 1, firstly, the drug molecule representation network and protein representation network of the MolrProtTrans model are pre-trained; The Transformer architecture is used to connect the feature vectors of drug molecules and proteins, the molecule vector is input into the Encoder module, and the protein vector is sent into the Decoder module; Each piece of molecular and protein data in the data set of the classification problem in the task is obtained through the designed representation model to obtain the respective feature vectors, which are spliced ​​together, and the true value label of whether the data is actually combined is recorded to train the spliced ​​model; The described use of MolrProtTrans for regression tasks; As described above, before training the overall network of MolrProtTrans, the Molr model is pre-trained to better represent the molecular features; The described ProtTrans model needs to be pre-trained to represent protein sequences; The obtained feature vectors of drugs and targets are transmitted to the Transformer framework; The described use of various machine learning models to dock into the feature vectors of molecules and proteins generated by MolrProtTrans; The modified output interface of MolrProtTrans is applied to regression tasks; The molecular representation models used include KCL, Grover and Mpnn, and the protein representation models include ESM and CNN.

3. The method for optimizing drug molecule generation based on deep reinforcement learning according to claim 1, characterized in that: In step 2, in order to solve the drug generation problem, the present invention uses a sequence model in a language model, called Seq2Seq, and uses a molecular attribute change dataset to train a molecular optimization model MolGen; In order to effectively solve the translation problem of long sequence sentences, the present invention introduces an attention mechanism; After each layer of the decoder receives a word and converts it into a decoding vector, it first receives the hidden vectors translated by the encoder for all words, then calculates the distance between the word vector calculated by the decoder and the vector of each input word, and uses this as the attention score of each input word for the output word, and calculates the final vector of the decoder for the word; The MolGen model is divided into an RNN encoder and an RNN decoder with an attention mechanism, and the RNN adopts a bidirectional LSTM structure; The RNN encoder converts the input word into a 128-dimensional vector first, and then passes it through three consecutive bidirectional LSTMs, with the middle layer dimension of 256. In order to prevent overfitting, the drop probability value is set to 0.2; When the model receives a word input, it first encodes it into the embedding space into a 128-dimensional vector representation, and then obtains the hidden representation of each word in two directions, from front to back and from back to front, and its dimension becomes 256. The same is true for the Decoder. The salient feature of the Seq2seq method is that it outputs in sequence. Each time the decoder outputs the next word vector, it needs to be supported by the previous word vector and all the embedding vectors of the encoder. The result is obtained through the model, and the result shape is [64, L, 64], where 64 is the batch size, L is the length of the molecular sequence, and 64 is the feature vector dimension. Finally, the 64-dimensional feature vector is mapped to an atom by comparing the atom dictionary to obtain the output molecular sequence of [64, L, 1]. The MolGen model is divided into two parts: RNN encoder (Encoder) and RNN decoder (Decoder) with attention mechanism; When the model receives a word input, it first encodes the word into a vector, then obtains the hidden representation of each word in sequence from front to back and from back to front through a bidirectional LST M structure, and then adds the forward and reverse hidden layer vectors of each input word to obtain the result through the model; In the decoding stage, the decoder outputs the next word vector based on the previous word vector and all the embedding vectors of the encoder; The model described uses batch training. Suppose there are N batches of training sequence data, xi and yi represent the source sequence and target sequence of the i-th batch respectively, and θ is the complete parameter set of the model. Then the training loss is: Among them, i traverses all data entries, t traverses all iteration batches of the model, and the model optimizes the t-1th iteration result generated by the i-th batch of data to obtain the t-th round of optimization, and compares it with the target value to calculate the predicted loss; The model is run on Windows system, the GPU used is NVIDIA GeForce, the default batch size is 64, the model parameters are updated using Adam optimizer, the initial learning rate is set to le-4, and there is a proportional decay of 0.8 every 10 rounds of training; In order to achieve better learning results, the MolGen inserts the following two modules (EAGCN and ECA) into the backbone network to enhance the performance of the model in the molecular representation stage and the transmission stage; The EAGCN described above is based on the CEM structure, which can assign different attention weights to different atomic nodes and put them into the representation of atoms. At the same time, it can add strong chemical bond consistency constraints, so that even in different molecules in the data set, the same type of bonds can still be assigned the same attribute encoding; For a given initial feature map, the size is H*W*C; First, the global average pooling GAP is used to average the H*W plane matrix of each channel to obtain an initial channel attention vector with a shape of 1*1*C; The ECA (Efficient Channel Attention Network) uses a convolution with a kernel size of k to slide on the original features and adds padding to keep the shape of the vector unchanged.

4. The method for optimizing drug molecule generation based on deep reinforcement learning according to claim 1, characterized in that: In step 3, a reinforcement learning training method is added to the molecular optimization model to help complete the drug molecule optimization task. A differential equation is used to describe the behavior of reinforcement learning: t+1 =f(x t ,a t ,e t ), where x t is the current state of the system, a t is the operation selected in the current stage, e t is a random deviation; In the reinforcement learning, the reward is recorded as R = R (xt, at). Since the action space of learning is continuous, the optimization target is not necessarily the reward Rt at a certain moment, but the sum of all Rt after a series of time and operations. The reinforcement learning formula is expressed as a system of equations: Among them, R t Given x t ,a t The reward function when ; The data processing and evaluation index described above takes X as the input molecule and Z as the expected requirement. The initial test set is divided into [X1, Z1], and 10% of the entire MMP molecule pairs are selected, in which all molecules X and attributes T appear in the training set, but the matching pairs of [X, T] do not overlap with the training set; The second test set [X2, T2] is called the molecular test, which is screened according to the new specification in all test sets. The starting molecules have not appeared in the training set, but the attribute changes are consistent with the training set. After screening, a total of 12,000 pairs of optimized samples are obtained; The third test set has two screening criteria. First, the attributes in the test set do not overlap with those in the training set. According to statistics, the logD value of the initial molecular attribute is between 2 and 4, and has a high Clint value and low solubility. The main optimization goal is to make the molecule have a lower logD value and Clint, while changing it to high solubility; The property changes are: logD changes from [-1-0], Clint from high to low, solubility from low to high; A total of 7000 sample pairs meeting the requirements were found; The model will generate at least five different solutions for the input molecule, all of which meet the expected property change requirements; The described firstly performs an ADMET property evaluation, meets its requirements, and uses the mean square error to determine that the generated molecule is significantly different from the original molecule; The logD index is that the difference between the logD of the generated molecule and the original value is within 0.5, which is excellent; The solubility evaluation index is that for the initial molecule, the dividing line between high and low solubility is 1.7-0.6, and for the obtained product, the dividing line is 1.7+0.6; The Clint indicator described sets the floating size to 0.3, and the mechanism is similar to logD; To ensure that the molecules generated by the model are synthesizable, we should first check whether there is a chemical reaction that can achieve the transformation of the pair of molecules. If not found, we should observe whether the proportion of heavy atoms in the R group in the generated molecule is qualified. The model was used to optimize the drug aspirin, and the hydrophilicity of the molecules generated in three rounds was 3.38, 3.57, and 4.06, respectively, showing an upward trend; The generated molecules retain the carbon chain main structure of the source molecules and optimize properties such as water solubility; As the number of training rounds increases, the generated molecules become more similar to the backbone of the source molecules, following the principle of achieving the best optimization by making the smallest possible structural modification.

Images