Method for predicting drug target combination based on deep learning classification

Abstract

The invention discloses a method for classifying and predicting drug target combination based on deep learning, in order to overcome the applicability and limitation of an existing model on molecule and protein data, a transformer is used as a main body structure, Molr and ProtTrans are respectively used as networks for extracting molecule and protein information, and a model MolrProtTrans is combined. Afterwards, simplification and comparison testing of the model are added, vectors generated by the original model are connected to a machine learning model in a butt joint mode, the logic of combination of the specified medicine and the target spot sequence is predicted, the model obtains the value of 0 or 1 to represent whether the medicine and the target spot sequence are combined or not, and testing indexes are a general ROC curve in a classification task, the accuracy rate, the recall rate and the like. The method aims at rapidly judging the most basic attributes of new drugs, and by adopting the model MolrProtTrans provided by the invention, effective drugs which have effects with specified proteins can be rapidly screened out.

Description

Technical Field

[0001] The present invention belongs to the technical field of drug research and development, and in particular relates to a method for predicting drug target binding based on deep learning classification. Background Art

[0002] In the process of drug development, a key step is to understand how drug molecules interact with macromolecules in the body (such as receptors, enzymes, ion channels, transporters, etc.), which are called "targets" in the task of drug development. Drug design is shifting from the traditional method that relies entirely on experiments to comprehensive development assisted by artificial intelligence simulation calculations. Simulation calculations, supported by big data, can significantly shorten the drug development cycle and save experimental costs. There are currently many models that can predict the binding ability of drugs to targets. However, due to data distribution and model structure, many models rely too much on small molecule information when making predictions and ignore a large number of protein features, so that they are more inclined to predict molecular properties.

[0003] At present, a large amount of drug molecule and protein data has been made public, and various algorithms and network models based on these data sets have emerged, and certain results have been achieved. However, the current models still have limitations, and the cutting-edge technologies in the field of artificial intelligence have not yet been fully applied in the biomedical field. Therefore, it is crucial to deeply explore and analyze the applicability and limitations of existing models on molecular and protein data, and then design new structures to make up for these deficiencies, and draw on the latest progress in fields such as natural language processing to promote biomedical research. Summary of the invention

[0004] 1. Pre-trained drug molecule representation network of MolrProtTrans model;

[0005] 2. Pre-trained protein representation network of MolrProtTrans model;

[0006] 3. Combine the Transformer architecture with the vector representations obtained in step 1 and step 2, design a dual attention mechanism, train and evaluate the MolrProtTrabs model;

[0007] Fourth, the drug and target vectors obtained by MolrProtTrans were directly spliced ​​and connected to various machine learning models to build a hybrid structure model.

[0008] Beneficial effects of the present invention:

[0009] 1. The traditional drug design process faces the challenges of high cost and low success rate of clinical trials, prompting the scientific research community to turn to software-assisted means to improve R&D efficiency. The present invention fully proves from a theoretical perspective that the architecture design made by MolrProtTrans is excellent enough, and it is very appropriate to choose Molr as the molecular processing branch and ProtTrans and prose as the protein processing branch.

[0010] 2. Ablation experiments have shown that the fusion part of the network uses the transformer architecture, designs bidirectional attention, and applies triple loss, which plays a significant role in improving the effect of the overall model.

[0011] 3. The present invention explores the interpretability by reducing the dimension of the vector of the model output layer to a plane, so that people can not only believe the prediction results of the model but also participate in the prediction process, and can view the characteristic spatial distribution of molecules and proteins, as well as the details of their binding relationships, providing convenience for researchers. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is the technical roadmap of the present invention.

[0013] Figure 2 It is the MolrProtTrans model structure in the present invention.

[0014] Figure 3 This is a structural diagram of the protein Transformer large model in the present invention.

[0015] Figure 4 This is the QKV attention calculation method in the present invention.

[0016] Figure 5 is the training result of MolrProtTrans in GPCR in the present invention; wherein, Figure 5 -a is the increase of GPCRAUC / PRC; Figure 5 -b is the GPCR ROC curve.

[0017] Figure 6 This is the ROC of MolrProtTrans_mini on four data sets in the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] The data used in the training of the model used in the present invention are collated from public data sets, with information such as the molecule name, protein name, and the degree of binding between them marked. Specifically, the following data sets are included:

[0020] GPCR is G protein-coupled receptors (GPCR). G protein-coupled receptors are one of the main types of transmembrane transporters, which can bind extracellular molecules and activate intracellular signaling pathways. Experiments have shown that many major classic diseases such as cancer and diabetes are related to GPCR dysfunction, so the development of drugs that bind to GPCRs has become a hot topic.

[0021] BindingDB is a dataset extracted from scientific research papers and patents. For example, it steadily collects 35,000 sets of protein-molecule relationships from US patents every year. Its records also include experimental conditions such as temperature, pH, and buffer composition, which increases reliability.

[0022] Kinase is a comprehensive human kinase pocket database that provides information on the sequence, structure, hydrophilicity and drug interactions of kinase proteins. It contains 1717 pockets from 255 kinases and divides them into 91 pocket clusters based on the structure and position information of the pockets, which can help with drug optimization and classification.

[0023] The human protein binding data in the experiment were collected from the DrugBank database. DrugBank is mainly used to study drug-target relationships for drug design, docking and virtual screening. The original dataset of BindingDB

[65] is available at: https: / / www.bindingdb.org / rwd / bind / index.jsp.

[0024] Step 1: Train the Molr molecular representation model in MolrProtTrans. The overall architecture of the MolrProtTrans model is as follows: Figure 2 .

[0025] The basis of the Molr model is the graph neural network GNN. The message aggregation function of GNN is the most basic and important operation. The formula is:

[0026] Formula 1: x′=D -1 AxW;

[0027] Among them, D is the degree matrix, the size is N*N, N is the number of atoms in the molecule. Each element on the diagonal represents the degree of the node, that is, the number of atoms directly connected to it. The function of this matrix is ​​to normalize the result of the operation; the shape of the A matrix is ​​also N*N, where the value of the i row and j column represents the connection relationship between the i-th atom and the j-th atom. The simplest value is 0 or 1, indicating whether the i and j atoms are directly connected; x is the eigenvector of all atoms, and W is the deformation function of the eigenvector.

[0028] The Molr method is experimented with GNN's basic model GCN, as well as improved models GraphSage, GAT, and TAG.

[0029] The main innovation of the Molr is to apply chemical reactions to the training of molecular representations. Taking into account the conservation laws in chemical reactions, such as atomic conservation, charge conservation, mass conservation, etc., reactants and products should have some similarity, that is, Σhr r∈R=Σh(P)p∈P, so the original training idea is to use the GCN network to obtain the vector representation of all reactants and all products in each reaction respectively, obtain hr and hp, and then make a difference to get the loss Loss=hr-hp. In actual operation, in order to prevent the situation where all substrate and product molecules are represented as 0 vectors, global parameters β and γ are introduced. The actual formula is:

[0030] Formula 2:

[0031] Where L is the model training loss, hr is the feature vector of the substrate, hp is the feature vector of the product, Ri is the i-th reactant, and Pj is the product of the j-th reaction;

[0032] In each batch of training, a series of reactions are obtained, split into substrate and product parts, labeled R1, .., Rn, and P1, .., Pn respectively.

[0033] Step 1.1: Before using the Molr model to train data, you first need to obtain an overview of the properties of all molecules in the entire dataset. The property list is represented by a dictionary, which is called feature encoder. The recorded data completely corresponds to the properties of each atom, including the atom name, charge number, whether it is in an aromatic ring, and the number of hydrogen atoms attached to it. In data processing, traverse the files of the dataset and divide them into three subsets according to 7:2:1 for training, testing, and verification.

[0034] Step 1.2: In order to express continuity, the data is traversed four times. Each time, the various values ​​of this attribute are counted and labeled. Considering that new attributes may be encountered when the dictionary is used later, unknown labels are set for the four attributes separately, named unknown, which also occupies one dictionary position. Subsequent attributes continue to be counted from the last attribute label of the previous round to avoid duplication of labels.

[0035] Step 1.3: Convert the smiles format of the molecule into a standard graph structure. In order to structure and uniformly represent the data set, an interface Load data is provided, which will return the graph structure data processed by the graph network according to the input data set type, and divide it into training set, test set and validation set. The key type inside Load data is SmilesDataset. It includes two key processes (read data and Process) in the data processing process. This process is a template provided by the molecular dataset provided by the dgl software. As long as the required logical operations are written in the corresponding function, the model will automatically call these methods in sequence.

[0036] Step 1.3.1: read data. This method receives the input dataset name, finds the dataset file according to the name and reads it line by line, identifies the reactant and product molecules in each data, and calls pysmiles to convert them into two molecular preliminary graph structures in two lists, reactant graphs and product graphs.

[0037] Step 1.3.2: Process, which converts the preliminary molecular graph structure into an augmented graph structure that can be recognized by the model. Then extract all the starting nodes and target nodes in the preliminary molecular graph and construct the graph structure of dgl. In Process, the graph to dgl function is called, and the feature encoder in step 1.1 is used as the reference dictionary to traverse all the nodes in the molecular dgl graph structure, and then the four focus attributes of the node are traversed, and the corresponding label values ​​of these attributes are searched in the dictionary one by one. Through such operations, the attribute features of the node are stored in the form of vectors. Then this enhanced graph structure is further modified, the unidirectional edges are expanded to bidirectional edges, and self-return edges are added, which means that the starting node and the destination node are the same point.

[0038] Step 1.4: When receiving the input, first obtain its feature matrix and convert it into a one hot format encoding vector with a shape of [N, feature dim]. N represents the number of atoms in the molecule, and feature dim represents the dimension of the feature of each molecule, which is 4 in this invention.

[0039] Step 1.5: The core structure of Molr is 2 graph convolution layers. The input feature dimension of the first layer is set to 4, and the output dimension is set to 1024. A relu activation function is added after the graph convolution operation to increase the nonlinearity of the model. The input feature dimension and output dimension of the second layer are both set to 1024. Then multiply it by the hyperparameter factor, the value of which is the square root of the hidden layer feature dimension 1024. Finally, the graph pooling operation is connected to perform complete standardization on the feature data. The dataloader reads out the substrate graph structure sequence and product graph structure sequence of each batch, and processes them into 1024-dimensional feature vectors using the Molr model respectively. Then, the losses of the two side vectors of this batch are calculated according to the rules, and the model is updated accordingly.

[0040] Step 2: Train the ProtTrans protein model in MolrProtTrans. This requires a large-scale protein structure space and property dataset, including UniProt, Pruitt, etc. Figure 3 The process of obtaining protein sequence information by the ProtTrans model is demonstrated.

[0041] The ProtTrans model is obtained using a comprehensive training method. The ProtTrans model mainly uses SOM, Uniref50 and Uniref100, and BFD databases. The database SCOP for structured protein classification was also added later.

[0042] In the data processing described above, ProtTrans translates individual amino acids into input words, or token symbols. After the protein is tokenized, each line of information consists of a series of tokens separated by spaces. Unprocessed amino acid sequences are classified as unknown.

[0043] The main body of the model is a bidirectional LSTM structure. The three main self-supervisory tasks of the model are to mask the local molecular sequence, make a contact prediction map between protein residues, and predict the similarity of the structure. Specifically, in the task of predicting the residue contact map, the LSTM hidden layer used will learn the bidirectional linear relationship from the three-dimensional residue structure, which is consistent with the bidirectionality of LSTM.

[0044] The ProtTrans model in the present invention performs sufficient preprocessing on protein sequences. First, all molecular symbols and protein symbols in the statistical data set are counted, and the word library is established respectively. Then, for each group of proteins and molecules input into the model, an amino acid region in the middle of the protein sequence and a region in the middle of the molecule are extracted respectively, and the corresponding ID value is obtained by comparing the word library according to their respective contents. In addition, ProtTrans also uses a model that receives a single protein sequence and a related stored co-evolutionary multi-sequence group together. The sequence structure analysis of a single protein is handed over to the aforementioned ProtTrans for processing, and the co-evolutionary analysis will promote the mutual transmission of evolutionary information between a group of proteins.

[0045] Step 2.1: First, process the input protein sequence into a token string, and then add the common operations of the natural language model, that is, add position encoding. The formula is as follows:

[0046] Formula 3:

[0047] Formula 4:

[0048] Among them, i is the position number of the element, which is divided into cardinality number and even number, d_model is the dimension of the model, and PE calculates the encoding of the corresponding position.

[0049] Step 2.2: After the model preprocessing in the ProtTrans series, the Decoder module is used to generate a more accurate contextual semantic vector for each token. The vector can also be extracted from the hidden layer of the last layer of the Transformer module as the initial input for downstream protein prediction. For structural symmetry, the output dimension of the last linear layer of the Decoder of ProtTrans is set to 1024 in the present invention, which is consistent with the Molr molecular representation model interface.

[0050] Step 3. The feature vectors of drugs and targets obtained in steps 1 and 2 are transmitted to the Transformer skeleton. The Transformer architecture adopts a dual attention mechanism, multiplying the target protein matrix with the source molecule matrix to obtain the correlation between vectors Q and K, and adding an inverted relationship matrix. The source molecule matrix is ​​used as Q, and the target protein matrix is ​​used as K. A new matrix is ​​obtained and transposed. The transposed new matrix has the same shape as the original relationship matrix. The original relationship matrix is ​​added to the transposed new matrix to obtain an enhanced relationship matrix. In the Encoder and Decoder, the important operations are self-attention operations and cross-attention operations, which involve vector transformations of Q, K, and V. The calculation method of QKV is as follows: Figure 4 shown.

[0051] Step 3.1: Taking the self-attention operation as an example, the feature vector X of the drug molecule has a shape of [B, 256, 4], where B represents the batch size, which is set to 64, 256 is the number of token strings, and 4 represents the feature dimension. The product of 256*4 is 1024, which is the result of splitting the original 1024-dimensional molecule vector into a token format. Then use the WQ, WK, WV matrices to multiply X to obtain Q, K, V vectors, all of which have a shape of [64, 256, 64].

[0052] Step 3.2: Multiply the vector Q by the transpose K' of K to get a vector M' of size [64, 256, 256], divide it by the square root of the token length of the matrix K to 16, and send it to the softmax activation layer to get the standardized Q, K relationship matrix M.

[0053] Step 3.3: Multiply vector M and V to get the final self-attention result Att, with a shape of [64, 256, 64]. The self-attention of the protein branch is the same as steps 3.1 and 3.2. After the attention mechanism operation, the shape of the vector does not change, only the value changes.

[0054] Step 3.4: In the cross-attention mechanism, the method of calculating the Att vector from Q, K, V is the same as step 3.1 and step 3.2. However, the sources of Q, K, and V are different. Among them, Q comes from the molecular branch input X, and K, V come from the protein branch input Y. As the other side of the dual attention mechanism, Q, K come from the molecular input X, and V comes from the protein input Y. Also due to the shape invariance of the attention operation, the final shapes of the two attention matrices Att1 and Att2 are both [64, 256, 64]. The final attention matrix Att is obtained by directly adding the two matrices. After several linear layer transformations, residual connections, and the softmax activation function of the last layer, the model prediction result of [B=64, classes=2] is output. The results of the model on GPCR are shown below. Figure 5 In order to reduce the complexity of the model, the present invention also designed a simplified version of MolrProtTrans_mini. The ROC curves of the simplified model on four data sets (GPCR, Kinase, BindingDB, human) are shown in Figure 6 As shown in Figure 2, the accuracy of the basic model on GPCR is 0.557, which is much lower than the 0.835 of the complete MolrProtTrans model, proving the importance of modules such as bidirectional attention and triple loss as well as the training method in the complete model.

[0055] Step 4: In order to verify the effectiveness of traditional machine learning in molecular and protein research tasks, each molecular and protein data in the classification problem dataset in the above task is subjected to the designed representation model to obtain their respective feature vectors, which are then spliced ​​together, and the true value label of whether the data is actually combined is recorded to train the spliced ​​model.

[0056] The classification models provided by sklearn are described, including SVM, decision tree, KNN clustering, etc.

[0057] The various machine learning models are connected to the feature vectors of molecules and proteins generated by MolrProtTrans. First, the two [64, 1024] vectors x and y are concatenated to obtain a [64, 2048] vector P, which is then connected to the machine learning model for vector-to-category training.

[0058] The evaluation indicators of the classification task include the following:

[0059] Accuracy = (TP + TN) / (TP + TN + FP + FN);

[0060] Precision = TP / (TP+FP);

[0061] Recall = TP / (TP+FN);

[0062] To obtain the AUC index, we first need to calculate FPR and TPR, where FPR = FP / (FP+TN); TPR = TP(TP+FN). With FRP as the x-axis and TPR as the y-axis, AUC is the area enclosed by the curve and the x-axis.

[0063] Among all classification tasks, the baseline model MolrProtTrans of the present invention has the best comprehensive performance, achieving an accuracy of 0.835 on the GPCR dataset, exceeding the best reported methods transformerCPI of 0.716 and DBSI of 0.825. The best result of docking the representation vector of MolrProtTrans to machine learning is 0.70 for docking to a Gaussian classifier, and the accuracy of the simplified basic version of the model on GPCR is 0.63. This fully demonstrates the superiority of our model on GPCR. On a difficult-to-learn dataset such as BindingDB, the baseline model still achieved an accuracy of 0.57, exceeding the 0.53 of the simplified model. On the Human dataset, the accuracy of the baseline model is as high as 0.950, exceeding the best reported methods transformerCPI of 0.916, GraphDTA of 0.882, other machine learning models SVM of 0.866, KNN of 0.907, and far exceeding the basic model of 0.46. When the vector of the baseline model was docked to machine learning, the highest accuracy of 0.70 of the Gaussian method and the lowest accuracy of 0.662 of the decision tree on the highest GPCR dataset were not much different, proving the reliability of the model output vector and the stability of the selected machine learning model.

Claims

1. A method for predicting drug target binding based on deep learning classification, characterized in that: The following steps are involved: Step 1: Pre-train the drug molecule representation network of the MolrProtTrans model; Step 2: Pre-train the protein representation network of the MolrProtTrans model; Step 3: Combine the Transformer architecture with the vector representations obtained in steps 1 and 2, use the public drug target in vivo binding data set, design a dual attention mechanism, and train and evaluate the effect of Molr ProtTrabs on the classification task; Step 4: Directly concatenate the drug and target vectors obtained by MolrProtTrans analysis, and connect the synthesized vectors to each machine learning model to construct a hybrid structure model.

2. According to step 1 in claim 1, it is characterized in that: Before training the MolrProtTrans overall network, the Molr model was pre-trained using the Brenda and Rhea datasets to better represent molecular features. The basis of the Molr model is the graph neural network GNN. The message aggregation function of GNN is the most basic and important operation. The formula is: Formula 1: x'=D -1 AxW; Among them, D is the degree matrix, the size is N*N, N is the number of atoms in the molecule. Each element on the diagonal represents the degree of the node, that is, the number of atoms directly connected to it. The function of this matrix is ​​to normalize the result of the operation; the shape of the A matrix is ​​also N*N, where the value of the i row and j column represents the connection relationship between the i-th atom and the j-th atom. The simplest value is 0 or 1, indicating whether the i and j atoms are directly connected; x is the eigenvector of all atoms, and W is the deformation function of the eigenvector. The Molr method described above conducts experiments based on the basic GNN model GCN, as well as the improved models GraphSage, GAT, and TAG. Considering the conservation laws in chemical reactions, such as atomic conservation, charge conservation, mass conservation, etc., the reactants and products have a certain similarity, that is, Σhr r∈R=Σh(P)p∈P, the GCN network is used to obtain the vector representation of all reactants and all products in each reaction, hr and hp are obtained, and then the loss Loss=hr-hp is obtained by difference. In actual operation, in order to prevent the situation where all substrate and product molecules are represented as 0 vectors, global parameters β and γ are introduced. The actual formula is: Formula 2: In each batch of training, a series of reactions are obtained, which are split into substrate and product parts, marked as R1, .., Rn and P1, .., Pn respectively. Among them, L is the model training loss, hr is the feature vector of the substrate, hp is the feature vector of the product, Ri is the i-th reactant, and Pj is the product of the j-th reaction. Before using the Molr model to train data, it is necessary to first obtain an overview of the attributes of all molecules in the entire data set. The attribute list is represented by a dictionary, and the dictionary is recorded as a feature encoder. In the data processing, the files of the data set are traversed and divided into three subsets according to 7:2:1 for training, testing and verification. The data is traversed four times, and each time various values ​​of statistical attributes are scanned and labeled. The smiles format of the molecule is converted into a standard graph structure. In order to structure and uniformly represent the data set, an interface Load data is provided. The key type in the Load data is SmilesDataset, which includes two key processes (read data and process) in the data processing process. The read data receives the input data set name, finds the data set file according to the name and reads it line by line, identifies the reactant and product molecules in each data, and calls pysmiles to convert them into two preliminary molecular graph structures in two lists, reactant graphs and product graphs. The process described above converts the preliminary molecular graph structure into an augmented graph structure that can be recognized by the model. Then all the starting nodes and target nodes in the preliminary molecular graph are extracted to construct the graph structure of dgl. The graphto dgl function is called in the process, and the feature encoder used is used as a reference dictionary to traverse all the nodes in the molecular dgl graph structure, and then the four focus attributes of the node are traversed, and the corresponding label values ​​of these attributes are searched in the dictionary one by one. Through such operations, the attribute features of the node are stored in the form of a vector. Then this enhanced graph structure is further modified, the unidirectional edge is expanded to a bidirectional edge, and a self-return edge is added, which means that the starting node and the destination node are the same point. When receiving the input, the feature matrix of the reactant and product molecules is first obtained and converted into a onehot encoding vector with a shape of [N, feature dim]. N represents the number of atoms in the molecule, and feature dim represents the dimension of the feature of each molecule, which is 4 in the present invention. The core structure of Molr is 2 graph convolution layers. The input feature dimension of the first layer is set to 4, the output dimension is set to 1024, and a relu activation function is added after the graph convolution operation to increase the nonlinearity of the model. The second layer input feature dimension and output dimension of the Molr core structure are both set to 1024. Then, the hyperparameter factor is multiplied, and the value is the square root of the hidden layer feature dimension 1024. Finally, the graph pooling operation is connected to perform complete standardization on the feature data. The dataloader reads out the substrate graph structure sequence and product graph structure sequence of each batch, processes them into 1024-dimensional feature vectors using the Molr model, calculates the loss of the two-side vectors of this batch according to the rules, and updates the model accordingly.

3. According to step 2 in claim 1, it is characterized in that: Pre-trained ProtTrans model to represent protein sequences. It requires ultra-large-scale protein structure space and property data sets, including UniProt, Pruitt, etc. The ProtTrans model is obtained using a comprehensive training method. First, the backbone network is pre-trained on a large number of natural protein sequences using self-supervised learning, and then supervised training methods are used on small structured datasets to enrich the learning content and improve the performance of downstream protein property prediction tasks. The ProtTrans model mainly uses the SOM, Uniref50 and Uniref100, BFD databases, and later added the SCOP database for structured protein classification. In the data processing described above, ProtTrans translates individual amino acids into input words, or token symbols. After the protein is tokenized, each line of information consists of a series of tokens separated by spaces. Unprocessed amino acid sequences are classified as unknown. The main body of the ProtTrans model is a bidirectional LSTM structure. The three main self-supervisory tasks of the model are the task of masking local molecular sequences, making contact prediction maps between protein residues and the task of predicting structural similarity. The ProtTrans model described above performs sufficient preprocessing on protein sequences. First, all molecular symbols and protein symbols in the statistical data set are counted, and the word library is established respectively. Then, for each group of proteins and molecules input into the model, an amino acid region in the middle of the protein sequence and a region in the middle of the molecule are extracted respectively, and the corresponding ID value is obtained by comparing the word library according to their respective contents. In addition, ProtTrans also uses a model that receives a single protein sequence and a related stored co-evolutionary multi-sequence group together. The sequence structure analysis of a single protein is handed over to the aforementioned ProtTrans for processing, and the co-evolutionary analysis will promote the mutual transmission of evolutionary information between a group of proteins. The training results process the input protein sequence into a token string, and then add common operations of natural language models, that is, adding position encoding. The formula is as follows: Formula 3: Formula 4: Among them, i is the position number of the element, which is divided into cardinality number and even number, d_model is the dimension of the model, and PE calculates the encoding of the corresponding position. After the model preprocessing in the ProtTrans series, the Deocer module is used to generate a more accurate contextual semantic vector for each token. The vector can also be extracted from the hidden layer of the last layer of the Transformer module as the initial input for downstream protein prediction. For structural symmetry, the present invention sets the output dimension of the last linear layer of the Decoder of Prot Trans to 1024, which is consistent with the Molr molecular representation model interface.

4. According to step 3 in claim 1, it is characterized in that: The Transformer architecture adopts a dual attention mechanism, multiplying the target protein matrix with the source molecule matrix to obtain the correlation between vectors Q and K, and adding an inverted relationship matrix. The source molecule matrix is ​​used as Q and the target protein matrix is ​​used as K. A new matrix is ​​obtained and transposed. The transposed new matrix has the same shape as the original relationship matrix. The original relationship matrix and the transposed new matrix are added to obtain an enhanced relationship matrix. Taking the self-attention operation as an example, the shape of the feature vector X of the drug molecule is [B, 256, 4], where B represents the batch size, which is set to 64, 256 is the number of token strings, and 4 represents the feature dimension. The product of 256*4 is 1024, which is the result of splitting the original 1024-dimensional molecule vector into token format. Then use the WQ, WK, WV matrices to multiply X to obtain Q, K, V vectors, all of which have a shape of [64, 256, 64]. The vector Q is multiplied by the transpose K' of K to obtain a vector M' of size [64, 256, 256], which is then divided by the square root of the token length of the matrix K to 16 and sent to the softmax activation layer to obtain the standardized Q, K relationship matrix M. In the cross-attention mechanism, Q, K, and V come from different sources. Q comes from the molecular branch input X, and K and V come from the protein branch input Y. On the other side of the dual attention mechanism, Q and K come from the molecular input X, and V comes from the protein input Y. Due to the shape invariance of the attention operation, the final shapes of the two attention matrices Att1 and Att2 are both [64, 256, 64]. The final attention matrix Att is obtained by directly adding the two matrices. After linear layer transformation, residual connection and the softmax activation function of the last layer, the model prediction result of [B = 64, classes = 2] is output. In order to reduce the complexity of the model, the present invention also designed a simplified version of MolrProtTrans_mini. The accuracy of the basic model on GPCR is 0.557, which is much lower than 0.835 of the complete MolrProtTrans model.

5. According to step 4 of claim 1, it is characterized in that: In order to verify the effectiveness of traditional machine learning in molecular and protein research tasks, each molecular and protein data in the classification problem dataset in the above task is subjected to the designed representation model to obtain its own feature vector, which is then spliced ​​together, and the true value label of whether the data is actually combined is recorded to train the spliced ​​model. The classification models provided by sklearn are described, including SVM, decision tree, KNN clustering, etc. The various machine learning models are connected to the feature vectors of molecules and proteins generated by MolrProtTrans. First, the two [64, 1024] vectors x and y are concatenated to obtain a [64, 2048] vector P, which is then connected to the machine learning model for vector-to-category training. The evaluation indicators of the classification task include the following: Accuracy = (TP + TN) / (TP + TN + FP + FN); Precision = TP / (TP+FP); Recall = TP / (TP+FN); To obtain the AUC index, we first need to calculate FPR and TPR, where FPR = FP / (FP+TN); TPR = TP(TP+FN). With FRP as the x-axis and TPR as the y-axis, AUC is the area enclosed by the curve and the x-axis. In all classification tasks, the baseline model MolrProtTrans of the present invention has the best overall performance, achieving an accuracy of 0.835 on the GPCR dataset. The best result of docking the representation vector of MolrProtTrans to machine learning was 0.70 when docked to a Gaussian classifier. The simplified basic version of the model has an accuracy of 0.63 on GPCR. As mentioned above, on a difficult-to-learn dataset like BindingDB, the baseline model still achieved an accuracy of 0.57, exceeding the 0.53 of the simplified model. As mentioned above, on the Human dataset, the accuracy of the baseline model is as high as 0.950.

Images