A molecular multi-angle contrast learning pre-training method for drug property prediction
By using a multi-angle comparative learning pre-training method, multimodal drug molecule samples are obtained and high-quality augmented samples are constructed, which solves the problems of single modality and unstable data augmentation in existing methods and improves the accuracy of drug property prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CENT SOUTH UNIV
- Filing Date
- 2024-07-05
- Publication Date
- 2026-05-12
AI Technical Summary
Existing molecular representation methods only focus on one or two modalities of molecules, lacking consideration for multimodal information of molecules. The quality is unstable during data augmentation, and the lack of attention to information at different levels of molecules leads to insufficient accuracy in drug property prediction.
A multi-angle contrastive learning pre-training method is adopted to obtain multimodal drug molecule samples including one-dimensional fingerprint sequences, two-dimensional topological maps, and three-dimensional spatial maps. The samples are enhanced with high quality by self-attention score sorting and masked atom construction, and the model training is optimized by adaptively controlling the contrastive learning loss function value.
It improves the accuracy of drug property prediction. Through multimodal characterization and adaptive control, it reduces the influence of low-quality augmented samples, prevents the model from getting trapped in local optima, and improves the model training accuracy.
Smart Images
Figure CN118762766B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of bioinformatics, in particular to a molecular multi-angle contrast learning pre-training method for drug property prediction. BACKGROUND
[0002] Molecular representation plays a fundamental role in drug discovery and understanding of molecular behavior. Strong molecular representation has been proven to be very valuable for various downstream tasks, including molecular property prediction and a series of drug-related tasks. The popular method for solving these tasks usually needs to develop a special module to extract molecular information from drugs or rely on conventional molecular representation such as fingerprints. However, the insights obtained from a specific task dataset are inherently limited, and traditional fingerprints lack the ability to encode the complex topological and geometric details of molecules. Meanwhile, the huge knowledge space exists in the vast amount of available molecular data, so a key research focus is to design a pre-training method to obtain high-quality molecular representation containing comprehensive and multi-view information of molecules.
[0003] So far, the molecular representation learning methods based on contrast learning can be mainly divided into two strategies, the first strategy involves using different data augmentation schemes to construct positive sample pairs; the second strategy designs to construct positive sample pairs by using different data modes of molecules. Although many contrast learning methods for molecular representation traditionally focus on the two-dimensional topological structure of molecules, from the perspective of bioinformatics, understanding the efficacy of drugs and molecular properties has a huge connection with the three-dimensional geometric structure of molecules. However, there are still many problems in the existing methods, first of all, they only focus on one or two modalities of molecules, without fully considering all modal information of molecules; secondly, when data augmentation is performed, most of the existing methods use the method of randomly masking atoms or bonds, which may lead to poor quality of augmented samples; finally, the current research lacks attention to different levels of information of molecules, and the perspective is too single. SUMMARY
[0004] The present application provides a molecular multi-angle contrast learning pre-training method for drug property prediction, which aims to fully consider the modal information of molecules from multiple perspectives, improve the training accuracy of the model, and improve the accuracy of drug property prediction.
[0005] In order to achieve the above purpose, the present application provides a molecular multi-angle contrast learning pre-training method for drug property prediction, comprising:
[0006] Step 1, obtaining a plurality of drug molecules including a plurality of modalities as original samples, the modalities including one-dimensional fingerprint sequence, two-dimensional topological graph and three-dimensional space graph;
[0007] Step 2: For each drug molecule, the original sample is input into the original neural network model. The modal representation of each drug molecule is extracted from the original neural network model, and the self-attention score of each atom in each drug molecule under each modality is calculated. The original neural network model includes a low-level part and a high-level part.
[0008] Step 3: Sort each atom in each drug molecule in each mode based on the self-attention score of each atom in each mode to obtain the atom sorting result of each drug molecule, and mask the atoms that meet the preset masking number in the atom sorting result of each drug molecule to obtain the enhanced sample of the drug molecule in each mode.
[0009] Step 4: Construct positive sample pairs using the original sample and the enhanced sample of the drug molecule. There are multiple positive sample pairs.
[0010] Step 5: After dividing the original neural network model into two parts, comparative learning is performed simultaneously. The learning speed of each positive sample pair is adaptively controlled by calculating the comparative learning loss function value through the modal representation of each drug molecule until the preset training conditions are met, and the pre-training is completed to obtain the neural network model.
[0011] Step 6: Input the training data into the neural network for training to obtain a prediction model for drug property prediction.
[0012] Furthermore, neural network models include:
[0013] First transform neural network module, first graph-level transform neural network module, second graph-level transform neural network module;
[0014] The input to the first transformation neural network module is a one-dimensional fingerprint sequence for each drug molecule;
[0015] The output of the first transform neural network module is a one-dimensional modal representation corresponding to each character in the fingerprint sequence of each drug molecule and the self-attention score of each atom in each drug molecule in the one-dimensional modality.
[0016] The input to the first graph-level transformation neural network module is the two-dimensional topological graph features of each drug molecule;
[0017] The output of the first graph-level transformation neural network module is the two-dimensional modal representation of each atom in each drug molecule and the self-attention score of each atom in each drug molecule in the two-dimensional modality.
[0018] The input to the second graph-level transformation neural network module is the three-dimensional spatial graph features of each drug molecule;
[0019] The output of the second graph-level transformation neural network module is the three-dimensional modal representation of each atom in each drug molecule and the self-attention score of each atom in each drug molecule in the three-dimensional modality.
[0020] Furthermore, the original neural network model consists of two parts: a low-level part and a high-level part.
[0021] The lower-level components include: a lower-level transform neural network encoder in the first transform neural network module, a graph isomorphic neural network layer in the first graph-level transform neural network module, and a deep neural network layer in the second graph-level transform neural network module;
[0022] The high-level components include: the high-level transform neural network encoder in the first transform neural network module, the transform neural network encoder in the first graph-level transform neural network module, and the transform neural network encoder in the second graph-level transform neural network module.
[0023] Furthermore, the first transformation neural network module includes:
[0024] A low-level transform neural network encoder and a high-level transform neural network encoder connected in sequence;
[0025] The one-dimensional fingerprint sequence of each drug molecule is input into a low-level transform neural network encoder for feature extraction, thereby obtaining a one-dimensional low-level modal representation corresponding to each character in the fingerprint sequence of each drug molecule.
[0026] The one-dimensional low-level modal representation is input into the high-level transform neural network encoder for feature extraction, resulting in a one-dimensional high-level modal representation corresponding to each character in the fingerprint sequence of each drug molecule and the self-attention score of each atom in each drug molecule under the one-dimensional mode.
[0027] Furthermore, the one-dimensional high-level modal representation E 1d The expression is:
[0028]
[0029] in, All represent learnable parameters, MultHead 1d This represents the output of the multi-head attention layer.
[0030] Furthermore, the first graph-level transformation neural network module includes:
[0031] A graph isomorphic neural network layer and a transform neural network encoder connected sequentially;
[0032] The two-dimensional topological graph features of each drug molecule are input into a graph isomorphic neural network for feature extraction, resulting in a two-dimensional low-level modal representation of each drug molecule.
[0033] The node degree representation and the shortest path distance representation of each atom are obtained by using the two-dimensional topological graph features of each drug molecule. The node degree representation serves as a supplementary vector for the position encoding in the feature learning process of the transform neural network encoder, and the shortest path distance representation of each atom serves as a supplementary vector for the attention score encoding in the feature learning process of the transform neural network encoder.
[0034] The two-dimensional low-level modal representation of each drug molecule, the node degree information of each drug molecule, and the shortest path distance representation of each atom are input into the transform neural network encoder for feature extraction, so as to obtain the two-dimensional high-level modal representation of each drug molecule and the self-attention score of each atom in each drug molecule in the two-dimensional mode.
[0035] Furthermore, the expression for the two-dimensional low-level modal characterization of each drug molecule is as follows:
[0036]
[0037] in, This represents the two-dimensional modal representation of atom a in the (l+1)th layer, where MLP stands for Multilayer Perceptron. This represents the two-dimensional modal characterization of atom a in the l-th layer. This represents the two-dimensional modal characterization of atom b in the l-th layer. This represents the neighbor bond characterization between atoms a and b in the l-th layer.
[0038] Furthermore, the second graph-level transform neural network module includes:
[0039] A deep neural network layer and a transform neural network encoder connected in sequence;
[0040] The three-dimensional spatial features of each drug molecule are input into a deep neural network layer for feature extraction, resulting in a three-dimensional low-level modal representation of each drug molecule.
[0041] The sum of the three-dimensional distances between atoms and the distance between atoms are obtained by using the three-dimensional spatial map features of each drug molecule. The sum of the three-dimensional distances between atoms is used as a supplementary vector for the position encoding of the transform neural network encoder during feature learning, and the distance between atoms is used as a supplementary vector for the attention score encoding of the transform neural network encoder during feature learning.
[0042] The three-dimensional low-level modal representation of each drug molecule, the representation of the sum of the three-dimensional distances between atoms, and the representation of the distance between each atom are input into a transform neural network encoder for feature extraction, so as to obtain the three-dimensional high-level modal representation of each drug molecule and the self-attention score of each atom in each drug molecule in the three-dimensional mode.
[0043] Furthermore, the expression for the three-dimensional low-level modal characterization of each drug molecule is as follows:
[0044]
[0045] in, This represents the low-level modal characterization of atom d in layer t+1. This represents the low-level modal characterization of atom v in layer t, and d represents the number of atoms in the drug molecule. e represents the learnable parameter. k r represents the distance characterization between atoms a and v with radial basis functions. a r represents the position coordinates of atom a. v This represents the position coordinates of atom v.
[0046] Furthermore, the expression for calculating the contrastive learning loss function value is as follows:
[0047]
[0048] Among them, AMCLOss p This represents the value of the contrastive learning loss function at round p. P represents the normalized score of the ratio of the loss function values of the previous round and the two rounds above the positive sample pair i and j at round p. M Let z represent the set of positive samples of drug molecule M. i Let z represent the representation vector of the drug molecule corresponding to the i-th positive sample pair in the positive sample set. j Other(P) represents the representation vector of the drug molecule corresponding to the j-th positive sample pair in the positive sample set. M z represents the set of samples of all molecules except for the positive sample set of drug molecule M. o Other(P) represents the sample set. M The representation vector of the drug molecule corresponding to the o-th sample in ).
[0049] The above-described solution of the present invention has the following beneficial effects:
[0050] This invention acquires multiple drug molecules with various modalities as raw samples. For each drug molecule, the raw sample is input into a separate original neural network model. Modal representations of each drug molecule are extracted from the original neural network model, and the self-attention score of each atom in each drug molecule under each modality is calculated. Based on the self-attention score of each atom under each modality, the atoms in each drug molecule under each modality are sorted to obtain the atom sorting result for each drug molecule. Atoms meeting a preset masking number are masked from the atom sorting result for each drug molecule to obtain enhanced samples of the drug molecule under each modality. The original neural network model is divided into two parts and comparative learning is performed simultaneously. Comparative learning is calculated using the modal representations of each drug molecule. The learning loss function value adaptively controls the learning speed of each positive sample pair until the preset training conditions are met, thus completing pre-training and obtaining a neural network model. The training data is then input into the neural network for training, resulting in a prediction model for drug property prediction. Compared with existing technologies, this invention considers multiple modalities of drug molecules, extracts modal representations under multiple modalities, and calculates the self-attention score of each atom under multiple modalities to construct high-quality augmented samples, which can reduce low-quality molecular augmented samples caused by unreasonable masks. By calculating the comparative learning loss function value through the modal representation of each drug molecule, the learning speed of molecular knowledge learning during the pre-training process is adaptively controlled, preventing the model from getting trapped in local optima and improving the training accuracy of the model, thereby enhancing the accuracy of drug property prediction.
[0051] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description
[0052] Figure 1 This is a flowchart illustrating an embodiment of the present invention;
[0053] Figure 2 This is a schematic diagram of the pre-training framework in an embodiment of the present invention;
[0054] Figure 3 This is a schematic diagram of the training framework in an embodiment of the present invention. Detailed Implementation
[0055] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0056] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0057] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0058] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0059] This invention addresses existing problems by providing a molecular multi-angle comparative learning pre-training method for drug property prediction.
[0060] like Figure 1 As shown, embodiments of the present invention provide a molecular multi-angle contrastive learning pre-training method for drug property prediction, comprising:
[0061] Step 1: Obtain multiple drug molecules including multiple modalities as raw samples, wherein the modalities include one-dimensional fingerprint sequences, two-dimensional topological maps, and three-dimensional spatial maps;
[0062] Step 2: For each drug molecule, the original sample is input into the original neural network model. The modal representation of each drug molecule is extracted from the original neural network model, and the self-attention score of each atom in each drug molecule under each modality is calculated. The original neural network model includes a low-level part and a high-level part.
[0063] Step 3: Sort each atom in each drug molecule in each mode based on the self-attention score of each atom in each mode to obtain the atom sorting result of each drug molecule, and mask the atoms that meet the preset masking number in the atom sorting result of each drug molecule to obtain the enhanced sample of the drug molecule in each mode.
[0064] Step 4: Construct positive sample pairs using the original sample and the enhanced sample of the drug molecule. There are multiple positive sample pairs.
[0065] Step 5: After dividing the original neural network model into two parts, comparative learning is performed simultaneously. The learning speed of each positive sample pair is adaptively controlled by calculating the comparative learning loss function value through the modal representation of each drug molecule until the preset training conditions are met, and the pre-training is completed to obtain the neural network model.
[0066] Step 6: Input the training data into the neural network for training to obtain a prediction model for drug property prediction.
[0067] In this embodiment of the invention, the original sample consists of various cancer drug molecules, and the modalities of the cancer drug molecules include one-dimensional fingerprint sequences, two-dimensional topological graphs, and three-dimensional spatial graphs.
[0068] For molecules, there are three modalities: one-dimensional fingerprint features, two-dimensional topological graph structure, and three-dimensional spatial graph structure. To achieve self-supervised comparative learning of molecular multimodalities, this invention first requires designing neural network models for these three different molecular modalities, specifically as follows: Figure 2 As shown, the neural network model includes:
[0069] First transform neural network module, first graph-level transform neural network module, second graph-level transform neural network module;
[0070] The input to the first transformation neural network module is a one-dimensional fingerprint sequence for each drug molecule;
[0071] The output of the first transform neural network module is a one-dimensional modal representation corresponding to each character in the fingerprint sequence of each drug molecule and the self-attention score of each atom in each drug molecule in the one-dimensional modality.
[0072] The input to the first graph-level transformation neural network module is the two-dimensional topological graph features of each drug molecule;
[0073] The output of the first graph-level transformation neural network module is the two-dimensional modal representation of each atom in each drug molecule and the self-attention score of each atom in each drug molecule in the two-dimensional modality.
[0074] The input to the second graph-level transformation neural network module is the three-dimensional spatial graph features of each drug molecule;
[0075] The output of the second graph-level transformation neural network module is the three-dimensional modal representation of each atom in each drug molecule and the self-attention score of each atom in each drug molecule in the three-dimensional modality.
[0076] Specifically, the original neural network model consists of two parts: a low-level part and a high-level part.
[0077] The lower-level components include: a lower-level transform neural network encoder in the first transform neural network module, a graph isomorphic neural network layer in the first graph-level transform neural network module, and a deep neural network layer in the second graph-level transform neural network module;
[0078] The high-level components include: the high-level transform neural network encoder in the first transform neural network module, the transform neural network encoder in the first graph-level transform neural network module, and the transform neural network encoder in the second graph-level transform neural network module.
[0079] Specifically, the first transformation neural network module includes:
[0080] A low-level transform neural network encoder and a high-level transform neural network encoder connected in sequence;
[0081] The one-dimensional fingerprint sequence of each drug molecule is input into a low-level transform neural network encoder for feature extraction, thereby obtaining a one-dimensional low-level modal representation corresponding to each character in the fingerprint sequence of each drug molecule.
[0082] The one-dimensional low-level modal representation is input into the high-level transform neural network encoder for feature extraction, resulting in a one-dimensional high-level modal representation corresponding to each character in the fingerprint sequence of each drug molecule. This representation is used to characterize the semantic information in the molecular fingerprint sequence and the self-attention score of each atom in each drug molecule under the one-dimensional modality.
[0083] To incorporate the position information of each atom, this embodiment of the invention first adds position information to the one-dimensional fingerprint sequence:
[0084]
[0085] in, To incorporate location information into the fingerprint sequence features, P m E represents the location information vector of the fingerprint sequence. m Represented as fingerprint sequence features;
[0086] For a low-level transform neural network encoder, taking drug molecule M as an example, the query vector of the drug molecule is first obtained. Target vector Sum value vector
[0087]
[0088]
[0089]
[0090] in, All of these represent learnable parameters. This represents a one-dimensional low-level modal characterization of drug molecule M;
[0091] To take into account multi-layered semantics, embodiments of the present invention employ a multi-head attention mechanism, the calculation formula of which is as follows:
[0092]
[0093]
[0094] in, Let represent the vector output by the i-th head in the multi-head attention mechanism, softmax(·) denotes the normalization function, and u represent the number of attention heads. Indicates learnable parameters, This represents the scaling factor.
[0095] After obtaining the output of the multi-head attention layer, the one-dimensional low-level modal representation is further processed by a high-level transform neural network to obtain the one-dimensional high-level modal representation E. 1d Its expression is:
[0096]
[0097] in, All represent learnable parameters, MultHead 1d This represents the output of the multi-head attention layer.
[0098] In this embodiment of the invention, both the low-level transform neural network encoder and the high-level transform neural network encoder are transform neural network encoders that incorporate a self-attention mechanism into a conventional transform neural network encoder.
[0099] Specifically, for the two-dimensional topological graph features of drug molecules, this embodiment of the invention employs a first graph-level transform neural network module, which includes:
[0100] A graph isomorphic neural network layer and a transform neural network encoder connected sequentially;
[0101] First, the two-dimensional topological graph features of each drug molecule are input into a graph isomorphic neural network for feature extraction, resulting in a two-dimensional low-level modal representation for each drug molecule. This representation is used to characterize the neighbor information of each atom in each drug molecule. The expression for the two-dimensional low-level modal representation of each drug molecule is as follows:
[0102]
[0103] in, This represents the two-dimensional modal representation of atom a in the (l+1)th layer. MLP stands for Multilayer Perceptron. This represents the two-dimensional modal characterization of atom a in the l-th layer. This represents the two-dimensional modal characterization of atom b in the l-th layer. This represents the neighbor bond characterization between atoms a and b in the l-th layer;
[0104] Then, a transform neural network encoder is used to mine long-range information. Before mining long-range information, the node degree representation and the shortest path distance representation of each atom of each drug molecule need to be obtained through the two-dimensional topological graph features of each drug molecule. The node degree representation serves as a supplementary vector for the position encoding in the feature learning process of the transform neural network encoder, and the shortest path distance representation of each atom serves as a supplementary vector for the attention score encoding in the feature learning process of the transform neural network encoder.
[0105] This invention first adds node degree information from a two-dimensional topological graph to the two-dimensional low-level modal representation, expressed as:
[0106]
[0107] in, Represents two-dimensional low-level modal features, characterized by multiple two-dimensional low-level modes. Composition, D m Represents node degree information;
[0108] The transformation neural network encoder applied to two-dimensional molecular data differs in its calculation of self-attention scores compared to its application to one-dimensional molecular data. It is necessary to incorporate the shortest path distance representation of atoms when calculating the multi-head attention mechanism.
[0109] The two-dimensional low-level modal representation of each drug molecule, the node degree information of each drug molecule, and the shortest path distance representation of each atom are input into the transform neural network encoder for feature extraction, so as to obtain the two-dimensional high-level modal representation of each drug molecule and the self-attention score of each atom in each drug molecule in the two-dimensional mode.
[0110] In this embodiment of the invention, the graph isomorphic neural network layer is a conventional graph isomorphic neural network.
[0111] Specifically, for the three-dimensional data of drug molecules, this embodiment of the invention employs a second graph-level transform neural network module, which includes:
[0112] A deep neural network layer and a transform neural network encoder connected in sequence;
[0113] First, the three-dimensional spatial features of each drug molecule are input into a deep neural network layer for feature extraction, resulting in a three-dimensional low-level modal representation for each drug molecule. The expression for the three-dimensional low-level modal representation of each drug molecule is as follows:
[0114]
[0115] in, This represents the low-level modal characterization of atom d in layer t+1. This represents the low-level modal characterization of atom v in layer t, and d represents the number of atoms in the drug molecule. e represents the learnable parameter. k r represents the distance characterization between atoms a and v with radial basis functions. a r represents the position coordinates of atom a. v This represents the position coordinates of atom v;
[0116] Before the low-level modal representation after deep neural network layer processing is input into the transform neural network encoder, the representation of the sum of the three-dimensional distances of atoms and the distance representation between atoms are obtained through the three-dimensional spatial graph features of each drug molecule. The representation of the sum of the three-dimensional distances of atoms serves as a supplementary vector for the position encoding of the transform neural network encoder during feature learning, and the distance representation between atoms serves as a supplementary vector for the attention score encoding of the transform neural network encoder during feature learning.
[0117] This invention first incorporates the sum of three-dimensional atomic distances into the characterization of low-level modes, expressed as:
[0118]
[0119] in, Represents three-dimensional low-level modal features, characterized by multiple three-dimensional low-level modes. composition, A representation of the sum of three-dimensional distances between atoms;
[0120] The transformation neural network encoder applied to three-dimensional molecular data differs in its calculation of self-attention scores from that applied to two-dimensional molecular data, requiring the inclusion of inter-atomic distance representation when calculating the multi-head attention mechanism;
[0121] The three-dimensional low-level modal representation of each drug molecule, the representation of the sum of the three-dimensional distances between atoms, and the representation of the distance between each atom are input into a transform neural network encoder for feature extraction, so as to obtain the three-dimensional high-level modal representation of each drug molecule and the self-attention score of each atom in each drug molecule in the three-dimensional mode.
[0122] In this embodiment of the invention, the deep neural network layer is a SchNet network layer that processes spatial graph information. SchNet is a deep learning framework that uses a continuous-filter convolutional neural network to predict the total energy and other related properties of molecules, providing fine spatial and chemical insights for resolving quantum mechanical observables of atomic systems.
[0123] To learn information about modal molecules, it is first necessary to construct an enhanced view of the molecules. Current methods for constructing enhanced molecular samples mainly include masking atoms, masking bonds, and masking subgraphs. However, all of these methods suffer from instability in the enhanced view. Therefore, this invention proposes a highly interpretable and reliable molecular data augmentation method to enhance the quality of positive sample pairs in contrastive learning. The process is as follows:
[0124] Obtain the self-attention score of each atom in each drug molecule under three modalities;
[0125] The self-attention scores of all atoms in a drug molecule are ranked by summing them column-wise. The expression is as follows:
[0126]
[0127] Where rank(i) represents the attention score ranking of atom i, r represents the proportion of covered atoms, and m i This represents the i-th atom in the molecule;
[0128] In order to enable the model to learn the core information of molecules through contrastive learning, this invention selects atoms in each modality that meet the preset number of masks for masking. The resulting enhanced samples get rid of the quality inhomogeneity problem caused by random masks, so as to stably obtain high-quality contrastive learning enhanced samples.
[0129] In this invention, original samples and their augmented samples from different modalities of the same drug molecule are treated as the same category during training. Sample pairs constructed in this way are considered positive sample pairs, and all other sample pairs are considered negative sample pairs. Thus, this invention combines intra-modal and cross-modal information to expand positive and negative samples for pre-training of self-supervised contrastive learning. At the same time, considering that different types of positive sample pairs composed of different positive samples from the same molecule have different learning difficulties, an adaptive method for controlling the convergence speed of different positive sample pairs is designed. Specifically, based on the ratio of the loss function values of each positive sample pair in the previous round and the previous two rounds, different ratios are given through normalization, thereby achieving adaptive control of the learning speed of different positive sample pairs.
[0130] Since this invention considers multiple modalities of the molecule, including one-dimensional, two-dimensional, and three-dimensional views, as well as multiple views of the molecule, including the original view and the enhanced view, inspired by the supervised contrastive learning loss function, this invention constructs an adaptive positive sample pair that can simultaneously bring different modalities and different angles closer together. Furthermore, it provides a multi-positive sample contrastive learning loss function with different learning weights for positive sample pairs with different learning difficulties. The expression is:
[0131]
[0132] Among them, AMCLOss p This represents the value of the contrastive learning loss function at round p. P represents the normalized score of the ratio of the loss function values of the previous round and the two rounds above the positive sample pair i and j at round p. M Let z represent the set of positive samples of drug molecule M. i Let z represent the representation vector of the drug molecule corresponding to the i-th positive sample pair in the positive sample set. j Other(P) represents the representation vector of the drug molecule corresponding to the j-th positive sample pair in the positive sample set. M z represents the set of samples of all molecules except for the positive sample set of drug molecule M. o Other(P) represents the sample set. M The representation vector of the drug molecule corresponding to the o-th sample in ).
[0133] In order to capture the shallow and deep semantic information of drug molecules, the embodiments of the present invention select the outputs of a low-level transform neural network encoder and a high-level transform neural network encoder for the first transform neural network module that processes one-dimensional data; select the outputs of a graph isomorphic neural network layer and a transform neural network encoder for the first graph-level transform neural network module that processes two-dimensional data; and select the outputs of a deep neural network layer and a transform neural network encoder for the second graph-level transform neural network module that processes three-dimensional data.
[0134] The main goal of contrastive learning is to simultaneously narrow the learning rates of different positive sample pairs. Since each positive sample pair consists of original or enhanced samples of different molecules, the difficulty of narrowing the learning rates of each positive sample pair varies. Therefore, this embodiment of the invention balances the learning rates of different positive sample pairs based on the loss change rate of different positive sample pairs during model pre-training, achieving dynamic balance of the learning rate of the contrastive learning loss function for different positive sample pairs. This prevents the model from only learning locally optimal parameters. In order to capture molecular information at both local and global levels and promote hierarchical alignment of the model, this embodiment of the invention applies an adaptive multi-positive sample contrastive learning loss function to pre-train in both the low-level and high-level parts. The pre-training process is as follows:
[0135] In the lower-level part, the basic semantic information of drug molecules is compared and learned through low-level modality representation, and the comparison learning loss function value is calculated by the multi-positive sample comparison learning loss function mentioned above to obtain the low-level comparison learning loss function value.
[0136] In the high-level part, the high-order syntactic information of drug molecules is compared and learned through high-level modal representation, and the comparison learning loss function value is calculated by the multi-positive sample comparison learning loss function mentioned above, so as to obtain the high-level comparison learning loss function value.
[0137] When the training iterations are reached, pre-training is completed, and the model parameters corresponding to the neural network model that minimize the sum of the low-level contrastive learning loss function value and the high-level contrastive learning loss function value are taken as the optimal parameters for learning. These optimal parameters are then used as the parameters of the neural network model for the next training step.
[0138] To address the issue of missing 3D data in downstream drug-related tasks, this embodiment of the invention, after pre-training, only selects the first transform neural network module and the first graph-level transform neural network in the neural network model used for mining one-dimensional and two-dimensional feature information, such as... Figure 3 As shown, the neural network model is fine-tuned using a dataset from downstream drug-related tasks to obtain a prediction model for drug property prediction. This model then enables the prediction of relevant drug properties, as detailed below:
[0139] This invention utilizes drug molecule data from the Cancer Drug Sensitivity Genomics (GDSC2) database, which is a comprehensive public resource for studying drug sensitivity and molecular markers of drug response.
[0140] Through screening and integration, a total of 805 cell lines' multi-omics transcriptional data, 154 drug SMILE sequences, and 103,492 sets of drug sensitivity data were obtained as raw samples.
[0141] For drug molecules used to treat cancer, one-dimensional and two-dimensional features of a neural network model were used to extract molecular features. For cell line features, the extraction modules of other methods were used, and the most basic feedforward neural network was used to extract features. Finally, the two parts of features were concatenated to obtain the final cancer drug sensitivity prediction results.
[0142] This invention compares multiple prediction methods, including MOLI, CDRscan, tCNNs, DeepCDR, and DeepTTA, and the comparison results are shown in Table 1 below:
[0143] Table 1
[0144] Method PCC SCC MOLI 81.3 78.2 CDRscan 87.1 85.2 tCNNs 91.0 88.9 DeepCDR 92.3 89.8 DeepTTA 94.1 91.4 Embodiments of the invention 94.3 93.7
[0145] Among them, MOLI (Multi-omics Data Integration Drug Response Predictor): improves drug response prediction performance by integrating multi-omics data and introducing a ternary loss function; CDRscan (Drug Response Profile Scan Predictor): uses molecular fingerprints for drug characterization and genomic mutation data as a cancer cell profile, then uses an ensemble convolutional neural network (CNN) for drug response prediction; tCNNs (Dual Convolutional Network Drug Response Predictor): uses two convolutional networks to learn the representation of drug and genomic mutation data, better capturing the features of drug and gene variations in prediction; DeepCDR (Graph Convolutional Network Drug Response Predictor): uses a unified graph convolutional network (UGCN) to learn the representation of drugs and integrates multi-omics data to improve drug response prediction performance; DeepTTA (End-to-End Drug Response Predictor): effectively processes and predicts the therapeutic effects of drugs on cancer by transforming the neural network architecture and considering the feature representation of drugs at the substructure level.
[0146] The embodiments of this invention evaluate the effectiveness of the method using the Pearson coefficient and the Spearman coefficient. As can be seen from Table 1 above, the prediction model proposed in the embodiments of this invention achieves state-of-the-art performance on all datasets and metrics. This indicates that the prediction model provided by the embodiments of this invention can effectively learn molecular knowledge from large-scale unlabeled molecular data, and the generated molecular characterization has strong generalization ability, which can effectively contribute to drug-related tasks.
[0147] In summary, this invention obtains multiple drug molecules with various modalities as original samples; for each drug molecule, the original samples are input into an original neural network model, and modal representations of each drug molecule are extracted from the original neural network model. The self-attention score of each atom in each drug molecule under each modality is calculated. Based on the self-attention score of each atom under each modality, the atoms in each drug molecule under each modality are sorted to obtain the atom sorting result for each drug molecule. Atoms meeting a preset masking number are masked in the atom sorting result for each drug molecule to obtain enhanced samples for each modality. Positive sample pairs are constructed using the original samples and enhanced samples of the drug molecules. The original neural network model is divided into two parts and comparative learning is performed synchronously. Furthermore, the enhanced samples are used to construct positive sample pairs for each drug molecule. The modal representation of molecules is used to calculate the contrastive learning loss function value to adaptively control the learning speed of each positive sample pair until the preset training conditions are met, thus completing pre-training and obtaining a neural network model. The training data is then input into the neural network model for training, resulting in a prediction model for drug property prediction. Compared with existing technologies, this invention considers multiple modalities of drug molecules, extracts modal representations under multiple modalities, and calculates the self-attention score of each atom under multiple modalities to construct high-quality augmented samples, which can reduce low-quality molecular augmented samples caused by unreasonable masks. By calculating the contrastive learning loss function value through the modal representation of each drug molecule, the learning speed of molecular knowledge learning during the pre-training process is adaptively controlled, preventing the model from getting trapped in local optima and improving the training accuracy of the model, thereby improving the accuracy of drug property prediction.
[0148] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A molecular multi-angle comparative learning pre-training method for drug property prediction, characterized in that, include: Step 1: Obtain multiple drug molecules including multiple modalities as raw samples, wherein the modalities include one-dimensional fingerprint sequences, two-dimensional topological maps, and three-dimensional spatial maps; Step 2: For each drug molecule, the original sample is input into the original neural network model, the modal representation of each drug molecule is extracted from the original neural network model, and the self-attention score of each atom in each drug molecule under each modality is calculated. Step 3: Sort each atom in each drug molecule in each mode based on the self-attention score of each atom in each mode to obtain the atom sorting result of each drug molecule, and mask the atoms that meet the preset masking number in the atom sorting result of each drug molecule to obtain the enhanced sample of the drug molecule in each mode. Step 4: Construct positive sample pairs using the original sample of the drug molecule and the enhanced sample, wherein the positive sample pairs include multiple pairs; Step 5: After dividing the original neural network model into two parts, comparative learning is performed simultaneously. The learning speed of each positive sample pair is adaptively controlled by calculating the comparative learning loss function value through modal representation of each drug molecule until the preset training conditions are met, thus completing the pre-training and obtaining the neural network model. The expression for calculating the comparative learning loss function value is as follows: in, Indicates the first The contrastive learning loss function value at each round. Indicates positive sample pairs and Between the first The normalized fraction of the ratio of the loss function values of the previous round and the two rounds before the round. Indicates drug molecules The set of positive samples Indicates the positive sample set that is related to the first... The representation vector of the corresponding drug molecule for each positive sample pair. Indicates the positive sample set that is related to the first... The representation vector of the corresponding drug molecule for each positive sample pair. In addition to drug molecules The set of samples of all other molecules besides the positive sample set. Represents the sample set Middle and the first The representation vector of the drug molecule corresponding to each sample; Step 6: Input the training data into the neural network for training to obtain a prediction model for drug property prediction.
2. The molecular multi-angle comparative learning pre-training method for drug property prediction according to claim 1, characterized in that, The neural network model includes: First transform neural network module, first graph-level transform neural network module, second graph-level transform neural network module; The input to the first transform neural network module is a one-dimensional fingerprint sequence for each drug molecule; The output of the first transform neural network module is a one-dimensional modal representation corresponding to each character in the fingerprint sequence of each drug molecule and the self-attention score of each atom in each drug molecule in the one-dimensional modality; The input to the first graph-level transformation neural network module is the two-dimensional topological graph features of each drug molecule; The output of the first graph-level transform neural network module is the two-dimensional modal representation of each atom in each drug molecule and the self-attention score of each atom in each drug molecule in the two-dimensional modality; The input to the second graph-level transformation neural network module is the three-dimensional spatial graph features of each drug molecule; The output of the second graph-level transformation neural network module is the three-dimensional modal representation of each atom in each drug molecule and the self-attention score of each atom in each drug molecule in the three-dimensional modality.
3. The molecular multi-angle comparative learning pre-training method for drug property prediction according to claim 2, characterized in that, The original neural network model consists of two parts: a low-level part and a high-level part. The lower-level part includes: a lower-level transform neural network encoder in the first transform neural network module, a graph isomorphic neural network layer in the first graph-level transform neural network module, and a deep neural network layer in the second graph-level transform neural network module; The high-level component includes: a high-level transform neural network encoder in the first transform neural network module, a transform neural network encoder in the first graph-level transform neural network module, and a transform neural network encoder in the second graph-level transform neural network module.
4. The molecular multi-angle comparative learning pre-training method for drug property prediction according to claim 3, characterized in that, The first transform neural network module includes: A low-level transform neural network encoder and a high-level transform neural network encoder connected in sequence; The one-dimensional fingerprint sequence of each drug molecule is input into the low-level transform neural network encoder for feature extraction, thereby obtaining a one-dimensional low-level modal representation corresponding to each character in the fingerprint sequence of each drug molecule. The one-dimensional low-level modal representation is input into the high-level transform neural network encoder for feature extraction, resulting in a one-dimensional high-level modal representation corresponding to each character in the fingerprint sequence of each drug molecule and the self-attention score of each atom in each drug molecule under the one-dimensional modality.
5. The molecular multi-angle comparative learning pre-training method for drug property prediction according to claim 4, characterized in that, The one-dimensional high-level modal representation The expression is: in, , , , All of these represent learnable parameters. This represents the output of the multi-head attention layer.
6. The molecular multi-angle comparative learning pre-training method for drug property prediction according to claim 5, characterized in that, The first graph-level transform neural network module includes: A graph isomorphic neural network layer and a transform neural network encoder connected sequentially; The two-dimensional topological graph features of each drug molecule are input into the graph isomorphic neural network for feature extraction, thereby obtaining a two-dimensional low-level modal representation of each drug molecule. The node degree representation and the shortest path distance representation of each atom are obtained by using the two-dimensional topological graph features of each drug molecule. The node degree representation serves as a supplementary vector for the position encoding in the feature learning process of the transform neural network encoder, and the shortest path distance representation of each atom serves as a supplementary vector for the attention score encoding in the feature learning process of the transform neural network encoder. The two-dimensional low-level modal representation of each drug molecule, the node degree information of each drug molecule, and the shortest path distance representation of each atom are input into the transform neural network encoder for feature extraction, thereby obtaining the two-dimensional high-level modal representation of each drug molecule and the self-attention score of each atom in each drug molecule in the two-dimensional mode.
7. The molecular multi-angle comparative learning pre-training method for drug property prediction according to claim 6, characterized in that, The expression for the two-dimensional low-level modal characterization of each drug molecule is as follows: in, Represents atoms In the Two-dimensional modal characterization of the layer, This represents a multilayer perceptron. Represents atoms In the Two-dimensional modal characterization of the layer, Represents atoms In the Two-dimensional modal characterization of the layer, Represents atoms and In the Neighbor key representation of a layer.
8. The molecular multi-angle contrastive learning pre-training method for drug property prediction according to claim 7, characterized in that, The second graph-level transform neural network module includes: A deep neural network layer and a transform neural network encoder connected in sequence; The three-dimensional spatial features of each drug molecule are input into the deep neural network layer for feature extraction, thereby obtaining a three-dimensional low-level modal representation of each drug molecule. The sum of the three-dimensional distances between atoms and the distance between atoms are obtained by using the three-dimensional spatial map features of each drug molecule. The sum of the three-dimensional distances between atoms is used as a supplementary vector for the position encoding of the transform neural network encoder during feature learning, and the distance between atoms is used as a supplementary vector for the attention score encoding of the transform neural network encoder during feature learning. The three-dimensional low-level modal representation of each drug molecule, the representation of the sum of the three-dimensional distances between atoms, and the representation of the distance between each atom are input into the transform neural network encoder for feature extraction, so as to obtain the three-dimensional high-level modal representation of each drug molecule and the self-attention score of each atom in each drug molecule in the three-dimensional mode.
9. The molecular multi-angle comparative learning pre-training method for drug property prediction according to claim 8, characterized in that, The expression for the three-dimensional low-level modal characterization of each drug molecule is as follows: in, Represents atoms In the Low-level modal characterization of the layer, Represents atoms In the Low-level modal characterization of the layer, Indicates the number of atoms in a drug molecule. Indicates learnable parameters, This represents atoms with radial basis functions. and Distance representation between them Represents atoms Location coordinates, Represents atoms The location coordinates.