Molecular property prediction method based on dual perception large language model

Abstract

The invention provides a molecular property prediction method based on a dual-perception large language model, which is characterized in that a multi-level instruction of task guidance, chemical perception and structure perception is designed, and the semantic comprehension capability of the large language model is fully utilized to obtain high-quality molecular embedding representation. Meanwhile, dual strategies of average pooling and whitening dimension reduction are combined: the average pooling effectively integrates the comprehensive contribution of each feature of the molecules, and key information loss is avoided; the isotropy of the feature space is remarkably improved through whitening dimension reduction, and optimization compression of the feature dimension is achieved at the same time. The subsequent calculation complexity is greatly reduced, and more accurate feature representation is provided for various downstream prediction tasks. And finally, comprehensive experiments are carried out on a real data set, so that the effectiveness of the method is proved, and the performance of molecular property prediction can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biological information processing, in particular to a molecular property prediction method based on a dual perception large language model. BACKGROUND

[0002] In the development of medical and pharmaceutical fields, molecules with ideal functional properties play a crucial role. However, drug discovery is a long and costly process, and the failure rate of traditional drug discovery has been high in the past few decades. With the rapid development of artificial intelligence and computing resources, in the process of accelerating drug discovery, molecular property prediction (MPP) uses machine learning or deep learning techniques to predict the functional properties of molecules, which has been proven to be an effective method. This method not only helps researchers evaluate drugs more quickly and accurately. It also helps pharmacists more accurately identify the pharmacokinetic and pharmacodynamic properties of drugs, thereby accelerating the development process of new drugs.

[0003] Molecular characterization is the first step of drug discovery, and its quality directly affects the performance of subsequent drug discovery methods. In the past period of time, molecular characterization can be mainly summarized into four categories: molecular fingerprint, SMILES representation of molecule, molecular graph, and molecular three-dimensional structure. GIMLET realizes zero-shot learning of molecular property prediction by combining molecular graph and text information and using instruction-guided method. LLM4SD extracts general rules from literature knowledge through large language models (LLMs), and combines large-scale molecular scientific data to infer knowledge related to the predicted target characteristics, thereby training standard machine learning models. MolecularGPT applies LLMs to the molecular field by using instruction fine-tuning strategy, and realizes efficient molecular property prediction under small sample conditions. SMILES-BERT uses self-supervised pre-training strategy to learn transferable molecular representation from large-scale unlabeled data by reconstructing randomly masked SMILES. MolFormer makes key improvements based on SMILES-BERT, introduces a rotation embedding mechanism, models the relative position information between atoms in the molecular sequence, and generates more effective molecular embedding, so that the model can more accurately capture the structure-property correlation. Uni-Mol integrates 3D molecular conformation and graph neural network (GNN), and realizes all-round modeling of molecular real space conformation and interaction by integrating local atomic environment and global spatial geometric features. GALLON uses a multi-modal fusion framework, integrates the semantic understanding ability of LLMs for SMILES and the representation advantage of graph neural network (GNN) for molecular graph structure through knowledge distillation, and uses MLP for cross-modal feature learning, realizing complementary representation of molecular information. However, there are still the following problems: i) complex architecture and a large amount of optimization are needed to achieve the best performance; ii) a large amount of computing resources is needed; iii) a large number of sample labels are needed to learn the representation of molecules. SUMMARY

[0004] To solve the above problems, the present application provides a molecular property prediction method based on a dual perception large language model, which can effectively improve the performance of molecular property prediction. To achieve the above purpose, the technical scheme of the present application is as follows:

[0005] A molecular property prediction method based on a dual perception large language model, comprising the following steps: (1) obtaining the action mechanism of the relevant task to constitute the instructions of the task guiding part; (2) obtaining the chemical description of the molecules of the drug by the large language model to constitute the instructions of the chemical perception part; (3) calculating the MACCS fingerprint of all drug molecules, and calculating the similarity of each molecule with other molecules, screening the molecules with a similarity greater than 0.7 with the current molecule to constitute the instructions of the structure perception part; (4) effectively fusing all instruction information into the large language model to perform zero-shot embedding and average pooling; (5) performing whitening dimension reduction on the embedding vector to obtain the final feature; (6) using the obtained final feature to perform prediction.

[0006] In step (1), the action mechanism of the relevant task of each data set is obtained by querying professional chemical literature, and is converted into task guiding instructions that can be understood by the large language model.

[0007] In step (2), the SMILES representation of the molecule, the specific task information, and the question that needs to be answered by the large language model are combined to construct the chemical perception instructions. The large language model is inputted, and the answer of the large language model is obtained and converted into chemical perception instructions that can be understood by the large language model.

[0008] In step (3), the molecular structure is converted into a binary fingerprint representation by the RDKit toolkit, the structural similarity of the query molecule with the molecules in the database is calculated using the Tanimoto coefficient, and the highest similarity candidate molecules are screened. To ensure the reliability of the results, the system implements double quality control: first, a strict similarity threshold is set to screen high correlation molecules, and second, when the number of candidate molecules is insufficient, the quality is prioritized over the quantity to avoid introducing noise, thereby obtaining a set of most similar molecules with reliable structural correlation, which is converted into structure perception instructions that can be understood by the large language model.

[0009] In step (4), the three instructions obtained above are combined and carefully designed into natural language instructions that can be understood by the large language model; the high-level semantic embedding of the last hidden state of the large language model is extracted, and the average pooling strategy is used to aggregate all hidden state features to obtain more compact and robust representations.

[0010] In step (5), the embedding vectors after average pooling are centered to eliminate the global bias of data distribution; then the covariance matrix is calculated to analyze the correlation between features, and the whitening transformation matrix is obtained by singular value decomposition of the covariance matrix; finally, all embedding vectors are whitened and dimensionally reduced to obtain the final feature vector.

[0011] In the step (6), the molecular features after whitening dimension reduction are input into a fully connected neural network for a prediction task; for a classification problem, the network outputs classification probabilities through linear transformation of hidden layers and output layers combined with Softmax normalization, and adopts a cross-entropy loss function to evaluate prediction accuracy; for a regression task, the network directly outputs a prediction value, and uses a mean square error loss to measure the deviation between the prediction value and the true value.

[0012] The present application has the beneficial effects that the present application proposes a molecular property prediction method, which fully utilizes the semantic understanding ability of a large language model to obtain high-quality molecular embedding representation by designing multi-level instructions of task guidance, chemical perception and structure perception. At the same time, the double strategies of average pooling and whitening dimension reduction are combined: the average pooling effectively integrates the comprehensive contribution of each feature of the molecule, avoiding the loss of key information; the whitening dimension reduction significantly improves the isotropy of the feature space, and at the same time realizes the optimization and compression of the feature dimension. The subsequent calculation complexity is greatly reduced, and more accurate feature representation is provided for various downstream prediction tasks. Finally, a large number of comprehensive experiments are carried out on real data sets, and the experimental results show that the AUC and RMSE indicators of the present method are significantly improved compared with other advanced models. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is the overall framework diagram of the present application;

[0014] Figure 2 is a comparison graph of whitening dimension reduction hyperparameter experiments. DETAILED DESCRIPTION

[0015] The present application will be further described below in conjunction with the drawings and examples. It should be understood that these examples are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent forms also fall within the scope defined by the present application.

[0016] The overall framework of the molecular property prediction method based on the double perception large language model proposed by the present application is shown in Figure 1 , which includes the following steps:

[0017] (1) Obtain the action mechanism of the related task to constitute the instructions of the task guidance part. Step (1) includes the following steps:

[0018] Collect the action mechanisms of various tasks from professional chemical literature, which include the physicochemical properties, structural properties and binding mechanisms with receptors of molecules. These contents are integrated and converted into instructions that can be understood by the large language model.

[0019] (2) Obtain the chemical description of the drug molecule by the large language model, and constitute the instruction of the chemical perception part. Step (2) includes the following steps:

[0020] Combine each molecule corresponding to the task, specific task information, and questions that need to be answered by the large language model to construct the chemical perception instruction. Then, input these instructions into the large language model in turn to obtain the corresponding answers:

[0021] R i = LLM(S i , T i , P i )

[0022] Where S i is the SMILES representation of the molecule, T i is the specific task information of the data set to which the molecule belongs, and P i is the question answered by the large language model. LLM(·) is the large language model, and R i is the chemical description of the molecule by the large language model.

[0023] (3) Calculate the MACCS fingerprint of all drug molecules, and calculate the similarity between each molecule and other molecules, and screen the molecules with similarity greater than 0.7 to the current molecule to constitute the instruction of the structure perception part. Step (3) includes the following steps:

[0024] Use the RDKit toolkit to process the SMILES string of each molecule to calculate its MACCS fingerprint. In the molecular similarity retrieval task, for a given query molecule, first use the MACCS fingerprint-based Tanimoto similarity measurement method to select the top k candidate molecules with the highest structural similarity to the query molecule from the target data set. The Tanimoto coefficient can be expressed as:

[0025]

[0026] Where A q is the MACCS fingerprint of the query molecule, A i is the MACCS fingerprint of the queried molecule in the corresponding data set. T(A q , A i ) is the Tanimoto coefficient between the query molecule and the queried molecule. Sort T(A q , A i ) obtained for each query molecule:

[0027] N(S q ) = {S1, S2, …, S k}

[0028] S qS k is the k-th ranked molecule in the corresponding dataset by similarity. N(S q ) is the neighborhood set of query molecule S q , in which the similarity of molecules to the query molecule is ranked from high to low. To guarantee the structural relevance of neighborhood molecules, we adopt a double quality control strategy: first, we set a strict similarity threshold (T(A q , A i ) ≥ 0.7) to filter high-similarity molecule pairs; second, when the number of candidate molecules in the database that meet the threshold requirement is insufficient for the pre-set k value, the system will follow the principle of "quality over quantity". By only outputting qualified molecules without quantity compensation, we effectively avoid introducing additional noise, thus ensuring that each selected neighborhood set of molecules has reliable structural relevance.

[0029] (4) All instruction information is effectively fused into the instruction input into the large language model, zero-shot embedding is performed, and the obtained embedding vectors are averaged and pooled. Step (4) includes the following steps:

[0030] Fuse the SMILES representation of the above-mentioned instructions and the molecule itself, carefully design natural language instructions that can be understood by the large language model, and convert molecular structure features and chemical property constraints and other professional knowledge into semantic instructions that can be directly executed by the large language model. The obtained instructions are input into the large language model to obtain the embedding of the large language model. The embedding of the last layer of the hidden layer of the large language model is better at capturing high-level semantic features, so we extract the embedding of the last layer.

[0031] z i = f LLM (S i , T i , R i , N(S q ))

[0032] where z i is the embedding of the i-th molecule, and f LLM (·) is the hidden layer mapping of the large language model. The dimension of the obtained feature matrix z i is 4096x1024, that is, each molecule has 4096 features, and each feature has 1024 dimensions.

[0033] In the feature aggregation stage, we use the average pooling strategy to process the extracted embedding, and generate a fixed-dimensional global vector representation by calculating the average value of all hidden states. This method can effectively integrate the feature contributions of each token, and avoid dilution of key features by local noise or non-important tokens. For the feature matrix z i, the average-pooled vector z avg,i which can be expressed as:

[0034]

[0035] where z i [:,j] represents the jth column of the ith molecular feature matrix. z avg,i is the average-pooled vector, with a dimension of 4096x1.

[0036] The embedding vectors are whitened and reduced in dimension to obtain the final features. Step (5) includes the following steps:

[0037] After obtaining the final features, in order to reduce the computational overhead of downstream tasks and solve the anisotropy of sentences output by large language models. The specific performance is that high-frequency words are concentrated in distribution, and low-frequency words are scattered in distribution, and the entire vector space is similar to a conical structure. The method of whitening and dimension reduction is adopted to enhance isotropy while reducing dimension. For the feature set of the molecule, first calculate the mean of the feature vector set, and center the data distribution to the origin, so as to eliminate the global bias:

[0038]

[0039] where N is the total number of molecules in the dataset, and Λ is the mean of the vector. Next, the covariance matrix is calculated, which describes the distribution of the data and is the core of the whitening operation:

[0040]

[0041] where ∑ is the covariance matrix. This formula calculates the outer product of the difference between each sample and the mean, and then averages all samples to obtain a symmetric and positive definite covariance matrix for subsequent whitening transformation. Since the covariance matrix ∑ is a symmetric positive definite matrix, it can be singular value decomposed (SVD):

[0042] ∑ = UΛU T

[0043] where U is an orthogonal matrix composed of eigenvectors, and Λ is a diagonal matrix of eigenvalues containing the eigenvalues of the corresponding eigenvectors. Then calculate the inverse square root matrix of singular values and the whitening matrix W:

[0044]

[0045]

[0046] where σ1, σ2,..., σ k are the first k singular values of the covariance matrix ∑. Λ kis a diagonal matrix consisting of the first k largest singular values of the covariance matrix ∑. is the inverse square root of k , i.e., a diagonal matrix. W is a whitening matrix, which is obtained by multiplying the first k columns of U and .

[0047] Finally, each embedding vector z i is whitened and reduced in dimension:

[0048]

[0049] where is the reduced embedding vector with dimension k. In this way, not only is the data reduced in dimension, but it is also enhanced in isotropy, providing better features for subsequent downstream tasks.

[0050] (6) The final features obtained are input into a fully connected layer for prediction. Step (6) includes the following steps:

[0051] The final features obtained above are input into a fully connected layer for prediction, as follows for a classification task:

[0052]

[0053] where is the input feature matrix, N represents the number of samples, and k represents the feature dimension of each sample; represents a weight matrix that maps the input features from k dimensions to h dimensions, and h represents the number of neurons in the hidden layer. b1 represents the bias vector of the hidden layer, providing an offset for each hidden unit. represents a weight matrix that maps the output of the hidden layer from h dimensions to C dimensions (the number of classes). b2 represents the bias vector of the output layer, providing an offset for each class. Softmax(·) is a normalization, function, and

[0054] The cross-entropy loss function is used to calculate the classification loss, and the cross-entropy loss function is defined as follows:

[0055]

[0056] where y i,c is the true label of sample i. If sample i belongs to class c, then y i,c = 1, otherwise 0. c is the probability that the model predicts that sample i belongs to class c, is the cross-entropy loss.

[0057] For a regression task, as follows:

[0058]

[0059] wherein, is the predicted value vector, and the loss function is:

[0060]

[0061] wherein, y i is the true value of the sample i, is the predicted value of the sample i by the model. is the mean square error loss.

[0062] Embodiment: In this embodiment, the molecular property prediction method provided by the present application is verified by experiment, and the following describes the data set, experimental setting, evaluation index, comparison method and experimental results of this embodiment in detail.

[0063] Data set and experimental setting: The present application uses 9 data sets in MoleculeNet in the real world for experiment, and the data set specific content includes data set name, task type, task number, molecule number, evaluation index, etc. As shown in Table 1, according to the requirements of the disclosed data set and the field, all data sets are divided by molecular scaffolds, and the division ratio of all data sets is: the ratio of training set, validation set and test set is 8:1:1.

[0064] Table 1 Data set specific content

[0065] Evaluation index: In order to evaluate the effect of the model, the present application adopts the following two widely used evaluation parameters: for classification task, the area under the receiver operating characteristic curve (AUC) is adopted, and for regression task, the root mean square error (RMSE) is adopted.

[0066] Area under the receiver operating characteristic curve: the value of AUC is between 0 and 1, and the calculation method of AUC is:

[0067]

[0068] wherein, TPR is the true positive rate, FPR is the false positive rate. AUC is the predicted value, and the higher the value represents the higher the accuracy of the model prediction, and the better the result.

[0069]

[0070]

[0071] Where FP is the false positive (the number of samples that are actually negative but are predicted as positive by the model), TN is the true negative (the number of samples that are actually negative and are correctly predicted as negative by the model), TP is the true positive (the number of samples that are actually positive and are correctly predicted as positive by the model), and FN is the false negative (the number of samples that are actually positive but are incorrectly predicted as negative by the model).

[0072] Root Mean Square Error: RMSE represents the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of observations, reflecting the dispersion of the samples. The calculation method of RMSE is:

[0073]

[0074] Where yi represents the i-th predicted value, and yi represents the i-th true value. The smaller the value of RMSE, the higher the accuracy of the model prediction, and the better the results. pred,i true,i

[0075] Comparison method:

[0076] [1] Pinheiro G A, Da Silva J L F, Quiles M G. Smiclr: Contrastive learning on multiple molecular representations for semi-supervised and unsupervised representation learning [J]. Journal of Chemical Information and Modeling, 2022, 62(17): 3948-3960.

[0077] Literature [1] learns contrastive representation by jointly training graph neural networks and long short-term memory networks from molecular graph representation and SMILES string representation. A variety of data augmentation techniques (such as node dropout, subgraph extraction, edge perturbation, etc.) are used to generate positive sample pairs, and normalized temperature scaling cross-entropy loss is used to optimize the representation of positive sample pairs to be closer and the representation of negative sample pairs to be farther apart.

[0078] [2] Zhou G, Gao Z, Ding Q, et al. Uni-mol: A universal 3D molecular representation learning framework [J]. 2023.

[0079] ​​Reference [2] adopts a backbone network based on SE(3)-equivariant Transformer, which is pre-trained on large-scale 3D molecular conformation data. Uni-Mol learns the 3D spatial representation of molecules, enabling it to effectively process 3D inputs and predict 3D positions. In the fine-tuning phase, Uni-Mol uses the [CLS] token representation to train a linear classifier to adapt to specific molecular property prediction tasks.

[0080] [3] Han S, Fu H, Wu Y, et al. HimGNN: a novel hierarchical molecular graph representation learning framework for property prediction [J]. Briefings in Bioinformatics, 2023, 24(5): bbad305.

[0081] Reference [3] learns hierarchical features of molecules through atomic-level and primitive-level graph representations. First, the molecule is represented as an atomic-level graph, and the features of atoms and chemical bonds are initialized. Then, by extracting primitives in the molecule, a primitive-level graph is constructed. Primitives are collections of atoms organized according to chemical rules, and their initial features are the accumulation of constituent atomic features. MPNN modules are used to perform message passing on atomic-level and primitive-level graphs, generating messages for neighboring nodes and performing cross-attention operations through a Transformer architecture to integrate atomic information into primitive-level graph message passing, enhancing the distinguishability of primitive features. In addition, the model adjusts the scaling strength of atomic and primitive representations through a context-adaptive rescaling mechanism to accurately predict the hierarchical dependencies of molecules.

[0082] [4] Jiang T, Wang Z, Yu W, et al. Mix-Key: graph mixup with key structures for molecular property prediction [J]. Briefings in Bioinformatics, 2024, 25(3): bbae165.

[0083] Reference [4] enhances molecular graph representation by combining a multi-view graph generator and Mixup operation. First, two isomers are generated from the original molecular graph, and then Mixup operation is performed based on molecular fingerprint similarity and node representation similarity to generate enhanced graphs. During training, the model is retrained using the original and enhanced graphs.

[0084] Experimental results and analysis: Table 2 gives all molecular property prediction evaluation results of the classification task, and Table 3 gives all molecular property prediction evaluation results of the classification task.

[0085] Table 2 Comparison of classification tasks with other methods

[0086] Table 3 Comparison of regression tasks with other methods

[0087] According to the experimental results, the following observations can be made: from Table 2, it can be seen that the AUC index of the present application on the BACE, BBBP, ClinTox, HIV, SIDER and Tox21 six benchmark data sets is better than that of the comparison method, and the performance on the ClinTox data set is particularly outstanding, with an AUC of 97.8%, which is 5.9 percentage points higher than the optimal comparison method (91.9%), fully embodying the excellent performance of the present application in the toxicity prediction task. On the BBBP data set, the AUC value of the present application is as high as 97.6%, which is 1.9 percentage points higher than the suboptimal method (95.7%), showing excellent blood-brain barrier permeability prediction ability.

[0088] From Table 3, it can be seen that the RMSE index of the present application on the ESOL, FreeSolv and Lipophilicity three data sets reaches the optimal. In particular, in the ESOL water solubility prediction task, the RMSE of the present application is 0.633, which is 16.2% lower than the optimal comparison method (0.755), significantly improving the prediction accuracy. In the FreeSolv hydration free energy prediction task, the RMSE of the present application is 1.270, which is further optimized than the suboptimal method (1.285), showing excellent regression prediction ability.

[0089] Hyperparameter comparison analysis: in the present application, when performing dimensionality reduction on the embedding vector, the target dimension parameter after dimensionality reduction needs to be reasonably selected. Therefore, the dimension size after dimensionality reduction is adjusted in the range of {64, 128, 256, 512}. The main role of the embedding layer is to convert the high-dimensional input vector into a low-dimensional vector for subsequent processing. The dimension size has a significant impact on the amount of information obtained by the model from the input vector. Generally speaking, a larger embedding dimension can enable the model to obtain more abundant information, but if the embedding dimension is too large, it will not only increase the amount of data that the model needs to process, but also violate the original intention of data dimensionality reduction, which may further reduce the performance of the model. From Table 4, it can be seen that the dimension size has a significant impact on the performance of the model. Figure 2 From Table 4, it can be seen that the dimension size has a significant impact on the performance of the model.

Claims

1. A method for predicting molecular properties based on a dual-sensory large language model, characterized in that, Includes the following steps: (1) Obtain the mechanism of action of relevant tasks to form the instructions of the task guidance part; (2) Obtain the chemical description of drug molecules by the large language model to form the instructions of the chemical perception part; (3) Calculate the MACCS fingerprint of all drug molecules and calculate the similarity of each molecule with other molecules, and screen molecules with a similarity greater than 0.7 with the current molecule to form the instructions of the structure perception part; (4) Effectively integrate all the instruction information into the instruction input of the large language model, perform zero-sample embedding and average pooling; (5) Whiten and reduce the dimension of the embedding vector to obtain the final features. (6) Use the obtained final features to make predictions.

2. The molecular property prediction method based on a dual-sensory large language model according to claim 1, characterized in that, In step (1), the mechanism of action of the relevant tasks in each dataset is obtained by querying professional chemical literature and transformed into task guidance instructions that can be understood by the large language model.

3. The molecular property prediction method based on a dual-sensory large language model according to claim 1, characterized in that, In step (2), the SMILES representation of the molecule, specific task information, and the questions that need to be answered by the large language model are combined to construct chemical perception instructions. These instructions are then input into the large language model, and the model's response is obtained and converted into chemical perception instructions that the large language model can understand.

4. The molecular property prediction method based on a dual-sensory large language model according to claim 1, characterized in that, In step (3), the molecular structure is converted into a binary fingerprint representation using the RDKit toolkit. The structural similarity between the queried molecule and the molecules in the database is calculated using the Tanimoto coefficient, and the candidate molecules with the highest similarity are selected. To ensure the reliability of the results, the system implements dual quality control: first, a strict similarity threshold is set to select highly correlated molecules; second, when the number of candidate molecules is insufficient, quality is prioritized over quantity to avoid introducing noise, thereby obtaining the set of the most similar molecules with reliable structural correlation, and converting them into structural perception instructions that can be understood by a large language model.

5. The molecular property prediction method based on a dual-sensory large language model according to claim 1, characterized in that, In step (4), the three instructions obtained above are combined and carefully designed into natural language instructions that the large language model can understand; input into the large language model, extract the high-level semantic embedding of its last hidden state, and in order to obtain a more compact and robust representation, the average pooling strategy is used to aggregate all hidden state features.

6. The molecular property prediction method based on a dual-sensory large language model according to claim 1, characterized in that, In step (5), the embedded vectors after average pooling are centered to eliminate the global bias of the data distribution; then the covariance matrix is ​​calculated to analyze the correlation between features, and the whitening transformation matrix is ​​obtained by singular value decomposition of the covariance matrix; finally, all embedded vectors are whitened and dimensionality reduced to obtain the final feature vector.

7. The molecular property prediction method based on a dual-perception large language model according to claim 1, characterized in that, In step (6), the whitened and dimensionality-reduced molecular features are input into a fully connected neural network for prediction. For classification problems, the network outputs the probability of each classification by combining linear transformation of the hidden layer and the output layer with Softmax normalization, and uses the cross-entropy loss function to evaluate the prediction accuracy. For regression tasks, the network directly outputs the predicted value and uses the mean squared error loss to measure the deviation between the predicted value and the true value.

Images