A method for constructing a large language model for molecular property prediction based on knowledge graph

Abstract

The present invention discloses a method for constructing a large language model for predicting molecular properties based on a knowledge graph. The method includes multiple steps: first, high-throughput data acquisition technology is used to obtain pre-training corpus, including relevant data such as molecular properties and metabolic pathways. Then, based on these data, a knowledge graph about fourteen endpoints is constructed, and the knowledge graph represents the relationship between molecules, their properties and endpoints in the form of triples. Next, the large language model is pre-trained using LoRA technology, and the model is optimized using an incremental learning strategy to form an expert large model. Subsequently, RAG technology is used to retrieve triplets related to the target endpoint from the knowledge graph, and the retrieval results are sorted. Through prompt word design, the expert large model filters the retrieved triplets and extracts key information. Finally, the filtered triple list is converted into natural language input to the expert large model to generate prediction results of molecular properties and provide a basis for reasoning. This method significantly improves the accuracy and efficiency of predicting chemical reactions between molecules and endpoints.

Description

Technical Field

[0001] The present invention belongs to the interdisciplinary field of pharmacy and artificial intelligence, and specifically relates to a method and technology for constructing a large language model for molecular property prediction based on a knowledge graph. Background Art

[0002] When it comes to predicting drug molecular properties, information such as their physicochemical properties, pharmacological characteristics, and toxicity is crucial in the drug development process. Traditional methods for predicting drug molecular properties rely primarily on physical and chemical methods such as quantum chemical calculations and molecular dynamics simulations. While accurate, these methods are computationally complex and time-consuming, making them difficult to adapt to the needs of large-scale drug screening.

[0003] In recent years, the rapid development of machine learning and deep learning technologies has provided new methods for predicting drug molecular properties. However, existing prediction models based on deep learning often rely on a large amount of labeled data for training. Although certain breakthroughs have been made in predictive performance, the following problems still exist: (1) Most existing models rely on large-scale, structured drug molecular data sets, which places high demands on data acquisition and labeling. Especially in the new drug development stage, the lack of labeled data limits the generalization ability of the model. (2) Models usually make predictions based only on molecular structure information, and it is difficult to effectively utilize external knowledge such as existing drug knowledge bases and literature, thereby limiting the model's predictive ability in small sample conditions. (3) Different models are often required to predict different physicochemical properties, pharmacokinetic properties, etc. of drugs, and there is a lack of a unified framework to simultaneously handle multiple drug molecular properties.

[0004] Based on this, a method for constructing a large language model for molecular property prediction based on a knowledge graph can effectively address the aforementioned issues. This method enhances the model's reasoning and understanding capabilities by integrating diverse external knowledge, such as drug knowledge bases and literature, and improves prediction accuracy even in data-scarce environments. Furthermore, this knowledge-enhanced large model possesses multi-task learning capabilities, enabling it to handle different types of drug molecular property prediction tasks within a unified framework, thereby improving prediction efficiency and reducing the time cost of model training and debugging.

[0005] In summary, a method for constructing a large language model for molecular property prediction based on knowledge graph is designed to solve the above problems. Summary of the Invention

[0006] The present invention aims to provide a method for constructing a large language model for molecular property prediction based on a knowledge graph, and realize the prediction of androgen receptor (AR), aryl hydrocarbon receptor (AhR), antioxidant response element (ARE), androgen receptor ligand binding domain (AR-LBD), aromatase, ATPase family, AAA domain containing 5, ATAD5, carcinogen (Carc), estrogen receptor (ER), estrogen receptor ligand binding domain (ER-LBD), glucocorticoid receptor (GR), heat shock element (HSE), matrix metalloproteinase (MMP), cyclin 53 (Tumor Protein P53, P53) and thyroid hormone receptor (Thyroid hormone receptor). This method can predict a total of fourteen endpoints, including the molecular receptor (TR), and improve the accuracy and efficiency of the reaction prediction between molecules and biological endpoints.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A method for constructing a large language model for molecular property prediction based on a knowledge graph, the method comprising:

[0009] Data acquisition:

[0010] Obtain the pre-training corpus and the relevant data required to build the knowledge graph. The pre-training corpus consists of three parts: SMILES sequences, scientific texts, and package sequences. The specific steps are:

[0011] Using crawler technology, SMILES sequences were crawled from the PubChem website, and the titles and abstracts of papers were crawled from the PubMed website as scientific text. Named Entity Recognition (NER) tools were used to identify compound entities from the scientific text and replace them with their corresponding SMILES sequences to construct package sequences.

[0012] The specific steps for acquiring the corpus required to build a knowledge graph are:

[0013] Using crawler technology, crawl the property characteristics of molecules and the involved metabolic pathways on the PubChem website. Specifically, for each molecule, crawl the data in the Computed Properties module and Pathways module on the PubChem website;

[0014] Knowledge graph construction:

[0015] Construct a knowledge graph about endpoints. The knowledge graph includes multiple nodes, such as SMILES nodes, endpoint nodes, property nodes, and metabolic pathway nodes. The relationships between nodes are represented by triples:

[0016] <SMILES, has property, property node>;

[0017] <SMILES, undergoes, metabolic pathway>;

[0018] <SMILES, affects, endpoint node>;

[0019] <Endpoint node, has property, property node>;[[ID=2X]]

[0020] Pre-training large language model:

[0021] Pre-train the LLaMA3.1-8B model using SMILES, scientific texts, and wrapped sequences to obtain an expert large model. The specific implementation methods include:

[0022] Based on the LoRA (Low-Rank Adaptation) technology for incremental learning to reduce the number of parameters and improve the training efficiency. By performing low-rank decomposition on specific layers or parameters, the number of parameters that need to be updated is effectively reduced, thus improving the training efficiency and resource utilization rate of the large model; LoRA fine-tuning will cover all layers of the model, and this method ensures that the entire model can obtain more efficient learning effects through low-rank adjustment at all levels;

[0023] Using the RAG technology for knowledge retrieval:

[0024] When the user inputs a molecular property prediction question, first define the query to identify the subject, extract the triple subject to be retrieved, and then retrieve relevant information about the subject from the knowledge graph, where vectorization technology is used to calculate the similarity;

[0025] Filtering retrieved data:

[0026] Use the trained expert large model to filter the retrieved triple list, identify and screen out relevant attribute or property data that may cause chemical reactions between molecules and endpoints, and obtain a filtered triple;

[0027] ]>Molecular property prediction and reasoning:

[0028] The external generated large model is called to convert the filtered triples into natural language and input them into the expert large model as context information, thereby enhancing the knowledge of the expert large model, and finally predicting whether the molecule can react with the endpoint and providing corresponding judgment basis.

[0029] By comprehensively utilizing knowledge graphs, RAG technology, and large language models, this paper can effectively improve the accuracy and efficiency of molecular property prediction, and promote the development of fields such as drug development and molecular design. This is specifically reflected in the following aspects:

[0030] Improve prediction accuracy by building a rich knowledge graph that combines the complex relationships between molecular structure and biological endpoints to more accurately capture potential reaction mechanisms. This accuracy is crucial in drug development, helping to identify new drug candidate molecules and reduce R&D risks.

[0031] Accelerate the R&D process by leveraging RAG technology for rapid knowledge retrieval, significantly reducing data processing and information acquisition time. R&D personnel can obtain the information they need in a shorter time, accelerating drug screening and validation, and improving overall R&D efficiency.

[0032] Enhanced decision support: By analyzing extensive historical data and real-time information, the system can provide R&D personnel with reliable predictions and recommendations. This intelligent decision support can help scientists make more forward-looking choices during the molecular design phase and optimize experimental resource allocation.

[0033] Reduce costs. Efficient molecular property prediction can reduce the number of laboratory tests and reduce the resources and capital investment required for physical experiments, thereby effectively controlling R&D costs and improving economic benefits.

[0034] Promoting innovation, the implementation of this invention will bring new research ideas to the fields of drug development and molecular design, promote the discovery and application of new materials and new drugs, and promote the innovative development of the biopharmaceutical industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to make the purpose, technical solutions and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0036] Figure 1 It is a flow chart of a method for constructing a large language model for molecular property prediction based on a knowledge graph according to the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the present invention and to make the above-mentioned objectives, technical solutions and advantages of the present invention more obvious and understandable, the present invention is further described in detail below in conjunction with embodiments and accompanying drawings.

[0038] like Figure 1 As shown, this embodiment provides a method for constructing a large language model for molecular property prediction based on a knowledge graph, specifically including:

[0039] S1: Data acquisition: obtain the relevant data required for pre-training corpus and knowledge graph construction through high-throughput data collection;

[0040] S2: Knowledge graph construction: During the data preparation phase, the collected information such as molecular properties and metabolic pathways is used to construct a knowledge graph about the endpoint. The knowledge graph consists of triples.

[0041] S3: Pre-training the large language model; use the acquired corpus to pre-train the large language model, and use LoRA technology to perform incremental learning on all layers of the model. The trained model is the expert large model;

[0042] S4: Use RAG technology to perform knowledge retrieval, retrieve relevant triple information from the knowledge graph of the endpoint, and organize the results into a triple list;

[0043] S5: Filter the retrieved data; construct a suitable prompt word and let the expert model filter the retrieved triple list to obtain a filtered triple list;

[0044] S6: Molecular property prediction and reasoning: The filtered triple list is converted into a natural language description and input into the expert model as a new prompt word to obtain the prediction results of the molecular properties.

[0045] This method can predict a total of fourteen endpoints, including AR, AhR, ARE, AR-LBD, Aromatase, ATAD5, Carc, ER, ER-LBD, GR, HSE, MMP, P53 and TR. This method improves the accuracy and efficiency of the reaction prediction between molecules and biological endpoints.

[0046] Furthermore, step S1 data acquisition specifically includes: using crawler technology to obtain SMILES sequences as molecular structure data; obtaining the titles and abstracts of scientific papers through crawler technology to build a scientific text corpus; using named entity recognition tools to identify compound entities from scientific texts, and replacing the compound entities with corresponding SMILES sequences to generate package sequences; obtaining the physicochemical properties and metabolic pathway data of molecules for building a knowledge graph.

[0047] Step 101: Use crawler technology to crawl 30M SMILES sequences from the PubChem website and crawl the titles and abstracts of 30M papers from the PubMed website as scientific texts; use NER tools to identify compound entities from the scientific texts and replace the entities with their corresponding SMILES sequences, thus constructing 8M wrapped sequences; crawl the property properties and involved metabolic pathways of molecules from the PubChem website. Specifically, for each molecule, crawl the data in the Computed Properties module and Pathways module on the PubChem website.

[0048] The construction of the knowledge graph in Step S2 specifically includes: defining nodes and relationships. The nodes include SMILES nodes, property nodes, metabolic pathway nodes, and endpoint nodes AR nodes; the relationships include types such as "has property", "undergoes", and "affects", etc.; the SMILES nodes are associated with the property nodes through the "has property" relationship to describe the physicochemical properties of the molecules; the SMILES nodes are associated with the metabolic pathway nodes through the "undergoes" relationship to describe the metabolic processes of the molecules; the SMILES nodes are associated with the endpoint nodes through the "affects" relationship to represent the influence of the molecules on the endpoints; the endpoint nodes are associated with the property nodes through the "has property" relationship to describe the biological properties of the endpoints.

[0049] Step 102: Based on the relevant data obtained in Step 101 for constructing the knowledge graph, construct a knowledge graph about the AR endpoint. Specifically, the knowledge graph contains SMILES nodes, AR nodes, property nodes, and metabolic pathway nodes, and the relationships between the nodes are represented by triples, including: <SMILES, has property, property node>, <SMILES, undergoes, metabolic pathway>, <SMILES, affects, AR node>, <AR node, has property, property node>.

[0050] Furthermore, the pre-training of the large language model in Step S3 specifically includes: using the obtained SMILES sequences, scientific texts, and wrapped sequences as pre-training corpora to pre-train the general large language model; adopting the LoRA technology for incremental learning to reduce the number of parameters and improve the training efficiency through low-rank decomposition; the LoRA fine-tuning covers all layers of the model. During the pre-training process, set the per-device batch size to 12, the gradient accumulation steps to 2, the learning rate to 1.0e-4, the number of training epochs to 1, and use the cosine annealing learning rate scheduler and BF16 precision for training.

[0051] Step 103: The LLaMA3.1-8B model is pre-trained using 30M SMILES sequences, 30M scientific text, and 8M parcel sequences as pre-training data. LoRA technology is used for incremental learning to reduce the number of parameters and improve training efficiency. By performing low-rank decomposition on specific layers or parameters, the number of parameters that need to be updated is effectively reduced, thereby improving training efficiency and resource utilization for large models. LoRA fine-tuning covers all layers of the model, ensuring that the entire model achieves more efficient learning results through low-rank adjustments at all levels. On eight A100 GPUs, a per-device training batch size of 12 is used, which helps provide the model with sufficient data to learn while maintaining memory efficiency. Secondly, by setting the number of gradient accumulation steps to 2, larger batch sizes can be simulated with limited hardware resources, thereby improving training stability and performance. The learning rate is set to 0.0001, a relatively small value, which helps the model update weights slowly and steadily in the early stages of training, avoiding instability caused by excessive update steps. The number of training rounds is set to 1, which means that we will traverse the training dataset completely once; the cosine annealing strategy is selected as the learning rate scheduler, which can periodically adjust the learning rate during training, helping the model to adjust the weights more finely in the later stages of training, thereby improving the final performance; the warm-up ratio is set to 0.1, which means that in the first 10% of the training time, the learning rate will gradually increase from 0 to the set value, which helps the model to be more stable in the early stages of training; in addition, BF16 training is enabled, which means that BFloat16 precision is used for calculations, which can significantly reduce memory usage and speed up the training process while largely maintaining the accuracy of the model.

[0052] Furthermore, the knowledge retrieval of step S4 specifically includes: based on the molecular property prediction question input by the user, calculating the similarity between the query subject and the nodes in the knowledge graph through vectorization technology; retrieving triples related to the query subject from the knowledge graph, and outputting a list of triples related to the query question.

[0053] Step 104: Use RAG technology to perform knowledge retrieval reasoning

[0054] This step introduces RAG (Retrieval-Augmented Generation) technology to match the molecular property prediction problem entered by the user with relevant information in the knowledge base to support the prediction process. The specific process is as follows:

[0055] (1) Retrieval input: The user inputs a property prediction question for a certain molecule. For example, the user inputs the question “Does {SMILES} react with AR endpoints?” Here, {SMILES} represents the structural information of the molecule, and AR endpoints are potential reaction targets.

[0056] (2) Define the query: Based on the user-entered molecules and reaction endpoints, the system automatically identifies key entities and attributes. First, the SMILES sequence is extracted as the primary entity, and the AR endpoint is identified as the target node of the reaction. During this process, the system uses predefined rules or models to determine the query focus, namely "{SMILES}" and "AR."

[0057] (3) Similarity retrieval: Based on the query focus determined in the previous step, the system searches the knowledge graph for other entities and attributes related to "{SMILES}" and "AR," especially those potentially related to chemical reactions. During the retrieval, the system uses vectorization technology to calculate the similarity between the SMILES molecular sequence and other molecules and endpoints in the knowledge base to find the most relevant triple data, including information such as reaction influencing factors and molecular structural characteristics.

[0058] (4) Search output: Based on the similarity search results, the system generates a list of triplets that show the molecules, endpoints, and their properties that are most relevant to the input question. This output will serve as input for subsequent steps to further analyze whether there is potential chemical reaction between the molecule and the endpoint.

[0059] Furthermore, the data filtering in step S5 specifically includes: using the expert big model to filter the triple list retrieved from the knowledge graph; by analyzing the molecular structure, physicochemical properties and their relationship with the reaction endpoint, screening out the key attributes or property data that may cause the molecule to react chemically with the endpoint; filtering out irrelevant triples to obtain a filtered triple list as input for subsequent reasoning.

[0060] Step 105: Filter the retrieved triples using the expert model

[0061] In this step, a specially trained expert model is used to filter the triple data retrieved from the knowledge graph to further screen out key attributes or factors that may trigger chemical reactions between molecules and endpoints. This process specifically includes:

[0062] (1) Input triples: Construct a suitable prompt word and input the triple list generated in step 104 into the expert model. The model will analyze the nodes and relationships in the triples based on the knowledge learned during the training process.

[0063] (2) Property Analysis: The expert model analyzes the molecular structure, physicochemical properties, and their relationship with the reaction endpoints (such as AR) involved in the triples to identify the relevant properties that may lead to chemical reactions. For example, the model evaluates factors such as the chemical groups, polarity, and electron cloud distribution of SMILES molecules and matches them with the reactivity of the endpoints to determine which triples have the potential for chemical reactions.

[0064] (3) Triple filtering: Based on the above analysis results, the model will filter out triplets that are not likely to react. For example, if the charge distribution or spatial configuration between the molecule and the endpoint is not suitable for reaction, the triple will be excluded. After filtering, only those triplets that are considered to be likely to trigger chemical reactions are retained, resulting in a concise list of triplets that are highly relevant to chemical reactions.

[0065] Furthermore, the molecular property prediction in step S6 specifically includes: converting the filtered triples into natural language prompts and inputting them into the expert model; the expert model performs knowledge-enhanced reasoning based on the input prompts to predict whether the molecule reacts with the endpoint; at the same time, the model provides reasoning process and judgment basis, explains the potential mechanism of the reaction between the molecule and the endpoint, and outputs the final prediction result.

[0066] Step 106: Use the filtered triple combination prompt and use the expert model to perform knowledge-enhanced prediction

[0067] This step, based on the filtered triples generated in step 105, further guides the expert model to perform knowledge-enhanced reasoning through the prompt mechanism, predicting whether a chemical reaction will occur between the molecule and the endpoint and providing an explanation. The specific process is as follows:

[0068] (1) Prompt generation: The filtered triples are constructed into natural language prompts in the form of “Will the {SMILES} molecule react with the AR endpoint? The relevant chemical properties are X, Y, Z. Please explain its reaction mechanism.” Here, X, Y, and Z represent the key chemical properties or reaction factors extracted from the filtered triples.

[0069] (2) Knowledge-enhanced prediction: The generated natural language prompt is fed into the expert model, which then performs knowledge-enhanced reasoning based on the prompt information. The expert model has been pre-trained with a large amount of medicinal chemistry data and chemical reaction mechanisms. Therefore, it can combine the attributes in the prompt to deeply reason about the likelihood of the molecule reacting with the endpoint.

[0070] (3) Prediction and explanation: Based on the input clues, the model infers and predicts whether the {SMILES} molecule will react with the AR endpoint. At the same time, the model also outputs a corresponding explanation to explain the mechanism of the reaction. For example, based on the electronic configuration or structural characteristics of the SMILES molecule, the model may indicate that it has a high affinity with the AR endpoint and is therefore likely to react. The explanation includes the type of reaction, the changes in the chemical bonds involved, and a detailed analysis of the relevant chemical properties.

[0071] Specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, those skilled in the art will appreciate that various modifications and substitutions may be made to the specific embodiments of the present invention without departing from the spirit and scope of the present invention. Such modifications and substitutions are intended to fall within the scope of the claims.

Claims

1. A method for constructing a large language model for molecular property prediction based on knowledge graph, characterized in that: Including: S1: Data acquisition; Through high-throughput data collection, obtain the pre-training corpus and relevant data required for constructing the knowledge graph; S2: Knowledge graph construction; In the data preparation stage, use the collected molecular properties and metabolic pathway information to construct a knowledge graph about endpoints; This knowledge graph consists of triples, which define nodes and relationships. The nodes include SMILES nodes, property nodes, metabolic pathway nodes, and endpoint nodes. The relationships include "has property", "experiences", and "affects" types. Specifically, the SMILES node is associated with the property node and the endpoint node is associated with the property node through the "has property" relationship, the SMILES node is associated with the metabolic pathway node through the "experiences" relationship, and the SMILES node is associated with the endpoint node through the "affects" relationship; There are a total of fourteen endpoints in the knowledge graph, including androgen receptor, aromatic hydrocarbon receptor, antioxidant stress response element, androgen receptor ligand binding domain, aromatase, ATPase-related protein 5, cancer, estrogen receptor, estrogen receptor ligand binding domain, glucocorticoid receptor, heat shock element, matrix metalloproteinase, cyclin 53, and thyroid hormone receptor; Specific triples include: <SMILES, has property, property node>; <SMILES, experiences, metabolic pathway>; <SMILES, affects, endpoint node>; <endpoint node, has property, property node>; S3: Pre-train the large language model; Use the obtained corpus to pre-train the large language model, and use the LoRA technology to perform incremental learning on all layers of the model. The trained model is an expert large model; S4: Use the RAG technology for knowledge retrieval, retrieve relevant triple information from the knowledge graph of endpoints, and organize the results into a triple list; S5: Filter the retrieved data; Construct a suitable prompt, and let the expert large model filter the retrieved triple list to obtain a filtered triple list; S6: Molecular property prediction and reasoning; Convert the filtered triple list into a natural language description, and input it as a new prompt into the expert large model to obtain the prediction result of the molecular property.

2. The method for constructing a large language model for molecular property prediction based on a knowledge graph according to claim 1, characterized in that: The data acquisition in step S1 specifically includes: Using the crawler technology to obtain the SMILES sequence as the data of the molecular structure; Using the crawler technology to obtain the titles and abstracts of scientific papers to construct a scientific text corpus; Using the named entity recognition tool to identify compound entities from scientific texts and replace the compound entities with the corresponding SMILES sequences to generate wrapped sequences; Obtain the physicochemical properties and metabolic pathway data of molecules for constructing the knowledge graph.

3. The method for constructing a large language model for molecular property prediction based on knowledge graph according to claim 1, characterized in that: The pre-training large language model in step S3 specifically includes: using the obtained SMILES sequence, scientific text and package sequence as pre-training corpus to pre-train the general large language model; using LoRA technology for incremental learning, reducing the number of parameters and improving training efficiency through low-rank decomposition; LoRA fine-tuning covers all layers of the model, setting the batch size per device to 12, the number of gradient accumulation steps to 2, the learning rate to 1.0e-4, the number of training rounds to 1, and using the cosine annealing learning rate scheduler and BF16 accuracy for training.

4. The method for constructing a large language model for molecular property prediction based on a knowledge graph according to claim 1, characterized in that: The knowledge retrieval in step S4 specifically includes: based on the molecular property prediction question input by the user, calculating the similarity between the query subject and the nodes in the knowledge graph through vectorization technology; retrieving triples related to the query subject from the knowledge graph, and outputting a list of triples related to the query question.

5. The method for constructing a large language model for molecular property prediction based on a knowledge graph according to claim 1, characterized in that: The data filtering in step S5 specifically includes: using the expert big model to filter the triple list retrieved from the knowledge graph; by analyzing the molecular structure, physicochemical properties and their relationship with the reaction endpoint, screening out the key attributes or property data that may cause the molecule to react chemically with the endpoint; filtering out irrelevant triplets to obtain a filtered triple list as input for subsequent reasoning.

6. The method for constructing a large language model for molecular property prediction based on knowledge graph according to claim 1, characterized in that: The molecular property prediction in step S6 specifically includes: converting the filtered triples into natural language prompts and inputting them into the expert model; the expert model performs knowledge-enhanced reasoning based on the input prompts to predict whether the molecule reacts with the endpoint; at the same time, the model provides reasoning and judgment basis, explains the potential mechanism of the reaction between the molecule and the endpoint, and outputs the final prediction result.

Images