Pilot compound design method, system, device and medium based on deep reinforcement learning

Abstract

This application provides a method, system, device, and medium for lead compound design based on deep reinforcement learning. It innovatively achieves de novo design of covalent inhibitors targeting affinity, proposes an innovative method for constructing novel drug molecules based on multiple different bioactive fragments, and optimizes the model's exploration space. The application of the physics computation software Autodock-GPU as a scoring function enhances the model's generalization ability and running speed. The application of Munchausen reinforcement learning techniques to the Soft Actor Critic discrete action space model improves the model's optimization performance. Compared to other models, this application expands and improves upon the model's functionality and the properties of the generated molecules.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and compound design technology, and in particular to pioneering compound design methods, systems, devices and media based on deep reinforcement learning. Background Technology

[0002] Small molecule drugs have always played a vital role in the fight against disease. In the first half of 2021, nearly half of the 52 new drugs approved by the FDA were primarily for cancer treatment. However, as time goes on, the development cycle of traditional drugs is becoming increasingly longer and more expensive. Currently, the average time from development to market for an FDA-approved drug is over 10 years, costing more than $2 billion. (The text then abruptly shifts to a seemingly unrelated topic: "From size 10...") 60 Searching for molecules with ideal therapeutic potential in molecular space is one of the key tasks in drug discovery. To reduce the cost and time of developing new drugs, using generative models with high sample efficiency to replace the mechanical screening of countless potential drug molecules has been considered a promising approach.

[0003] In the field of computer-aided drug design, two fundamental challenges are how to efficiently search for potential drug molecules in the vast molecular space and how to improve the generalization ability of the model. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a method, system, device and medium for designing lead compounds based on deep reinforcement learning, in order to solve the technical problems of how to efficiently search for potential drug molecules in a huge molecular space and how to improve the generalization ability of the model.

[0005] To achieve the above and other related objectives, a first aspect of this application provides a method for designing lead compounds based on deep reinforcement learning, comprising: constructing a molecular fragment library of multiple different bioactive molecular fragments required to constitute a lead compound; constructing molecular graph data of each of the molecular fragment libraries, and extracting features from the molecular graph data based on a graph convolutional neural network; using the extracted features of the molecular graph data and the extended connectivity fingerprint features of each molecular fragment in the molecular fragment library as input parameters of a deep reinforcement learning model to select specific bioactive molecular fragments and corresponding connection sites for splicing to form molecular fragments for constituting a lead compound.

[0006] In some embodiments of the first aspect of this application, during the design of covalent compounds, one Hinge Binder molecular fragment, one Linker molecular fragment, one Other Group molecular fragment, and one Warhead molecular fragment are selected from each molecular fragment library; the features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment are extracted and input into the deep reinforcement learning model, so that the deep reinforcement learning model outputs the molecular fragments required for the design of the covalent compound, which are then spliced ​​together to form the covalent compound; and / or, during the design of non-covalent compounds, one Hinge Binder molecular fragment, one Linker molecular fragment, and two Other Group molecular fragments are selected from each molecular fragment library; the features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment are extracted and input into the deep reinforcement learning model, so that the deep reinforcement learning model outputs the molecular fragments required for the design of the non-covalent compound, which are then spliced ​​together to form the non-covalent compound.

[0007] In some embodiments of the first aspect of this application, the process of constructing the molecular fragment library includes: extracting effective molecules and candidate molecular fragments from relevant literature or databases and generating molecular fragment files in database format; reading molecular fragment files in database format from different sources, splitting and screening molecular fragments with different biological activities using corresponding screening conditions to form different molecular fragments; and storing the screened molecular fragments in a preset molecular language format to form the corresponding molecular fragment library.

[0008] In some embodiments of the first aspect of this application, a graph convolutional neural network with a 3-layer network structure and ReLU activation function in each layer is selected to extract features from the molecular graph data; the propagation mode between network layers is as follows:

[0009]

[0010] Among them, H (l+1) The node information for layer (l+1); AGG is the aggregation function; ReLU is the modified linear unit activation function; H (l) This refers to the node information of the l-th layer; A is the adjacency matrix of the graph nodes, I represents the identity matrix; H (l) This is the output of the previous layer, and its shape is (n, F). l ), where n is the number of nodes in the graph, F l W is the number of output channels of the l-th layer. (l) Let be the transformation parameter matrix of the l-th convolutional layer; It is the degree matrix obtained by A. H represents the features of each layer.

[0011] In some embodiments of the first aspect of this application, the deep reinforcement learning model is constructed through the following steps: the agent of the deep reinforcement learning model perceives the environment and performs the action of selecting molecular fragments to be spliced ​​and splicing sites; the environment splices molecular fragments according to the selected molecular fragments to be spliced ​​and splicing sites, causing a change in the state of the environment, and performing receptor-ligand affinity prediction, and using the predicted affinity value as the reward score for the action performed in the previous step; the agent updates its policy at least according to the reward score; the above steps are repeated until a deep reinforcement learning model capable of generating ideal lead compounds is obtained.

[0012] In some embodiments of the first aspect of this application, the structure of the deep reinforcement learning model includes: three 3-layer fully connected networks, namely policy networks π1, π2, and π3. Given an intermediate molecule, after feature extraction by the encoder, the model's decision-making includes three processes: selection of the connection site of the current intermediate molecule as action 1, selection of molecular fragments from the candidate molecular fragment library as action 2, and selection of the connection site of the selected molecular fragment as action 3; the probability distribution of each action includes:

[0013] p act1 =π1(Z 1st Z 1st =MI(h g H att );

[0014]

[0015]

[0016] Where, p act1 p act2 p act3 Let Z represent the probability distributions of action 1, action 2, and action 3 (act3), respectively; MI represents the multiplication interaction; Z represents the probability distributions of action 1, action 2, and action 3 (act3). 1st h g and H att The result of the multiplication interaction between them is in matrix form; h g The graphical representation information of the intermediate molecules in the current state; H att Z represents the node characterization information of all connection sites of the intermediate molecule in the current state; 2nd Let be a matrix, representing and ECFP(m cand The result of multiplication between ) It is an element in the matrix, representing the multiplication interaction result between the connection site selected by action 1 and the graph representation information of the intermediate molecular graph; ECFP(m candZ represents the extended connectivity fingerprint of all candidate molecular fragments; 3rd It is a matrix, representing The result of the multiplication interaction between Ucand and Ucand; It is an element in the matrix, representing the multiplicative interaction information between the connection site selected by action 2 and the extended connectivity fingerprint representation of the selected molecular fragment; U cand This provides node characterization information for all connection sites on the selected molecular fragment.

[0017] In some embodiments of the first aspect of this application, when the policy network π2 makes a molecular fragment decision, the logits value output by the policy network π2 is multiplied by the molecular fragment mask term, and then the most suitable available molecular fragment is obtained based on the Gumbel Softmax algorithm.

[0018] In some embodiments of the first aspect of this application, the deep reinforcement learning model is based on the maximum entropy SoftActor Critic deep reinforcement learning framework, and its objective function includes:

[0019]

[0020] Among them, s t As the current state, a t For the current action, R(s) t a t ) represents state s t Perform a t The immediate reward score for the action, α is the balanced reward score r(s) t a t ) and exploration ability hyperparameters, For state s t The action probability entropy, ρ π This represents the state-action transition distribution generated by policy π. Indicates that in state s t The next strategy π generates action a t At that time, the expected value of the model's instantaneous reward score and action probability entropy, This means selecting the strategy that maximizes the sum of the reward score and the action probability entropy from all possible strategies.

[0021] In some embodiments of the first aspect of this application, the method further includes using Auto-GPU as the reward function of the deep reinforcement learning model, and using the predicted affinity value as the reward of the deep reinforcement learning model.

[0022] In some embodiments of the first aspect of this application, the reward function script of the deep reinforcement learning model includes a covalent docking script and a non-covalent docking script.

[0023] In some embodiments of the first aspect of this application, based on the Munchausen reinforcement learning algorithm, a Munchausen term is introduced in the temporal difference process of the deep reinforcement learning model to limit the error caused by excessive differences between the old and new policies.

[0024] In some embodiments of the first aspect of this application, a reward predictor is set in the deep reinforcement learning model; the reward predictor is updated after each molecular docking process; when a new state appears, the prediction result of the reward predictor will have a higher error, while for states that have already appeared, the prediction result of the reward predictor will have a smaller error.

[0025] To achieve the above and other related objectives, a second aspect of this application provides a lead compound design system based on deep reinforcement learning, comprising: a molecular fragment library construction module for constructing a molecular fragment library of multiple different bioactive molecular fragments required to constitute a lead compound; a feature extraction module for constructing molecular graph data of each of the molecular fragment libraries and extracting features from the molecular graph data based on a graph convolutional neural network; and a model construction module for using the features of the extracted molecular graph data and the extended connectivity fingerprint features of each molecular fragment in the molecular fragment library as input parameters of a deep reinforcement learning model to select specific bioactive molecular fragments and corresponding connection sites for splicing to form molecular fragments for constituting a lead compound.

[0026] In some embodiments of the second aspect of this application, the method further includes: a covalent compound design module, configured to select one Hinge Binder molecular fragment, one Linker molecular fragment, one Other Group molecular fragment, and one Warhead molecular fragment from each molecular fragment library; extract the features of the molecular graph of each molecular fragment and the extended connectivity fingerprint features, and input them into the deep reinforcement learning model, so that the deep reinforcement learning model outputs the molecular fragments required for designing the covalent compound, for splicing to form the covalent compound; and a non-covalent compound design module, configured to select one Hinge Binder molecular fragment, one Linker molecular fragment, and two Other Group molecular fragments from each molecular fragment library; extract the features of the molecular graph of each molecular fragment and the extended connectivity fingerprint features, and input them into the deep reinforcement learning model, so that the deep reinforcement learning model outputs the molecular fragments required for designing the non-covalent compound, for splicing to form the non-covalent compound.

[0027] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep reinforcement learning-based lead compound design method.

[0028] To achieve the above and other related objectives, a fourth aspect of this application provides a computer device, comprising: a processor and a memory; the memory for storing a computer program, and the processor for executing the computer program stored in the memory to cause the terminal to execute the lead compound design method based on deep reinforcement learning.

[0029] As described above, the lead compound design method, system, device, and medium based on deep reinforcement learning in this application have the following beneficial effects: This invention innovatively realizes the de novo design function of covalent inhibitors targeting affinity, proposes an innovative method for constructing novel drug molecules based on multiple different bioactive fragments, and optimizes the model's exploration space; it utilizes the physics computing software Autodock-GPU as a scoring function to improve the model's generalization ability and running speed; and it applies the Munchausen reinforcement learning technique to the Soft Actor Critic discrete action space model, improving the model's optimization performance. Compared to other models, it expands and improves upon the model's functionality and the properties of the generated molecules. Attached Figure Description

[0030] Figure 1 The diagram shown is a flowchart illustrating a lead compound design method based on deep reinforcement learning in one embodiment of this application.

[0031] Figure 2 The diagram shown is a simplified model structure diagram of a deep reinforcement learning model in one embodiment of this application.

[0032] Figure 3 The diagram shows the structure of the Soft Actor Critic model based on the Munchausen reward.

[0033] Figure 4 The diagram shown is a schematic representation of a lead compound design system based on deep reinforcement learning, as described in one embodiment of this application.

[0034] Figure 5 The diagram shown is a structural schematic of a computer device according to an embodiment of this application. Detailed Implementation

[0035] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0036] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of this application. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical, and operation may be made without departing from the spirit and scope of this application. The following detailed description should not be considered limiting, and the scope of the embodiments of this application is defined only by the claims of the published patent. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, type, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, types, and / or groups. The terms “or” and “and / or” as used herein are to be interpreted inclusively, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition only occur when the combination of elements, functions, or operations is inherently mutually exclusive in some way.

[0037] In medicinal chemistry, a lead compound is typically divided into multiple bioactive molecular fragments. Modifying some fragments or designing lead compounds de novo based on different fragments is a common inhibitor design approach. Constructing a library of high-performance bioactive molecular fragments to design kinase inhibitors can effectively optimize the model's exploration space. Furthermore, software that predicts affinity based on physicochemical calculations (such as Autodock) can effectively overcome the biases and poor generalization issues inherent in other data-driven prediction models.

[0038] Based on this, the present invention provides a method, apparatus, terminal, and medium for designing lead compounds based on deep reinforcement learning. The present invention first constructs a library of molecular fragments with different biological activities, designs graph data representations of molecules, and extracts features using a graph convolutional neural network. Then, the extracted features and the extended connectivity fingerprint (ECFP) features of each molecular fragment in the molecular fragment library are used as input to a deep reinforcement learning model to select specific biologically active molecular fragments and corresponding connection sites. Furthermore, the present invention uses the Soft Actor Critic reinforcement learning architecture as the model body and optimization method, and integrates Autodock-GPU software as the reward function to improve the model's generalization ability.

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the invention.

[0040] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0041] (1) Reinforcement learning: The interaction process of reinforcement learning is that the agent selects an action in the current state according to its policy, the environment responds to these actions, and the agent moves to a new state. At the same time, a reward signal (usually a numerical value) is generated. The discounted summation of the reward is called the gain / reward, which is the goal that the agent wants to maximize in the action selection process.

[0042] (2) Graph database: A data management system based on points and edges as the basic storage units, designed for efficient storage and retrieval of graph data. A graph is a collection of points and edges, where "points" represent entities and "edges" represent relationships.

[0043] (3) Graph Convolutional Neural Network (GCN): It is used to extend convolution operations from traditional data (images or grids) to graph data. Its key is to learn a function f, which generates a representation of node Vi by aggregating its own features Xi and the features Xj of its neighbors.

[0044] (4) ECFP: Extended Connectivity Fingerprints, is a type of molecular fingerprint used to construct quantitative structure-activity relationship (QSAR) models of compounds.

[0045] This invention provides a lead compound design method based on deep reinforcement learning, a system for designing lead compounds based on deep reinforcement learning, and a storage medium storing an executable program for implementing the lead compound design method based on deep reinforcement learning. Regarding the implementation of the lead compound design method based on deep reinforcement learning, this invention will describe an exemplary implementation scenario of designing covalent / non-covalent kinase inhibitors based on deep reinforcement learning.

[0046] like Figure 1 The diagram illustrates a flowchart of a lead compound design method based on deep reinforcement learning, as shown in this embodiment of the invention. The lead compound design method based on deep reinforcement learning in this embodiment mainly includes the following steps:

[0047] Step S1: Construct a molecular fragment library of multiple different bioactive molecular fragments required to form the lead compound.

[0048] It should be understood that, in order to train the deep reinforcement learning model, a molecular fragment library needs to be constructed as the action space of the deep reinforcement learning model. Furthermore, since this invention aims to combine molecular fragments with different definitions to design specific protein inhibitors, it is necessary to construct separate libraries of molecular fragments with different biological activities.

[0049] The method for constructing multiple molecular fragment libraries with different biological activities specifically includes: extracting effective molecules and candidate molecular fragments from molecular-related literature or databases; processing and screening the segmented molecular fragments; and dividing the fragments into multiple biologically active molecular fragments according to preset screening conditions. The specific execution steps are as follows:

[0050] Step S11: Extract effective molecules and candidate molecular fragments from relevant literature or databases and generate molecular fragment files in database format.

[0051] The relevant literature in this embodiment includes, but is not limited to, published papers in pharmaceutical journals (such as JMC or EJMC journals) or publicly available patent documents; molecular-related databases include, but are not limited to, the ChEMBL database, the Enamine Hinge Binder database, and the Enamine Warhead database. For example, candidate molecular fragments can be extracted from published papers in JMC journals, and effective molecules can be extracted from the ChEMBL database and the Enamine Hinge Binder database.

[0052] Specifically, the sources of available molecular fragments disclosed in relevant literature are as follows:

[0053] The Hinge Binder molecular fragments were derived from the Enamine Hinge Binder database and / or published papers;

[0054] Linker molecular fragments are derived from published papers and / or the ChEMBL database;

[0055] The Polar Group molecular fragments are derived from published papers and / or the ChEMBL database;

[0056] Warhead molecular fragments are derived from published papers and / or the Enamine Warhead database.

[0057] Step S12: Read molecular fragment files from databases of different sources, split and screen molecular fragments with different biological activities using corresponding screening conditions to form different molecular fragments; and store the screened molecular fragments according to a preset molecular language format to form a corresponding molecular fragment library.

[0058] In some examples, the splitting and filtering process resulted in four different molecular fragments: Hinge Binder, Warhead, Linker, and Other Group.

[0059] In some examples, the molecular fragment splitting method includes: using the CReM tool to split the molecular fragments and remove erroneous fragments, generating a database file (containing the SMILES expression of the molecular fragments, the number of atoms in the fragments, and the frequency information of the fragments, with the extension .db) of the DataBase file. It should be understood that the CReM tool is an open-source Python framework that uses a fragment-based approach to generate chemical structures.

[0060] The screening criteria for the four molecular fragments—Hinge Binder, Warhead, Linker, and Other Group—are as follows:

[0061] Screening criteria a) Hinge Binder molecular fragment library screening criteria: number of atoms ∈ [6, 18], frequency of occurrence > 20. Sources of molecular fragments to be screened: Enamine Hinge Binder database, published papers.

[0062] Screening criteria b) The Linker molecular fragment library screening criteria are: number of atoms ∈ [1, 12], frequency of occurrence > 10, and number of linking sites ≥ 2. Sources of molecular fragments to be screened: published papers and the ChEMBL database.

[0063] Screening criteria c) Other Group molecular fragments are selected based on the following criteria: number of atoms ∈ [3, 12], frequency of occurrence > 10, and AlogP > 2 or AlogP < 0.5. Sources of the molecular fragments to be screened: published papers and the ChEMBL database.

[0064] Screening criteria d) The Warhead molecular fragment library screening criteria are: number of atoms ∈ [3, 12], frequency of occurrence > 10. Sources of molecular fragments to be screened: published papers, Enamine Warhead database.

[0065] In this embodiment, storage is performed according to a preset molecular language format, preferably SMILES format, due to its uniqueness and space-saving characteristics. SMILES (Simplified Molecular InputLine Entry System) is a linear symbol used to input and represent molecular reactions, and is an ASCII encoding. For example, the SMILES format corresponding to ethane is CC, and the SMILES format corresponding to hydrated hydrogen ions is [OH3+], etc.

[0066] Step S2: Construct molecular graph data for each of the molecular fragment libraries, and extract features from the molecular graph data based on a graph convolutional neural network.

[0067] Since molecular graph data is a structure representing non-Euclidean space, it cannot be constructed using tools commonly used to build tabular data (such as pandas.DataFrame). Therefore, this embodiment uses the RDKit package and the DGL graph neural network framework to construct molecular graph data, or NetworkX can also be used to construct molecular graph data.

[0068] In some examples, NetworkX is a graph theory and complex network modeling tool developed in Python. It includes commonly used graph and complex network analysis algorithms, facilitating complex network data analysis and simulation modeling. NetworkX supports the creation of undirected graphs, directed graphs, and multigraphs, and incorporates many standard graph theory algorithms (such as shortest path algorithms and connected graph algorithms). Therefore, NetworkX can be used for graph structure storage, generating various random and classic structure graphs, performing graph structure analysis and modeling, and visualizing graph structures.

[0069] In some examples, the RDKit package is used to extract given node (atom) and edge (bond) features from a molecule, and the molecular graph is constructed using DGL (Deep Graph Library). DGL is a dedicated deep learning algorithm framework for graph-structured data, supporting PyTorch / TensorFlow / MXNet as backends. It focuses on message passing within the graph, providing a more flexible programming interface and efficient low-level operators. RDKit is an open-source cheminformatics and machine learning toolkit that provides APIs in C++ and Python.

[0070] The structural information of a molecule includes node and edge information in a molecular diagram. Each molecular diagram represents a molecule, where nodes represent atoms and edges represent chemical bonds. Physicochemical information includes atom type, degree of atom, number of hydrogen atoms bonded to the atom, implicit valence of the atom, whether it is an aromatic atom, and type of atomic bond, etc.

[0071] In this embodiment, the graph convolutional neural network used for feature extraction from the molecular graph data is a 3-layer network structure with ReLU activation function in each layer. The propagation between layers is as follows:

[0072]

[0073] Among them, H (l+1) The node information for layer (l+1); AGG is the aggregation function; ReLU is the modified linear unit activation function; H (l) This refers to the node information of the l-th layer; A is the adjacency matrix of the graph nodes, I represents the identity matrix; H (l) This is the output of the previous layer, and its shape is (n, F). l ), where n is the number of nodes in the graph, F l W is the number of output channels of the l-th layer. (l) Let be the transformation parameter matrix of the l-th convolutional layer; It is the degree matrix obtained by A. H represents the features of each layer.

[0074] It should be understood that Graph Convolutional Neural Networks (GCNs) are feature extractors suitable for graph data. Graphs are typically used to represent the structure of entities and their relationships, serving as an effective information carrier in complex systems. A graph can be denoted as G = (V, E), where V represents the set of nodes and E represents the set of edges. The structure of a graph is generally highly irregular, and can be considered as infinite-dimensional data, thus exhibiting almost no translation invariance. The surrounding structure of each node may be unique, and this type of data structure can cause traditional convolutional neural network (CNN) and recurrent neural network (RNN) structures to instantly fail.

[0075] Step S3: Use the features of the extracted molecular graph data and the extended connectivity fingerprint features of each molecular fragment in the molecular fragment library as input parameters for the deep reinforcement learning model to select specific bioactive molecular fragments and corresponding connection sites for splicing to form molecular fragments that constitute lead compounds.

[0076] It should be understood that the method for generating lead compounds in this invention can be used to design covalent and non-covalent high-affinity lead compounds for different proteins, preferably kinase proteins. In some embodiments of this invention, the lead compound is a kinase inhibitor lead compound. The molecular fragment library of kinase inhibitor lead compounds includes four libraries: Hinge Binder molecular fragment library, Linker molecular fragment library, Other Group molecular fragment library, and Warhead molecular fragment library. Meanwhile, the specific bioactive molecular fragment refers to the molecular fragment required for the current lead compound to be designed.

[0077] It should be noted that the simplified model structure of a deep reinforcement learning model is as follows: Figure 2 As shown: An agent selects an action 'a' for the environment. Upon receiving this action, the environment's state 's' changes, and a reinforcement signal is generated and fed back to the agent. The agent in a deep reinforcement learning model is also called a reinforcement learning system. The agent can perceive the state of the environment. By perceiving the environment (candidate molecular fragments, intermediate molecules formed by splicing selected molecular fragments, etc.) and making decisions (selecting the molecular fragment to be spliced ​​and the splicing site), the environment receives and executes the decision, thus changing its state (e.g., acquiring a new molecular structure) and providing a reward. The agent then makes the next decision based on the current state of the environment, repeating the above process to obtain a deep reinforcement learning model capable of generating ideal lead compounds.

[0078] In some embodiments of the present invention, the deep reinforcement learning model is constructed by the following method:

[0079] Step S31: The agent of the deep reinforcement learning model perceives the environment and performs the action of selecting the molecular fragments to be spliced ​​and the splicing sites.

[0080] Step S32: The environment splices the molecular fragments according to the selected molecular fragments to be spliced ​​and the splicing site, causing a change in the state of the environment, and predicts the receptor-ligand affinity. The predicted affinity value is used as the reward score for the action described in step S31.

[0081] Step S33: The agent updates its policy based at least on the reward score.

[0082] Step S34: Repeat steps S21 to S23 until a deep reinforcement learning model capable of generating an ideal lead compound is obtained. It should be understood that the ideal lead compound is not an absolute concept and can be preset by the user to serve as the target and direction for model learning.

[0083] Furthermore, the specific execution process in step S31 above is as follows:

[0084] Step S311: The agent perceives the environment at the initial time, denoted as t=0. In the initial state (s0), the environment perceived by the agent at time t=0 is the extended connectivity fingerprint (ECFP) feature and node representation of each molecular fragment in the molecular fragment library, as well as the initial values ​​representing the graph-level representation and node-level representation of the intermediate molecular graph.

[0085] Step S312: The agent establishes an equilibrium state s based on the value function. t The reward score (i.e., affinity score) r(s) of the model below t a t and the ability to explore (i.e., molecular uniqueness). Generate policy π based on the policy function and select the corresponding action a according to policy π. t Action a t This involves selecting the molecular fragments to be spliced ​​and the splicing sites.

[0086] Step S313: The environment executes the selected action a t The molecular fragments to be spliced ​​are taken from the molecular fragment library and spliced ​​according to the selected splicing sites to generate new intermediate molecules. At the same time, it is determined whether the newly generated intermediate molecules still have splicable sites.

[0087] Step S314: Determine whether there is a splicable site; if there is no splicable site, return to step S311; if there is a splicable site, proceed to step S315.

[0088] Step S315: Assign the model a small immediate reward r = 0.05, complete the environment update at time t, and advance to time t+1, state s. t Transition to the next new state s t+1 .

[0089] Step S316: Generate s represented as graph data t+1The intermediate molecules in the state are identified, and features of the intermediate molecules are extracted using a graph convolutional neural network. The extracted features of the intermediate molecules are then incorporated into the environment.

[0090] Step S317: The agent perceives the environment at time t+1, namely the extended connectivity fingerprint (ECFP) features and node representations of each molecular fragment in the molecular fragment library, as well as the features of the extracted intermediate molecules.

[0091] Step S318: Repeat steps S312 to S317 above. When t = 4, i.e., s t The round ends when the intermediate molecule of a state consists of four molecular fragments. After deleting the remaining spliceable sites of the current intermediate molecule, covalent reaction simulation and receptor-ligand affinity prediction are performed according to the task type, or the receptor-ligand affinity is directly predicted, and the predicted affinity value is used as the final reward score R for that round. At the same time, the agent's trajectory in that round is stored in the replay cache.

[0092] Step S319: Repeat S312 to S318. After each 256 iterations, sample 256 state sequences (transitions) from the playback buffer. Update the parameters of the model's Critic network by minimizing the square of the temporal difference learning (TD) error of the action-state pairs in the sampled data. At the same time, calculate the gradient of the action to maximize the expected reward, that is, maximize the state value function Q value of the state-action pairs in the sampled data estimated by the Critic network. Update the parameters of the Actor network using the Adam optimization algorithm.

[0093] Step S3110: Repeat steps S312 to S319 until a deep reinforcement learning model capable of generating ideal lead compounds is obtained.

[0094] As can be seen from the specific processes S311 and S318 above, the environment referred to in step S31 is different at different times and under different states. For example, in the initial state, the feature vector representing the intermediate molecule in the environment has no practical meaning.

[0095] In step S319, the critic of the reinforcement learning model consists of four neural networks, each representing an action-value function Q1. ， Q2 has two corresponding target action value functions, Q′1 and Q′2, which are further detailed below.

[0096] In some embodiments of the present invention, the specific structure of the deep reinforcement learning model includes three 3-layer fully connected networks, namely π1, π2, and π3. Given an intermediate molecule, after feature extraction by the encoder, the model's decision-making includes three processes: selection of the connection site of the current intermediate molecule (act1), selection of molecular fragments from the candidate molecular fragment library (act2), and selection of the connection site of the selected molecular fragment (act3). The selection of the connection site of the current intermediate molecule with the highest probability, the selection of the molecular fragment from the candidate molecular fragment library, and the selection of the connection site of the selected molecular fragment are determined respectively to splice together a molecular fragment used to constitute a lead compound. The specific model function is shown below:

[0097] p act1 =π1(Z 1st Z 1st =MI(h g H att ); (Formula 2)

[0098]

[0099]

[0100] Where, p act1 p act2 p act3 Let Z represent the probability distributions of action 1, action 2, and action 3 (act3), respectively; MI represents the multiplication interaction; Z represents the probability distributions of action 1, action 2, and action 3 (act3). 1st h g and H att The result of the multiplication interaction between them is in matrix form; h g The graphical representation information of the intermediate molecules in the current state; H att Z represents the node characterization information of all connection sites of the intermediate molecule in the current state; 2nd Let be a matrix, representing and ECFP(m cand The result of multiplication between ) It is an element in the matrix, representing the multiplication interaction result between the connection site selected by action 1 and the graph representation information of the intermediate molecular graph; ECFP(m cand Z represents the extended connectivity fingerprint of all candidate molecular fragments; 3rd It is a matrix, representing The result of the multiplication interaction between Ucand and Ucand; It is an element in the matrix, representing the multiplicative interaction information between the connection site selected by action 2 and the extended connectivity fingerprint representation of the selected molecular fragment; U cand This provides node characterization information for all connection sites on the selected molecular fragment. H att and Ucand The node information of connectable sites in intermediate molecules and selected molecular fragments is obtained by aggregating the data using a graph convolutional neural network (GCN). act1 p act2 p act3 Let represent the probability distributions of action 1 (act1), action 2 (act2), and action 3 (act3), respectively.

[0101] The selection of the molecular connection site in the current intermediate state and the maximum probability value of the selected molecular fragment connection site are obtained by solving the Softmax function, that is, solving for the maximum probability values ​​of action 1 (act1) and action 3 (act3) based on the Softmax function. The Softmax function is used in multi-class classification, mapping the outputs of multiple neurons to the (0,1) interval, and its formula is shown below:

[0102]

[0103] Where z is an array with j elements; z i This represents the i-th element in array z.

[0104] Preferably, when the policy network π2 makes molecular segment decisions, due to different states s t Not all bioactive molecular fragments can be selected; therefore, this invention utilizes the logits value v output by the strategy network π2. logits Multiplied by the molecular fragment mask term v mask Then, the most suitable usable molecular fragment is obtained based on the Gumbel Softmax algorithm, as shown in the following expression:

[0105]

[0106] F = GumbelSoftmax(v F ,τ); (Formula 7)

[0107] Among them, v logits This represents the logits value output by the policy network π2; v mask τ represents the molecular fragment mask term; τ represents the temperature function. represents dot product; F represents the final selected molecular fragment.

[0108] In some embodiments of the present invention, the deep reinforcement learning model is based on the maximum entropy Soft ActorCritic (SAC) deep reinforcement learning framework, and the following objective function is set:

[0109]

[0110] Among them, st As the current state, a t For the current action, R(s) t a t ) represents state s t Perform a t The immediate reward score for the action, α is the balanced reward score r(s) t a t ) and exploration ability hyperparameters, For state s t The action probability entropy, ρ π This represents the state-action transition distribution generated by policy π. Indicates that in state s t The next strategy π generates action a t At that time, the expected value of the model's instantaneous reward score and action probability entropy, This means selecting the strategy that maximizes the sum of the reward score and the action probability entropy from all possible strategies.

[0111] It should be understood that the Actor-critic framework starts with a basic policy, calculates the reward value for each action, and then updates the policy based on the maximum reward value. The goal is to gradually improve the policy by focusing on the maximum reward value. Soft Actor-critic is an off-policy algorithm developed for maximum entropy reinforcement learning, which has certain advantages over deterministic policies.

[0112] For typical deep reinforcement learning models, the learning objective is straightforward: to learn a policy that maximizes the expected value of the accumulated reward. The specific formula is as follows:

[0113]

[0114] However, the purpose of maximum entropy reinforcement learning used in this embodiment of the invention is to randomize the policy, that is, to distribute the probability of each output action as widely as possible, rather than concentrating it on a single action. Maximum entropy reinforcement learning is suitable for situations where there is not a unique way to achieve a goal, and there are many implementation paths (i.e., many actions). Therefore, a deep reinforcement learning model needs to be able to provide a random policy that can output the probability of each action in each state. For example, if there are three optimal actions with the same maximum probability, then one of these actions can be randomly selected as the output action. By using a maximum entropy deep reinforcement learning model, there are several advantages: First, the learned policy can serve as the initialization for more complex and specific tasks. Because through maximum entropy, the policy learns not just one way to solve the task, but all of them. Therefore, such a policy is more conducive to learning new tasks. Second, it has stronger exploratory capabilities and can more easily find better patterns under multimodal rewards. Finally, it is more robust because it explores various optimal possibilities in different ways, and therefore can more easily make adjustments when faced with disturbances.

[0115] In view of this, the maximum entropy reinforcement learning in this embodiment of the invention, in addition to the basic objective of affinity score, also requires maximizing the action entropy of each output of the policy. That is, this embodiment of the invention does not simply pursue the maximization of affinity score, but strives to balance any state (s t The reward score (i.e., affinity score) r(s) of the model under ) t a t and the ability to explore (i.e., molecular uniqueness). And make the optimal decision (π) * Select the molecular fragments that need to be spliced.

[0116] In some embodiments of the present invention, the reward function of the deep reinforcement learning model is based on affinity prediction using a physical docking simulation module. In other words, the physical docking simulation module is used as the reward function of the deep reinforcement learning model, and the predicted affinity value is used as the reward of the deep reinforcement learning model.

[0117] Preferably, to avoid bias and poor generalization ability caused by data-driven models, the present invention uses Auto-GPU as the reward function of the deep reinforcement learning model, which can further accelerate the docking simulation and affinity prediction process compared with the more widely used AutoDock docking prediction software.

[0118] In some embodiments, the deep reinforcement learning-based lead compound design method provided by the present invention performs different steps when designing covalent and non-covalent compounds, as follows:

[0119] In the process of designing covalent compounds, one Hinge Binder molecular fragment, one Linker molecular fragment, one Other Group molecular fragment, and one Warhead molecular fragment are selected from each molecular fragment library. The features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment are extracted and then input into the deep reinforcement learning model. The deep reinforcement learning model outputs the molecular fragments required for the design of the lead compound, which are then spliced ​​together to form a covalent compound.

[0120] In the process of designing non-covalent compounds, one Hinge Binder molecular fragment, one Linker molecular fragment, and two Other Group molecular fragments are selected from each molecular fragment library. The features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment are extracted and then input into the deep reinforcement learning model. The deep reinforcement learning model outputs the molecular fragments required for the design of the non-covalent compound, which are then spliced ​​together to form the non-covalent compound.

[0121] In some implementations, since covalent and non-covalent binding to receptors are two important drug action mechanisms in the field of kinase inhibitor design, the reward function scripts of deep reinforcement learning models are also divided into covalent docking scripts and non-covalent docking scripts.

[0122] The input to the non-covalent docking script consists of two parts: the receptor and the ligand. The receptor input is a .pdbqt format file of the receptor protein, and the ligand input is the SMILES expression of the first compound successfully generated in each episode of the deep reinforcement decision network. It should be understood that the process of the deep reinforcement decision network running through the entire training dataset is called an epoch; an epoch contains multiple episodes; and each episode completes a model validation, saving the optimal model.

[0123] In some examples, the implementation process of a non-covalent docking script includes the following steps:

[0124] Step a) Receptor file preprocessing, including ligand removal, water molecule removal, binding region segmentation, and receptor file format conversion in the receptor file; preprocessing can be completed using AutoDockTools 1.5.7 software.

[0125] Step b) Batch preprocessing of ligand files, including ligand hydrogenation, structural rationality judgment, 3D structure generation and structure optimization, can all be completed using the RDKit toolkit.

[0126] Step c) Obtain the atomic type of the input ligand, the center of the ligand binding region, the size of the ligand binding region, and the preprocessed receptor file based on the interatomic affinity grid maps, using Autogrid for calculation.

[0127] Step d) Affinity prediction, which includes inputting preprocessed ligand files and atomic bond affinity gridmaps, Autodock-GPU generates different poses by continuously changing the ligand rotatable angle, etc., and calculates the predicted affinity of each pose to the receptor, outputting the score of the highest affinity prediction value and the corresponding binding pose.

[0128] Step a) is the receptor file preprocessing completed before the model runs; steps b), c), and d) are part of the process of real-time affinity calculation as part of the model environment.

[0129] In some examples, the implementation of a covalent docking script includes the following steps:

[0130] Step a) Receptor file preprocessing, including obtaining covalently bound amino acid information, ligand removal, water molecule removal, covalently bound region segmentation, and receptor file format conversion; the covalently bound amino acid information can be obtained through AutoDockTools 1.5.7.

[0131] Step b) Batch preprocessing of ligand files, including ligand hydrogenation, covalent reaction simulation, 3D structure generation, and structure optimization. The covalent reaction simulation is performed using a reaction predictor written in the RDKit toolkit, primarily including predictions of cysteine ​​and lysine covalent reactions.

[0132] Step c) Ligand covalent processing: Input the pdbqt format file of the receptor protein and the atom numbers of the amino acid side chains after the ligand covalent reaction, the ligand structure after the covalent reaction, and the amino acid information of the covalent reaction obtained from step b). Use the meeko package (https: / / github.com / forlilab / Meeko) to fix the ligand atoms and calibrate the atomic coordinates.

[0133] Step d) Obtaining atomic affinity grid maps: Input the ligand atom type, ligand binding space center and size, and preprocessed acceptor file, and calculate using Autogrid.

[0134] Step e) Affinity calculation: Input the preprocessed ligand file and mesh diagram, and use AutoDock-GPU to perform the calculation to obtain the predicted affinity value.

[0135] Step a) is the receptor file preprocessing completed before the model runs, and steps b), c), d), and e) participate in the real-time affinity calculation as part of the model environment.

[0136] In some preferred implementations, based on a stochastic policy maximum entropy reinforcement learning model, to enhance the robustness and other performance characteristics of the reinforcement learning model, the Munchausen reinforcement learning technique is deployed to a SoftActor Critic model for discrete spaces. The Munchausen reinforcement learning technique utilizes the "current policy" π θ (a t |s t To guide the agent in learning the value function Q i Specifically, the Munchausen reinforcement learning technique introduces the "current policy" π during the temporal difference process. θ (a t |s t The term is used as part of the reward term to modify the update of the value function Q:

[0137]

[0138] Where r mt For the Munchausen award, r t For time step t, α is the reward item. m ∈[0,1], where τ is the conversion factor and τ is the temperature function. Represents state s t Action a t The distribution of .

[0139] Furthermore, for each state sequence (s) t ,a t ,r t ,s t+1 Update the action value function Q using the following formula. i ,as follows:

[0140]

[0141]

[0142] in, θ represents the loss function of the i-th Q-network; i Denotes the parameters of the i-th Q-network; D represents the empirical replay buffer; a′ is the next action calculated based on the output of the Actor network; γ is the hyperparameter discount factor; d t This is a termination term, indicating whether the round ends after this time step.

[0143] The Munchausen reinforcement learning technique introduces the Munchausen term lnπ during the model's temporal differencing process. θ (a t |s t This implicitly utilizes the KL divergence between the old and new strategies to limit the error caused by excessive differences between the old and new strategies and suppress the negative impact of the overestimation function Q.

[0144] Furthermore, in step S319 above, the objective loss function of the actor network is as follows:

[0145]

[0146] in, The parameter is Action network, The parameter is θ i The action value function is given by α, which is a temperature function. The loss function for temperature function α is as follows:

[0147]

[0148] in is a constant vector, equal to the hyperparameter representing the target entropy.

[0149] In some preferred implementations, priority experience replay is introduced to enhance the model's exploratory capabilities. In this invention, if an agent has never experienced a certain state before, that state is considered to have priority. To determine whether a state has priority, the priority prediction function is defined as a reward predictor, which includes a graph data encoder and a 3-layer fully connected network. The reward predictor is updated after each molecular docking process. When a new state appears, the reward predictor's prediction result will have a higher error, while for states that have already appeared, the prediction result of the reward predictor's prediction result has a smaller error. Therefore, this invention uses the prediction error of the reward predictor to determine the priority of any state.

[0150] Specifically, such as Figure 3 As shown, after each decision-making process in the Policy Networks is completed, the trajectory of the current state (i.e., the process of splicing different bioactive molecular fragments) is stored in the Replay Buffer. When the reinforcement learning model is updated, the trajectory is extracted from the Replay Buffer using the priority experience replay technique, and the critic and policy are updated using SAC reinforcement learning and the Munchausen RL technique.

[0151] To demonstrate the technical effectiveness of the method for generating the lead compound of the present invention, the generation of a kinase inhibitor is used as an example for illustration. In the following examples, molecular information was obtained using the RDKit toolkit, and graph data was constructed using the DGL toolkit; the deep learning framework used was PyTorch 1.12.0, the operating system was Ubuntu LTS 20.04, and all computations were performed on a workstation with an i9 10900K CPU and an NVIDIA GeForce RTX-3090 GPU.

[0152] Table 1 shows the average docking score of the top 5% of molecules generated by different models for the non-covalent inhibitor generation task targeting Bruton's tyrosine kinase (BTK), calculated by Autodock-GPU. A higher value indicates better model performance. The results show that the Kingrose model of this invention has a significant advantage over other mainstream models under this task condition. Compared to the reinforcement learning model FREED, which generates molecules based on a single molecular fragment library, using multiple molecular fragment libraries with different chemical properties for molecule generation shows certain advantages, but whether or not the Munchausen technique is used has little impact on this value.

[0153] Table 1 Top 5% of non-covalently generated molecular docking scores

[0154] Model Name mean Standard deviation variance Kingrose (This invention) 18.9358 0.60948 0.37147 mFREED 18.109 0.1011 0.01022 FREED 18.1504 0.15913 0.02532 REINVENT 15.4155 0.77776 0.60491 Morld 15.3535 0.2775 0.07701

[0155] Table 2 shows the proportion of molecules generated by different models that, when performing a non-covalent inhibitor generation task targeting Bruton's tyrosine kinase (BTK), achieved docking scores calculated by Autodock-GPU that were higher than the average docking scores of BTK inhibitors in the ChEMBL database. A higher proportion indicates better model performance. The results show that the Kingrose model of this invention has a significant advantage over other mainstream models in this regard. This further validates the advantages of using multiple molecular fragments with different chemical properties for molecule generation, and that the Munchausen technique can improve the model's results on this metric.

[0156] Table 2. Proportion of non-covalently generated molecules with docking scores higher than the average docking score in the ChEMBL database.

[0157] Model Name mean Standard deviation variance kingrose (This invention) 0.2644 0.04863 0.00237 mFREED 0.2298 0.01736 0.0003 FREED 0.2172 0.02594 0.00067 REINVENT 0.01175 0.00507 0.00003 Morld 0.0155 0.0045 0.00002

[0158] Table 3 shows the average docking score of molecules generated by the Kingrose model in this invention during the random sampling phase and the post-training generation phase when performing covalent and non-covalent inhibitor generation tasks targeting Bruton's tyrosine kinase (BTK). A higher value indicates a better model performance. The analysis results show that compared to randomly selecting molecular fragments from different molecular fragment libraries and splicing them into new molecules, the Kingrose model generates molecules with a higher docking score, indicating that the model has been effectively learned. Furthermore, the Kingrose model can not only perform de novo design of non-covalent inhibitors optimized by affinity for a specific protein, but also innovatively achieve de novo design of covalent inhibitors optimized by affinity for proteins with covalent reaction conditions.

[0159] Table 3. Mean Affinity of Generated Molecules

[0160] Covalently formed molecules (standard deviation) Non-covalently formed molecules (standard deviation) Kingrose (This invention) 13.58(0.42) 15.25(0.42) Random 12.59(0.03) 13.76(0.16)

[0161] Table 4 shows the main properties of the molecules generated by the Kingrose model of this invention when performing covalent and non-covalent inhibitor generation tasks targeting Bruton's tyrosine kinase (BTK). The PAINS and Glaxo columns represent the proportion of molecules generated by different models that do not contain the alarm structures defined by PAINS and Glaxo. The Validity column represents the proportion of molecules generated by different models that do not contain obvious chemical structural errors. The Uniqueness column represents the proportion of non-repeating molecules in the generated molecules. Higher proportions of these four factors indicate better generated molecules. The results show that the Kingrose model of this invention has superior overall performance compared to other models, containing fewer alarm structures and generating molecules without obvious structural errors.

[0162] Table 4 Properties of generated molecules

[0163]

[0164] like Figure 4 The diagram illustrates the structure of a lead compound design system based on deep reinforcement learning, as shown in an embodiment of the present invention. The lead compound design system 400 includes: a molecular fragment library construction module 401, a feature extraction module 402, and a model construction module 403.

[0165] The molecular fragment library construction module 401 is used to construct a molecular fragment library containing multiple different bioactive molecular fragments required to constitute the lead compound. The feature extraction module 402 is used to construct molecular graph data for each of the molecular fragment libraries and to extract features from the molecular graph data based on a graph convolutional neural network. The model construction module 403 is used to take the extracted features of the molecular graph data and the extended connectivity fingerprint features of each molecular fragment in the molecular fragment library as input parameters for a deep reinforcement learning model, in order to select specific bioactive molecular fragments and corresponding connection sites for splicing to form molecular fragments used to constitute the lead compound.

[0166] In some embodiments of this example, the lead compound design system 400 further includes a covalent compound design module 404 and a non-covalent compound design module 405.

[0167] The covalent compound design module 404 is used to select one Hinge Binder molecular fragment, one Linker molecular fragment, one Other Group molecular fragment, and one Warhead molecular fragment from each molecular fragment library; extract the features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment and input them into the deep reinforcement learning model, so that the deep reinforcement learning model can output the molecular fragments required for the covalent compound to be designed, so as to splice them to form a covalent compound.

[0168] The non-covalent compound design module 405 is used to select one Hinge Binder molecular fragment, one Linker molecular fragment, and two Other Group molecular fragments from each molecular fragment library; extract the features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment and input them into the deep reinforcement learning model, so that the deep reinforcement learning model can output the molecular fragments required for the design of the non-covalent compound, so as to splice them to form a non-covalent compound.

[0169] It should be noted that the deep reinforcement learning-based lead compound design system provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the deep reinforcement learning-based covalent / non-covalent kinase inhibitor design device and the deep reinforcement learning-based lead compound design method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0170] The lead compound design method based on deep reinforcement learning provided in this invention can be implemented on the terminal side or the server side. For the hardware structure of the terminal for designing covalent / non-covalent kinase inhibitors based on deep reinforcement learning, please refer to [link to relevant documentation]. Figure 5 This is a schematic diagram of an optional hardware structure of a computer device 500 provided in an embodiment of the present invention. The computer device 500 can be a mobile phone, tablet device, personal digital processing device, factory back-end processing device, etc. The computer device 500 includes: at least one processor 501, a memory 502, at least one network interface 504, and a user interface 506. The various components in the device are coupled together through a bus system 505. It is understood that the bus system 505 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general will label all buses as bus systems.

[0171] The user interface 506 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0172] It is understood that memory 502 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0173] In this embodiment of the invention, the memory 502 is used to store various types of data to support the operation of the deep reinforcement learning-based covalent / non-covalent kinase inhibitor design terminal 500. Examples of this data include: any executable program for operation on the deep reinforcement learning-based covalent / non-covalent kinase inhibitor design terminal 500, such as operating system 5021 and application program 5022; operating system 5021 includes various system programs, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. Application program 5022 may include various applications, such as media player, browser, etc., for implementing various application services. The deep reinforcement learning-based lead compound design method provided in this embodiment of the invention can be included in application program 5022.

[0174] The methods disclosed in the above embodiments of the present invention can be applied to processor 501, or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 501 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0175] In an exemplary embodiment, the deep reinforcement learning-based covalent / non-covalent kinase inhibitor design terminal 500 can be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0176] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0177] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0178] In summary, this application provides a method, apparatus, terminal, and medium for designing lead compounds based on deep reinforcement learning. This invention innovatively achieves de novo design of covalent inhibitors targeting affinity, proposes an innovative method for constructing novel drug molecules based on multiple different bioactive fragments, and optimizes the model's exploration space. The use of the physics computation software Autodock-GPU as a scoring function enhances the model's generalization ability and running speed. The application of the Munchhausen reinforcement learning technique to the Soft Actor Critic discrete action space model improves the model's optimization performance. Compared to other models, this application expands and improves upon the model's functionality and the properties of the generated molecules. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial applicability.

[0179] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A lead compound design method based on deep reinforcement learning, characterized in that, include: To construct a molecular fragment library containing multiple different bioactive molecular fragments required to form lead compounds; Molecular graph data for each of the aforementioned molecular fragment libraries are constructed, and features are extracted from the molecular graph data based on a graph convolutional neural network. The features of the extracted molecular graph data and the extended connectivity fingerprint features of each molecular fragment in the molecular fragment library are used as input parameters of the deep reinforcement learning model to select specific bioactive molecular fragments and corresponding connection sites for splicing to form molecular fragments that constitute lead compounds. The deep reinforcement learning model is constructed through the following steps: The agent in the deep reinforcement learning model perceives the environment and performs actions such as selecting molecular fragments to be spliced ​​and splicing sites. The environment splices molecular fragments according to the selected molecular fragments to be spliced ​​and splicing sites, causing changes in the state of the environment, and predicting receptor-ligand affinity. The predicted affinity value is used as the reward score for the action performed in the previous step. The agent updates its strategy at least based on the reward score; Repeat the above steps until a deep reinforcement learning model capable of generating the ideal lead compound is obtained; The deep reinforcement learning model consists of three 3-layer fully connected networks, namely the policy network. Given a molecule in an intermediate state, after feature extraction by the encoder, the model's decision-making process includes three steps: selecting the connection sites of the current intermediate molecule (action 1), selecting molecular fragments from the candidate molecular fragment library (action 2), and selecting the connection sites of the selected molecular fragment (action 3). The probability distribution of each action includes: ； ； ； in, Let represent the probability distributions of action 1, action 2, and action 3 (act3), respectively; MI represents the multiplication interaction. express The results of the multiplication interactions between them are in matrix form; The graphical representation information of the intermediate molecules in the current state; Node characterization information representing all connection sites of the intermediate molecule in the current state; Let be a matrix, representing and The result of the multiplication interaction between them; It is an element in the matrix, representing the result of the multiplication interaction between the connection site selected by action 1 and the graph representation information of the intermediate molecular graph; An extended connectivity fingerprint representation for all candidate molecular fragments; It is a matrix, representing and The result of the multiplication interaction between them; It is an element in the matrix, representing the multiplication interaction information between the connection site selected by action 2 and the extended connectivity fingerprint representation of the selected molecular fragment; This provides node characterization information for all connection sites on the selected molecular fragment.

2. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, include: In the process of designing covalent compounds, one Hinge Binder molecular fragment, one Linker molecular fragment, one Other Group molecular fragment, and one Warhead molecular fragment are selected from each molecular fragment library. The features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment are extracted and then input into the deep reinforcement learning model. The deep reinforcement learning model outputs the molecular fragments required for the covalent compound to be designed, which are then spliced ​​together to form the covalent compound. And / or, in the process of designing non-covalent compounds, one HingeBinder molecular fragment, one Linker molecular fragment, and two Other Group molecular fragments are selected from each molecular fragment library respectively; the features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment are extracted and input into the deep reinforcement learning model, so that the deep reinforcement learning model outputs the molecular fragments required for the design of the non-covalent compound, which are then spliced ​​together to form the non-covalent compound.

3. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, The process of constructing the molecular fragment library includes: extracting effective molecules and candidate molecular fragments from relevant literature or databases and generating molecular fragment files in database format; reading molecular fragment files in database format from different sources, splitting and screening molecular fragments with different biological activities using corresponding screening conditions to form different molecular fragments; and storing the screened molecular fragments in a preset molecular language format to form the corresponding molecular fragment library.

4. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, A graph convolutional neural network with a 3-layer structure and ReLU activation function in each layer is selected to extract features from the molecular graph data; the propagation mode between network layers is as follows: ； in, For the first Layer node information; It is an aggregate function; To modify the activation function of the linear unit; For the first Layer node information; , Let be the adjacency matrix of the graph nodes. Represents the identity matrix; This is the output of the previous layer, and its shape is... , The number of nodes in the graph. For the first Number of output channels for The transformation parameter matrix of a convolutional layer; yes The calculated degree matrix, ; Features for each layer.

5. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, In the policy network In the process of making decisions about molecular fragments, the policy network is used. The output logits value is multiplied by the molecular fragment mask term, and then the most suitable usable molecular fragment is obtained based on the Gumbel Softmax algorithm.

6. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning model is based on the maximum entropy Soft Actor Critic deep reinforcement learning framework, and its objective function includes: ； in, This is the current state. For the current action, For state conduct The immediate reward score for the action, where α is the balanced reward score. hyperparameters, For state The action probability entropy, Indicates by strategy The resulting state-action transition distribution Indicates the state Next strategy Generate Actions At that time, the expected value of the model's instantaneous reward score and action probability entropy, This means selecting the strategy that maximizes the sum of the reward score and the action probability entropy from all possible strategies.

7. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, The method also includes using Auto-GPU as the reward function for the deep reinforcement learning model, and using the predicted affinity value as the reward for the deep reinforcement learning model.

8. The lead compound design method based on deep reinforcement learning according to claim 7, characterized in that, The reward function script of the deep reinforcement learning model includes covalent docking scripts and non-covalent docking scripts.

9. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, Based on the Munchausen reinforcement learning algorithm, a Munchausen term is introduced into the temporal difference process of the deep reinforcement learning model to limit the error caused by excessive differences between the old and new policies.

10. The lead compound design method based on deep reinforcement learning according to claim 1, characterized in that, A reward predictor is set in the deep reinforcement learning model; the reward predictor is updated after each molecular docking process; when a new state appears, the prediction result of the reward predictor will have a higher error, while for states that have already appeared, the prediction result of the reward predictor will have a smaller error.

11. A lead compound design system based on deep reinforcement learning, characterized in that, include: The molecular fragment library construction module is used to construct a molecular fragment library containing multiple different bioactive molecular fragments required to form a lead compound; The feature extraction module is used to construct molecular graph data for each of the molecular fragment libraries and to extract features from the molecular graph data based on a graph convolutional neural network. The model building module is used to take the features of the extracted molecular graph data and the extended connectivity fingerprint features of each molecular fragment in the molecular fragment library as input parameters for a deep reinforcement learning model. This allows for the selection of specific bioactive molecular fragments and corresponding connection sites for splicing into molecular fragments that can form lead compounds. The deep reinforcement learning model is constructed through the following steps: the agent of the deep reinforcement learning model perceives the environment and performs the action of selecting the molecular fragments to be spliced ​​and the splicing sites; the environment splices the molecular fragments according to the selected fragments and splicing sites, causing a change in the state of the environment and predicting receptor-ligand affinity. The predicted affinity value is used as the reward score for the action performed in the previous step; the agent updates its policy at least based on the reward score; the above steps are repeated until a deep reinforcement learning model capable of generating ideal lead compounds is obtained. The deep reinforcement learning model consists of three 3-layer fully connected networks, namely the policy network. Given a molecule in an intermediate state, after feature extraction by the encoder, the model's decision-making process includes three steps: selecting the connection sites of the current intermediate molecule (action 1), selecting molecular fragments from the candidate molecular fragment library (action 2), and selecting the connection sites of the selected molecular fragment (action 3). The probability distribution of each action includes: ； ； ； in, They represent action 1, action 2, and action 3 respectively. The probability distribution of ). Indicates multiplication interaction; express The results of the multiplication interactions between them are in matrix form; The graphical representation information of the intermediate molecules in the current state; Node characterization information representing all connection sites of the intermediate molecule in the current state; Let be a matrix, representing and The result of the multiplication interaction between them; It is an element in the matrix, representing the result of the multiplication interaction between the connection site selected by action 1 and the graph representation information of the intermediate molecular graph; An extended connectivity fingerprint representation for all candidate molecular fragments; It is a matrix, representing and The result of the multiplication interaction between them; It is an element in the matrix, representing the connection point selected by action 2 and... It is the multiplicative interaction information of the extended connectivity fingerprint representation of selected molecular fragments; This provides node characterization information for all connection sites on the selected molecular fragment.

12. The lead compound design system based on deep reinforcement learning according to claim 11, characterized in that, Also includes: The covalent compound design module is used to select one Hinge Binder molecular fragment, one Linker molecular fragment, one Other Group molecular fragment, and one Warhead molecular fragment from each molecular fragment library; extract the features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment and input them into the deep reinforcement learning model, so that the deep reinforcement learning model can output the molecular fragments required for the covalent compound to be designed, so as to splice them to form a covalent compound; The non-covalent compound design module is used to select one Hinge Binder molecular fragment, one Linker molecular fragment, and two Other Group molecular fragments from each molecular fragment library; extract the features of the molecular graph and the extended connectivity fingerprint features of each molecular fragment and input them into the deep reinforcement learning model, so that the deep reinforcement learning model can output the molecular fragments required for the design of the non-covalent compound, so as to splice them to form the non-covalent compound.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the lead compound design method based on deep reinforcement learning as described in any one of claims 1 to 10.

14. A computer device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory to enable the terminal to perform the lead compound design method based on deep reinforcement learning as described in any one of claims 1 to 10.

Images