Method for generating molecule on basis of reinforcement learning model

Abstract

Disclosed is a method, performed by a computing device, for generating a molecule on the basis of a reinforcement learning model, according to an embodiment of the present disclosure. The method comprises the steps of: inputting target pocket information into a molecule generation model; performing a reinforcement learning process on the basis of the target pocket information by using the molecule generation module; and generating a final molecule corresponding to the target pocket information on the basis of the reinforcement learning process by using the molecule generation model, wherein the reinforcement learning process may use an action or state associated with a partially generated molecule.

Description

A molecular generation method based on a reinforcement learning model

[0001] The present disclosure relates to a pocket-conditioned and reinforcement learning-based molecule generation method, and more specifically, to a method for generating molecules corresponding to pocket information based on a reinforcement learning (RL) process using a molecule generation model.

[0002]

[0003] Identifying molecules that effectively bind to and inhibit or activate target proteins is a critical challenge in the field of new drug development. Existing molecular generation methods have focused on generating molecules that possess desired chemical properties and bind to target proteins, but suffer from several fundamental limitations. First, these techniques do not explicitly utilize the target protein's binding site, but rather indirectly assess target protein compatibility based on the molecule's properties. This approach fails to adequately reflect local constraints on protein binding and can limit the accuracy of molecule generation. Furthermore, these techniques require retraining whenever the target protein changes, making it difficult to generate molecules for new targets. Furthermore, the molecules generated by these techniques often have unrealistic structures, which can compromise their drug potential and chemical efficacy.

[0004] To address the above issues, pocket-conditioned molecule generation technology was introduced. Building on advances in protein structure prediction technologies like AlphaFold, this technology uses 3D binding pocket information as input to generate molecules compatible with specific binding sites. Furthermore, moving beyond the conventional 2D-based string generation method, it generates 3D molecular structures that interact with the pocket information, thereby improving compatibility with the target protein.

[0005] However, these techniques still present several challenges. For example, the resulting molecules may exhibit unrealistic stereochemical problems, such as non-planar ring structures or inappropriate bond distances, and may limit the optimization of drug compatibility or synthetic feasibility. Furthermore, these techniques simply mimic the distribution of training data rather than learning molecules optimized for specific target pockets, which can hinder the generation of molecules optimized for specific target properties.

[0006] Therefore, a new approach is required to solve the problems of pocket condition molecule production technology.

[0007] Meanwhile, U.S. Patent Publication No. 2024-0344123 (October 17, 2024) discloses a computer-implemented automated synthesis platform configured to use an artificial intelligence (AI) engine.

[0008]

[0009] The present disclosure aims to provide a molecule generation method based on pocket conditions and reinforcement learning. For example, the present disclosure aims to provide a technique for generating a final molecule corresponding to target pocket information based on target pocket information and a reinforcement learning process, utilizing a molecule generation model.

[0010] Meanwhile, the technical task to be achieved by the present disclosure is not limited to the technical task mentioned above, and may include various technical tasks within a scope obvious to a person skilled in the art from the contents described below.

[0011]

[0012] According to one embodiment of the present disclosure for realizing the task as described above, a method for generating molecules based on a reinforcement learning model, which is performed by a computing device, is disclosed. The method includes the steps of inputting target pocket information into a molecule generation model, performing a reinforcement learning process based on the target pocket information using the molecule generation model, and generating a final molecule corresponding to the target pocket information based on the reinforcement learning process using the molecule generation model, wherein the reinforcement learning process may utilize an action or state associated with a partially generated molecule.

[0013] In one embodiment, the molecular generation model may be a model that generates the final molecule in an atom-autoregressive manner based on the target pocket information of the target protein.

[0014] In one embodiment, the state of the reinforcement learning process may include state information related to the partially generated molecule and the target pocket.

[0015] In one embodiment, the state information related to the partially generated molecule and the target pocket may include graph information in which the state information related to the partially generated molecule and the target pocket are jointly expressed, wherein a node of the graph corresponds to an atom, an edge within the partially generated molecule in the graph corresponds to a bond, and an edge between the partially generated molecule and the target pocket in the graph may correspond to a spatial proximity.

[0016] In one embodiment, the action of the reinforcement learning process may include an atomic add action, or a stop action.

[0017] In one embodiment, the atom addition action may be performed based on index information of a reference atom to which a new atom is to be bonded among the atoms of the partially generated molecule, coordinate information of the reference atom, type information of the new atom, and bond information of the reference atom and the new atom.

[0018] In one embodiment, an episode of the reinforcement learning process may be obtained based on an operation of sampling the atomic addition action based on a softmax function and a Gaussian mixture distribution, and an operation of returning the stop action based on a sigmoid function.

[0019] In one embodiment, parameters associated with the operation of sampling the atomic addition behavior or the operation of returning the stop behavior may be set differently during the learning process and the inference process of the molecular generation model.

[0020] In one embodiment, the parameters that are set differently in the learning process and the inference process of the molecular generation model may include a threshold value associated with the sigmoid function and a covariance value of the Gaussian mixture distribution.

[0021] In one embodiment, the episode of the reinforcement learning process may include at least one of a success episode in which the stop action is returned and the final molecule corresponds to a valid molecule, a first type of failure episode in which the stop action is returned but the final molecule corresponds to an invalid molecule, a second type of failure episode in which the atom addition action is repeated beyond a predetermined threshold, or a third type of failure episode in which the intermediate molecule is detected to have a nonsensical structure.

[0022] In one embodiment, the reward of the reinforcement learning process may be determined based on a success reward for the success episode and a failure penalty reward for all types of failure episodes.

[0023] In one embodiment, the success reward may be determined based on properties of interest and coefficients for each property, and the failure penalty reward may be determined based on the mean and standard deviation of the success rewards.

[0024] According to one embodiment of the present disclosure for realizing the above-described task, a computer program stored in a computer-readable storage medium is disclosed, which causes a computing device to perform operations for generating molecules based on a reinforcement learning model. The operations include: inputting target pocket information into a molecule generation model, performing a reinforcement learning process based on the target pocket information using the molecule generation model, and generating a final molecule corresponding to the target pocket information based on the reinforcement learning process using the molecule generation model, wherein the reinforcement learning process may utilize an action or state associated with a partially generated molecule.

[0025] In one embodiment, the molecular generation model may be a model that generates the final molecule in an atom-autoregressive manner based on the target pocket information of the target protein.

[0026] In one embodiment, the state of the reinforcement learning process may include state information related to the partially generated molecule and the target pocket.

[0027] In one embodiment, the action of the reinforcement learning process comprises an atom addition action or a stop action, and the atom addition action may include information about a reference atom to which a new atom among the atoms of the partially generated molecule is to be bonded, and information about the new atom.

[0028] According to one embodiment of the present disclosure for realizing the task as described above, a computing device for generating molecules based on a reinforcement learning model is disclosed. The device includes one or more processors and a memory, wherein the one or more processors are configured to: input target pocket information into a molecule generation model, perform a reinforcement learning process based on the target pocket information using the molecule generation model, and generate a final molecule corresponding to the target pocket information based on the reinforcement learning process using the molecule generation model, wherein the reinforcement learning process can utilize an action or state associated with a partially generated molecule.

[0029] In one embodiment, the molecular generation model may be a model that generates the final molecule in an atom-autoregressive manner based on the target pocket information of the target protein.

[0030] In one embodiment, the state of the reinforcement learning process may include state information related to the partially generated molecule and the target pocket.

[0031] In one embodiment, the action of the reinforcement learning process comprises an atom addition action or a stop action, and the atom addition action may include information about a reference atom to which a new atom among the atoms of the partially generated molecule is to be bonded, and information about the new atom.

[0032]

[0033] The present disclosure can provide a molecule generation method based on pocket conditions and reinforcement learning. For example, the present disclosure can input target pocket information into a molecule generation model, utilize the molecule generation model, and generate a final molecule corresponding to the target pocket information based on the target pocket information and a reinforcement learning process.

[0034] Furthermore, the present disclosure can address problems associated with conventional pocket-conditional molecule generation techniques. For example, the present disclosure can address unrealistic stereochemical issues, such as the generation of molecules with non-planar ring structures or inappropriate bond distances. Furthermore, the present disclosure can address the problem of generating molecules by simply simulating the distribution of training data. Furthermore, the present disclosure can generate molecules optimized for target properties.

[0035] Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within a range apparent to those skilled in the art from the contents described below.

[0036]

[0037] FIG. 1 is a block diagram of a computing device for generating molecules based on a reinforcement learning model according to one embodiment of the present disclosure.

[0038] FIG. 2 is a schematic diagram illustrating a network function according to one embodiment of the present disclosure.

[0039] FIG. 3 is a conceptual diagram illustrating a reinforcement learning algorithm according to one embodiment of the present disclosure.

[0040] FIG. 4 is a flowchart illustrating a molecule generation method according to one embodiment of the present disclosure.

[0041] FIG. 5 is a schematic diagram illustrating a reinforcement learning process according to one embodiment of the present disclosure.

[0042] FIG. 6 is a schematic diagram illustrating the actions and states of a reinforcement learning process according to one embodiment of the present disclosure.

[0043] Figures 7a to 7c are drawings illustrating a molecule production process according to an example of the present disclosure.

[0044] FIG. 8 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0045]

[0046] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific details.

[0047] As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and / or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and / or data transmitted to another system via a network such as the Internet via signals).

[0048] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

[0049] Additionally, the terms "comprises" and / or "comprising" should be understood to imply the presence of the features and / or components in question. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, components, and / or groups thereof. Furthermore, unless otherwise specified or clear from the context to refer to the singular form, the singular in the specification and claims should generally be construed to mean "one or more."

[0050] And, the term "at least one of A or B" should be interpreted to mean "if it includes only A", "if it includes only B", or "if it is combined in the composition of A and B".

[0051] Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0052] The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not limited to the embodiments disclosed herein. The present invention is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

[0053] In the present disclosure, network function and artificial neural network and neural network can be used interchangeably.

[0054]

[0055] FIG. 1 is a block diagram of a computing device for generating molecules based on a reinforcement learning model according to one embodiment of the present disclosure.

[0056] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0057] A computing device (100) may include a processor (110), memory (130), and network unit (150).

[0058] The processor (110) may be configured with one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in the memory (130) and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network model. The processor (110) may perform calculations for learning a neural network model, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network model using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process learning of the neural network model. For example, a CPU and a GPGPU can work together to train a neural network model and classify data using the neural network model. Furthermore, in one embodiment of the present disclosure, processors of multiple computing devices can be used together to train a neural network model and classify data using the neural network model. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0059] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0060] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the internet. The description of the above-described memory is merely an example, and the present disclosure is not limited thereto.

[0061] The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0062] In addition, the network unit (150) presented in the present disclosure can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

[0063] In the present disclosure, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth. The technologies described in the present disclosure may also be used in other networks mentioned above.

[0064]

[0065] FIG. 2 is a schematic diagram illustrating a network function according to one embodiment of the present disclosure.

[0066] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. A neural network may be comprised of a set of interconnected computational units, generally referred to as nodes. These nodes may also be referred to as neurons. A neural network comprises at least one node. The nodes (or neurons) comprising a neural network may be interconnected by one or more links.

[0067] Within a neural network, one or more nodes connected via links can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, meaning that any node that is in an output node relationship with one node can also be in an input node relationship with another node, and vice versa. As described above, the relationship between input nodes and output nodes can be created based on links. One input node can be connected to one or more output nodes via links, and vice versa.

[0068] In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and output nodes can have a weight. The weight can be variable and can be varied by the user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values ​​input to the input nodes connected to the output node and the weight set on the link corresponding to each input node.

[0069] As described above, a neural network is a network in which one or more nodes are interconnected through one or more links, forming input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links within the network, the relationships between the nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weight values ​​for the links, the two neural networks can be perceived as different from each other.

[0070] A neural network can be composed of a set of one or more nodes. A subset of the nodes comprising the neural network can form a layer. Some of the nodes comprising the neural network can form a layer based on their distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links required to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by its distance from the final output node.

[0071] An initial input node may refer to one or more nodes within a neural network into which data is directly input without going through links with other nodes. Alternatively, within a neural network, it may refer to nodes that do not have other input nodes connected by links in the relationship between nodes based on links. Similarly, a final output node may refer to one or more nodes within a neural network that do not have output nodes in their relationship with other nodes. Furthermore, a hidden node may refer to nodes that constitute a neural network other than the initial input node and the final output node.

[0072] A neural network according to one embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the above-described neural networks.

[0073] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using DNNs, one can identify latent structures in data. For example, one can identify the latent structures of images, text, videos, audio, and music (e.g., what objects are in the image, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). DNNs can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siamese networks, and generative adversarial networks (GANs). The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

[0074] In one embodiment of the present disclosure, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network that outputs output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be arranged between input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to the output layer (symmetrical to the input layer). The autoencoder may perform nonlinear dimensionality reduction. The number of input layers and output layers may correspond to the dimensionality after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may have a structure in which the number of nodes decreases as it moves away from the input layer. The number of nodes in the bottleneck layer (the layer with the fewest nodes between the encoder and decoder) may be kept above a certain number (e.g., more than half of the input layer), as too few nodes may not transmit enough information.

[0075] Neural networks can learn using at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Neural network learning can be the process of applying knowledge to the neural network to perform a specific action.

[0076] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. Supervised learning uses training data with the correct answer labeled for each training data (i.e., labeled training data). Unsupervised learning, on the other hand, may not have the correct answer labeled for each training data. For example, in the case of supervised learning for data classification, the training data may be data with each category labeled. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the training data labels. Alternatively, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network training to quickly achieve a certain level of performance, thereby increasing efficiency. A lower learning rate can be used in the later stages of training to increase accuracy.

[0077] In neural network training, training data can typically be a subset of real-world data (i.e., the data to be processed using the trained neural network). Therefore, there can be a learning cycle where errors on the training data decrease but errors on the real-world data increase. Overfitting is a phenomenon where excessive training on the training data leads to increased errors on the real-world data. For example, a neural network trained on yellow cats may fail to recognize cats when shown non-yellow colors, a type of overfitting. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. These methods include increasing the training data, regularization, dropout, which disables some nodes in the network during the learning process, and the use of batch normalization layers.

[0078]

[0079] FIG. 3 is a conceptual diagram illustrating a reinforcement learning model according to one embodiment of the present disclosure.

[0080] Reinforcement learning is a type of learning method that trains an artificial neural network model based on the rewards generated for actions selected by the artificial neural network model, enabling the model to determine better actions based on the input state. Reinforcement learning can be understood as a "trial and error learning method" in that rewards are given for decisions (i.e., actions). The rewards given to the artificial neural network model during reinforcement learning can be the accumulated rewards of multiple actions. Through learning, reinforcement learning creates an artificial neural network model that maximizes the reward itself or return (the sum of rewards) by considering various states and rewards based on actions. In this disclosure, the term "reinforcement learning model" refers to the entity that determines actions and can be used interchangeably with the term "agent." In the field of reinforcement learning, the term "environment" (Env) or "environment model" refers to a model that returns results that consider the agent's behavior. An environment model can be a model that returns output data (e.g., state information) in response to given input data (e.g., control information). An environment model can be a model whose structure from input to output or whose causal relationship between input and output data is unknown. The agent and environment can interact by exchanging data.

[0081] In Fig. 3, the reinforcement learning model (310) can be understood as an "agent" as a subject that determines actions based on state information and rewards. In the present disclosure, state information may include current state information and next state information. The current state information and the next state information can be distinguished based on the time or order in which the state information is acquired, and the current state information (S) can be distinguished based on the chronological relationship.t ) and the following status information (S t+1 ) can be named respectively.

[0082] In the present disclosure, the environment (330) may be a "current state information (S)" that can serve as a basis for the reinforcement learning model (310) to determine actions. t )" can be produced. The processor (110) outputs current state information (S) including at least one state variable from the environment (330). t ) can be obtained, the current state information can be input into the reinforcement learning model (310).

[0083] In the present disclosure, the processor (110) inputs current state information into the reinforcement learning model (310), and then performs an action (A) based on the reinforcement learning model (310). t ) can be produced. The reinforcement learning model (310) obtains state information (S) from the environment (330) at any point in time t. t ) can be used to compute a probability distribution regarding a plurality of selectable actions. The processor (110) can compute an action (A) based on the computed probability distribution. t ) can be produced. For example, the processor (110) may produce an action (A) with the largest value among the probability distributions for multiple actions. t ) can be determined.

[0084] In the present disclosure, the processor (110) can input an action calculated based on a reinforcement learning model (310) into the environment (330). The processor (110) can input the next state information (S) updated from the environment (330) as a result of the input of the action. t+1 ) and compensation (R t) can be obtained. Reinforcement learning in which the environment (330) knows the “reward function” that serves as the criterion for determining the reward or the “transition probability distribution function” that serves as the criterion for determining the next state information after the environment (330) receives an action from the reinforcement learning model (310) can be called “model-based” reinforcement learning. On the other hand, reinforcement learning in which the reward function of the environment (330) and the transition probability distribution function of the environment (330) are not known can be called “model-free” reinforcement learning.

[0085] A reinforcement learning model according to the present disclosure can be trained based on at least one episode. In the present disclosure, an "episode" may be used as a term meaning a data sequence having a serial order. An episode may be a data set composed of a plurality of N-tuple data each including N elements (N is a natural number greater than or equal to 1). The plurality of N-tuple data included in an episode may have a serial order. As an example of N-tuple data, when N is '4', each 4-tuple data may include current state information, action information, reward, and next state information as elements. As another example of N-tuple data, when N is '5', each 5-tuple data may include current state information, action information corresponding to the current state information, reward, next state information, and action information corresponding to the next state information as elements.

[0086] The processor (110) according to the present disclosure can obtain one episode by repeatedly performing a plurality of steps identical or similar to the above-described learning method embodiment from an initial state (t=0) to a final state (t=T). The final state may be derived when a preset termination condition is satisfied or may be derived when a preset number of steps have been performed. The step refers to at least one operation unit in which a reinforcement learning model receives a state, determines an action, and then receives reward or updated state information for the action. The preset number of steps may be set to any natural number, and may consist of, for example, 200 steps.

[0087] The processor (110) may train a reinforcement learning model based on at least one training data. For example, the processor (110) may train a reinforcement learning model based on training data corresponding to each step at the end of each step. In another example, the processor (110) may train a reinforcement learning model based on a training data set including training data for each of a plurality of steps at the end of each episode including a plurality of steps. In yet another example, the processor (110) may train a reinforcement learning model based on a training data set including training data for each of the steps after steps of a predetermined batch size have been performed. The batch size may be predetermined as any natural number.

[0088] According to the present disclosure, the process of learning a reinforcement learning model by the processor (110) may include a step of modifying each node weight or bias value of the neural network included in the reinforcement learning model. The step of modifying each node weight or bias value of the neural network included in the reinforcement learning model may be performed by the processor (110) in a method identical or similar to the backpropagation technique for the neural network described above with reference to FIG. 2. For example, the reward (R) included in the learning data (T_t) for any time point t t ) is positive, the processor (110) performs the action (A) included in the learning data (T_t). t ) can be adjusted to strengthen the weight or bias value of one or more nodes included in the reinforcement learning model. At this time, one or more nodes included in the reinforcement learning model are configured so that the reinforcement learning model can adjust the weight or bias value of one or more nodes included in the reinforcement learning model to strengthen the state information (S) included in the training data (T_t). t ) after entering the above action (A t ) may be a node involved in determining the

[0089] The learned reinforcement learning model (310) can determine an action for each state information so that the cumulative value (i.e., return) of the reward given from the environment (330) is maximized. The method by which the reinforcement learning model (310) determines an action may be based on at least one of a value-based action decision method, a policy-based action decision method, and an action decision method based on both value and policy, for example. The value-based action decision method is a method of determining the action that provides the highest value in each state based on a value function. Examples of the value-based action decision method may include Q-learning, DQN (Deep Q-Network), etc. The policy-based action decision method is a method of determining an action based on the final return and policy function without a value function. Examples of the policy-based action decision method may include the Policy Gradient technique, etc. The action decision method based on both value and policy is a method of determining the action of an agent by learning in a way that the policy function determines the action and the value function evaluates the action. Action decision-making methods based on both values ​​and policies may include, for example, the Actor-Critic algorithm, the Soft Actor-Critic algorithm, the A2C algorithm, and the A3C algorithm.

[0090] The specific descriptions regarding the learning of the aforementioned reinforcement learning model are for illustrative purposes only and do not limit the present disclosure.

[0091]

[0092] FIG. 4 is a flowchart illustrating a molecule generation method according to one embodiment of the present disclosure.

[0093] According to one embodiment of the present disclosure, steps S410 to S430 below may be performed by a computing device (e.g., the computing device (100) of FIG. 1). For example, the computing device may directly obtain information for molecule generation or receive it from an external system. The external system may be a server, database, or the like that stores and manages information about a molecule generation model or data for reinforcement learning. The computing device may include a processor (e.g., the processor (110) of FIG. 1), a memory (e.g., the memory (130) of FIG. 1), and a network unit (e.g., the network unit (150) of FIG. 1).

[0094] According to FIG. 4, the molecule generation method of the present disclosure may include a step of inputting target pocket information into a molecule generation model (S410), a step of performing a reinforcement learning process based on the target pocket information by utilizing the molecule generation model (S420), and a step of generating a final molecule corresponding to the target pocket information based on the reinforcement learning process by utilizing the molecule generation model (S430).

[0095]

[0096] According to one embodiment of the present disclosure, a computing device can input target pocket information into a molecular generation model (S410).

[0097] For example, target pocket information may include binding site information of a target protein, and may also include target information regarding what kind of molecule to generate as a final molecule based on the binding site information of the target protein. For example, the target protein may include an enzyme, a G-protein coupled receptor (GPCR), an ion channel, a protein containing a protein-protein interaction site, or a biological target protein associated with a specific disease. For example, the binding site information of a target protein may include three-dimensional spatial structure information of the target pocket, information on the arrangement and chemical properties of residues, charge distribution information, information on hydrophobic / hydrophilic interaction regions, and information on the location of hydrogen bond acceptors and donors. The molecule generation model can identify binding site characteristics for a target protein based on the target pocket information and set a target (directivity) for the final molecule.

[0098] In one embodiment, the molecular generation model may be a model that generates a final molecule in an atom-autoregressive manner based on target pocket information of the target protein. For example, the final molecule may be a molecule ultimately desired to be generated through the molecular generation model. Based on the atom-autoregressive manner of the molecular generation model, the final molecule may be generated as a molecule suitable for the target protein.

[0099]

[0100] According to one embodiment of the present disclosure, a computing device can perform a reinforcement learning process based on target pocket information by utilizing a molecular generation model (S420).

[0101] In one embodiment, the reinforcement learning process may utilize actions or states associated with partially generated molecules. For example, a partially generated molecule may be an intermediate result of a molecule generation model, consisting of atoms generated so far. Alternatively, a partially generated molecule may not be an intermediate output of a molecule generation model, but may be a configuration generated from another source. For example, a partially generated molecule may be a new molecule fragment generated by another model, an existing molecule fragment acquired from a database or the cloud, etc. In other words, the molecule generation model may perform the reinforcement learning process from zero using an atomic regression method, or may perform the reinforcement learning process starting from an intermediate output acquired from an external source.

[0102] In one embodiment, the state of the reinforcement learning process may include state information related to the partially generated molecule and the target pocket. For example, the state information related to the partially generated molecule may include the 3D structure and coordinates of the partially generated molecule, atomic arrangements, bond types, and chemical property information evaluated during the generation process to date (e.g., synthesizability, drug compatibility, binding affinity, etc.). For example, the state information related to the target pocket may include the 3D spatial structure of the target pocket, charge distribution, hydrophobic / hydrophilic properties, and spatial proximity information between the binding site and the partially generated molecule.

[0103] In one embodiment, the state information related to the partially generated molecule and the target pocket may include graph information in which the state information related to the partially generated molecule and the target pocket is jointly expressed. For example, the graph information may be a data structure that comprehensively represents the atoms of the partially generated molecule and the atoms of the target pocket and represents the relationship between them. For example, nodes of the graph may correspond to atoms, representing atoms within the partially generated molecule and atoms within the target pocket, respectively. For example, edges within the partially generated molecule in the graph may correspond to bonds, representing chemical bonds between atoms. For example, edges between the partially generated molecule and the target pocket in the graph may correspond to spatial proximity. Based on the graph information in which the state information related to the partially generated molecule and the target pocket is jointly expressed, a reinforcement learning agent of a molecule generation model may represent a state and determine the next action in a reinforcement learning process.

[0104] In one embodiment, the action of the reinforcement learning process may include an atom addition action or a stop action. For example, the atom addition action may be an action of adding a new atom to a partially generated molecule. The atom addition action may be performed based on the index information of a reference atom to which the new atom is to be bonded among the atoms of the partially generated molecule, the coordinate information of the reference atom, the type information of the new atom, and the bond information of the reference atom and the new atom. For example, the stop action may be an action of terminating the atom creation process, and if the molecule creation is possible (a success episode, a first type of failure episode, and a second type of failure episode described below), it may be an action of generating a completed final molecule based on the graph of the last state. The graph of the last state may include the final structural information of the partially generated molecule, and based on the stop action, the final molecule may be converted into a form with a completed 3D structure and chemical properties based on the graph.

[0105] In one embodiment, an episode of a reinforcement learning process may be obtained based on an operation of sampling an atom addition action and an operation of returning a stop action. For example, an episode of a reinforcement learning process may refer to a series of steps from the start to the end of a simulation in a molecule generation process. For example, the operation of sampling an atom addition action may be performed based on a softmax function and a Gaussian mixture distribution, and the operation of returning a stop action may be performed based on a sigmoid function.

[0106] In one embodiment, parameters associated with the operation of sampling an atomic addition behavior or the operation of returning a stop behavior may be set differently during the learning and inference processes of a molecular generation model. For example, parameters set differently during the learning and inference processes of a molecular generation model may include a threshold associated with a sigmoid function and a covariance value of a Gaussian mixture distribution.

[0107] Below, we describe the atomic addition and stopping actions that are set differently during the learning and inference processes, respectively.

[0108]

[0109]

[0110] For example, the operation of sampling the atom addition behavior during the learning process can be performed by sampling the coordinates, types, and bonding information of the new atom based on a softmax function and a Gaussian mixture distribution. More specifically, the operation of sampling the atom addition behavior can include the operation of determining the reference atom (f), the operation of determining the coordinates (p) of the new atom, the operation of determining the type (a) of the new atom, and the operation of determining the bonding type (b) between the reference atom and the new atom. For example, the operation of determining the reference atom (f) can be performed using a softmax function, and the molecule generation model can probabilistically select a specific reference atom by considering the priority of each candidate reference atom. This can enable learning the optimal molecular structure by detecting various reference atoms. For example, the operation of determining the coordinates (p) of the new atom can be performed by sampling the positions (coordinates) of the new atom using a Gaussian mixture distribution (GMM). The learning process can be designed to explore various coordinates by maintaining the covariance value (Σ). For example, the operation of determining the type (a) of a new atom (e.g., C, N, O, etc.) and the operation of determining the bonding type (b) of the reference atom and the new atom can be performed using a softmax function.

[0111] For example, the behavior that returns the stopping behavior during the learning process can be determined probabilistically based on the sigmoid function (max(yfocal)) of the reference atom, and the threshold can be designed to be randomly set rather than fixed during learning so that various termination conditions can be learned.

[0112] For example, the act of sampling additional atomic behaviors during the inference process, unlike the learning process, can deterministically select the atom with the highest probability based on the sigmoid function (max(yfocal)) of the reference atom. Furthermore, for Gaussian mixture distributions, the covariance value (Σ) can be set to 0 and coordinates can be determined using only the mean (μ) of the distribution, which ensures stable reproduction of the learned results.

[0113] For example, during the inference process, the action of returning a stop action can be performed when the probability calculated based on the sigmoid function (max(yfocal)) exceeds a fixed threshold (e.g., 0.5). This allows the inference process to terminate reliably and deterministically using a fixed threshold.

[0114] In one embodiment, an episode of a reinforcement learning process may include at least one of a success episode in which a stop action is returned and the final molecule corresponds to a valid molecule, a first type of failure episode in which a stop action is returned but the final molecule corresponds to an invalid molecule, a second type of failure episode in which an atom addition action is repeated more than a predetermined threshold (e.g., 50), or a third type of failure episode in which an intermediate molecule is detected to have a nonsensical structure.

[0115] In one embodiment, the reward of the reinforcement learning process can be determined based on a success reward for a successful episode and a failure penalty reward for all types of failure episodes. For example, the failure penalty reward can be calculated as R = FP, and the success reward can be calculated as R = c1P1+ … + c k P k can be calculated as c i is a fixed coefficient, and P imay be properties of interest. More specifically, each coefficient can be defined as follows.

[0116]

[0117] For example, the success reward for a successful episode can be determined based on properties of interest and coefficients for each property. For example, properties of interest can include Vina scores (without conformer optimization), QED scores, and fractions of stereochemical properties (bond lengths, bond angles, ring planarity values, and distances between non-covalently bonded atom pairs) that fall within an appropriate range.

[0118] For example, the failure penalty reward for each type of failure episode can be determined based on the mean and standard deviation of the success rewards.

[0119]

[0120] According to one embodiment of the present disclosure, a computing device can generate a final molecule corresponding to target pocket information based on a reinforcement learning process by utilizing a molecular generation model (S430).

[0121] For example, if an episode of the reinforcement learning process is determined to be a Type 3 failure episode, the final molecule may not be generated. This is because, if the intermediate molecule is detected as having a nonsensical structure, the molecule may be determined to be chemically inappropriate or unlikely to bind to the target pocket. On the other hand, if an episode of the reinforcement learning process is determined to be a success episode, a Type 1 failure episode, or a Type 2 failure episode, the final molecule corresponding to the target pocket information can be generated based on the final state graph.

[0122]

[0123] FIG. 5 is a schematic diagram illustrating a reinforcement learning process according to one embodiment of the present disclosure.

[0124] Referring to FIG. 5, target pocket information (510), a reinforcement learning process (520), and a final molecule (530) are illustrated. For example, the target pocket information (510) may be the initial input data of the reinforcement learning process (520), and the final molecule (530) may be the final output data of the reinforcement learning process (520).

[0125] In one embodiment, the reinforcement learning process (520) may be performed by utilizing a molecule generation model (522) based on target pocket information (510). More specifically, the reinforcement learning process (520) may be performed by repeatedly updating actions or states associated with partially generated molecules (524) through the molecule generation model (522).

[0126] In one embodiment, the reinforcement learning process (520) may gradually expand a partially generated molecule (524) based on a molecule generation model (522). The partially generated molecule (524) may converge to a final molecule (530) through atomic addition and stop actions of the reinforcement learning process (520). For example, depending on the episode of the reinforcement learning process (520), whether the final molecule (530) is generated may be determined. For example, if the episode result of the reinforcement learning process (520) is determined to be a success episode, a first type of failure episode, or a second type of failure episode, the final molecule (530) corresponding to the target pocket information (510) may be generated based on the graph of the final state. For example, if the episode result of the reinforcement learning process (520) is determined to be a third type of failure episode, the final molecule (530) may not be generated.

[0127] In one embodiment, a partially generated molecule (524) may be represented as a state along with target pocket information (510), which may be used as input to a molecule generation model (522). For example, the reinforcement learning process (520) may sample atomic addition actions using a softmax function and a Gaussian mixture distribution, and may return a stopping action based on a sigmoid function.

[0128]

[0129] FIG. 6 is a schematic diagram illustrating the actions and states of a reinforcement learning process according to one embodiment of the present disclosure.

[0130] Referring to FIG. 6, target pocket information (610), reinforcement learning process (620), and final molecule (670) are illustrated.

[0131] In the reinforcement learning process (620), the molecular generation model (630) can be understood as an "agent" that determines actions based on state information and rewards. In the present disclosure, state information may include current state information (662) and next state information (664). The current state information (662) and next state information (664) may be distinguished based on the time or order in which the state information was acquired, and may be named current state information (662) and next state information (664), respectively, based on the chronological relationship.

[0132] In the reinforcement learning process (620), the environment (640) can produce current state information (662) that can serve as a basis for the molecular generation model (630) to determine an action. The reinforcement learning process (620) can obtain current state information (662) including at least one state variable from the environment (640) and then input the current state information (662) into the molecular generation model (630).

[0133] The reinforcement learning process (620) can input current state information (662) into the molecule generation model (630), and then output an action (650) based on the molecule generation model (630). The molecule generation model (630) can output or sample a specific action among a plurality of selectable actions (652, 654) based on state information (662) acquired from the environment (640) at any point in time t. The action of the reinforcement learning process (620) can include an atom addition action (652) or a stop action (654). When the atom addition action (652) is output, an operation of adding a new atom to a molecule partially generated from the molecule generation model (630) can be performed, and when the stop action (654) is output, the atom generation process can be terminated.

[0134] The reinforcement learning process (620) can obtain the next state information (664) and reward updated from the environment (640) as an input result of the action. The reward of the reinforcement learning process (620) can be determined on a step-by-step basis or an episode-by-episode basis. For example, the reward of the reinforcement learning process (620) can be determined on a step-by-step basis by evaluating the partially generated molecule (524) of the current step generated by the current action, can be determined on an episode-by-episode basis by utilizing the expected value for episodes predicted after the current action, or can be determined on an episode-by-episode basis by utilizing actual episodes sampled after the current action.

[0135] An episode of the reinforcement learning process (620) may include at least one of a success episode, a first type of failure episode, a second type of failure episode, or a third type of failure episode.

[0136] The reinforcement learning process (620) can obtain one episode by repeatedly performing a plurality of steps identical or similar to the embodiment of the above-described reinforcement learning model-based molecule generation method from the initial state (t=0) to the final state (t=T). For example, the final state may be derived when a preset termination condition is satisfied or may be derived when a preset number of steps have been performed. For example, the preset termination condition may be a condition in which a stop action (654) is produced, or a condition in which a third type of failure episode is obtained among the episodes of the reinforcement learning process (620). For example, the number of preset steps may be set to any natural number, and may consist of, for example, 50 steps.

[0137] The reinforcement learning process (620) can generate a final molecule (670) based on the partially generated molecules generated so far, if the episode of the reinforcement learning process (620) is determined to be a success episode, a first type of failure episode, or a second type of failure episode after the stop action (654).

[0138]

[0139] Figures 7a to 7c are drawings illustrating a molecule production process according to an example of the present disclosure.

[0140] Referring to FIGS. 7A to 7C, a target protein (710), a partially generated molecule (730), and a new partially generated molecule (750) are illustrated.

[0141] Referring to FIG. 7A, target pocket information may include binding site information of a target protein (710), and may also include target information regarding which molecule to generate as a final molecule based on the binding site information of the target protein (710). In step S410 described above with reference to FIG. 4, the computing device may input target pocket information into a molecule generation model.

[0142] Thereafter, in step S420 described above with reference to FIG. 4, a reinforcement learning process can be performed based on target pocket information by utilizing a molecular generation model.

[0143] Referring to FIGS. 7A and 7B , for example, when an atom addition action is generated for a target protein (710), the molecule generation model can generate a partially generated molecule (730) based on target pocket information. The atom addition action can be performed based on index information of a reference atom (712), coordinate information of the reference atom (712), type information of a new atom (732), and binding information of the reference atom (712) and the new atom (732).

[0144] For example, after a partially generated molecule (730) is generated, the reinforcement learning process can determine whether a preset termination condition (e.g., a condition in which a stop action is generated or a condition in which a third type of failure episode is acquired) is met based on the state of the partially generated molecule (730). If the partially generated molecule (730) does not meet the preset termination condition, an atomic addition action can be generated for the partially generated molecule (730), and the molecule generation model can generate a new partially generated molecule based on the target pocket information of the partially generated molecule (730).

[0145] Referring to FIG. 7C, a reference atom (732) and a new atom (752) of a newly partially generated molecule (750) are illustrated. After the newly partially generated molecule (750) is generated, the reinforcement learning process can determine whether a preset termination condition (e.g., a condition in which a stop action is generated or a condition in which a third type of failure episode is obtained) is met based on the state of the newly partially generated molecule (750). If, based on the newly partially generated molecule (750), the third type of failure episode is not obtained and a stop action is generated, a final molecule can be generated based on the state of the newly partially generated molecule (750).

[0146]

[0147] According to one embodiment of the present disclosure, a method for generating molecules based on a reinforcement learning model, a computing device for performing the same, and a computer program stored in a computer-readable storage medium for performing operations therefor are disclosed.

[0148] The molecule generation method based on a reinforcement learning model according to the present disclosure can achieve two main goals: minimizing stereochemical problems and improving properties such as drug similarity and binding affinity by improving existing pocket conditional molecule generation models (e.g., Pocket2Mol model).

[0149] In experiments to implement the method according to the present disclosure, metrics such as the Vina Score (which evaluates the binding affinity of a molecule and is used without structural optimization), the Quantitative Estimate of Drug-likeness (QED) score, and the fractions of stereochemical properties (bond lengths, bond angles, ring flatness values, and distances between non-covalently bonded atom pairs) that fall within an appropriate range can be used as rewards for reinforcement learning. During model development, a randomly sampled subset from the training set, which has the same size as the test set, can be used for validation.

[0150] As experimental results for implementing the method according to the present disclosure, the characteristics of binding affinity, drug-likeness, synthetic accessibility, and the presence of specific stereochemical problems can be evaluated.

[0151] The following are the results of comparing the method according to the present disclosure with existing methods, including existing models related to (1) lattice-based (Ragoza et al. (2022)), (2) atomic autoregressive (Luo et al. (2021), Liu et al. (2022), Peng et al. (2022)), and (3) diffusion-based (Guan et al. (2022)) methods. The resulting figures are averaged for the molecules generated for each target protein in the test set, and are about the mean and median of these averages, and are explained with reference to and below.

[0152]

[0153] Table 1

[0154]

[0155]

[0156] Table 2

[0157]

[0158]

[0159] Based on the experimental results, it can be confirmed that the method according to the present disclosure successfully improves the binding affinity and druggability of existing pocket-conditional molecular production models (e.g., Pocket2Mol model) while reducing stereochemical issues, particularly distorted ring issues. Furthermore, synthesizability was also improved concomitantly, although not as part of the compensation. Binding affinity was comparable to that of the diffusion-based reference model (Guan et al. (2022)), and while some types of stereochemical issues still occurred more frequently than in some reference models, all other indicators were significantly improved compared to the best reference model.

[0160] Based on experimental results, the method described herein significantly improves binding affinity, druggability, and synthetic accessibility while reducing stereochemical issues. The method described herein not only improves upon existing models but also achieves state-of-the-art results for most metrics. These results demonstrate that reinforcement learning is effective in resolving existing issues in pocket-conditional molecule generation models and improving desired properties, and can play a key role in solving the problem of molecule generation.

[0161] The method according to the present disclosure successfully improves the existing pocket condition molecule production model for all considered indicators, and based on this, it can be confirmed that chemical / structural properties can be optimized.

[0162]

[0163] According to one embodiment of the present disclosure, a computer-readable medium storing a data structure is disclosed. A data structure may refer to the organization, management, and storage of data that enables efficient access and modification of the data. A data structure may refer to the organization of data to solve a specific problem (e.g., data retrieval, data storage, or data modification in the shortest possible time). A data structure may also be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements may include a connection relationship between user-defined data elements. A physical relationship between data elements may include an actual relationship between data elements physically stored in a computer-readable storage medium (e.g., a persistent storage device). A data structure may specifically include a set of data, relationships between data, and functions or commands applicable to the data. An effectively designed data structure enables a computing device to perform operations while minimizing the use of its resources. Specifically, a computing device can improve the efficiency of operations, reading, insertion, deletion, comparison, exchange, and searching through an effectively designed data structure.

[0164] Data structures can be categorized as linear or nonlinear, depending on their form. A linear data structure can be a structure in which only one data item is linked to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a series of data sets with an internal order. Lists can also include linked lists. A linked list is a data structure in which data is linked in a single line, each item having a pointer. In a linked list, a pointer can contain information about the next or previous item. Linked lists can be expressed as singly linked lists, doubly linked lists, or circular linked lists, depending on their form. A stack can be a data listing structure with limited data access. A stack can be a linear data structure in which data operations (e.g., insertion or deletion) can only be performed at one end of the data structure. Data stored in a stack can be a Last-in-First-out (LIFO) data structure. A queue is a data structure with limited access to data. Unlike a stack, it can be a first-in, first-out (FIFO) data structure, with later data being retrieved later. A deck can be a data structure that can process data at both ends.

[0165] A nonlinear data structure can be a structure in which multiple pieces of data are connected behind a single piece of data. Nonlinear data structures can include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structures can include tree data structures. A tree data structure can be a data structure in which there is only one path connecting two different vertices among multiple vertices included in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

[0166] Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they are collectively referred to as neural networks. The data structure may include a neural network. And the data structure including the neural network may be stored on a computer-readable medium. The data structure including the neural network may also include preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. The data structure including the neural network may include any of the components disclosed above. That is, the data structure including the neural network may be configured to include all or any combination of preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. In addition to the aforementioned configurations, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated in the computational process of the neural network, and is not limited to the aforementioned. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

[0167] The data structure may include data input to a neural network. The data structure including the data input to the neural network may be stored on a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and / or input data input to the neural network after training has been completed. The data input to the neural network may include data that has undergone preprocessing and / or data that is the target of preprocessing. Preprocessing may include a data processing process for inputting data to the neural network. Accordingly, the data structure may include data that is the target of preprocessing and data generated by the preprocessing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0168] The data structure may include weights of the neural network. (In this specification, the terms "weight" and "parameter" may be used interchangeably.) The data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weights may be variable and may be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine a data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set for links corresponding to each input node. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0169] By way of example and not limitation, the weights may include weights that vary during the neural network training process and / or weights that have completed neural network training. The weights that vary during the neural network training process may include weights at the start of the training cycle and / or weights that vary during the training cycle. The weights that have completed neural network training may include weights that have completed the training cycle. Accordingly, a data structure including the weights of a neural network may include a data structure including weights that vary during the neural network training process and / or weights that have completed neural network training. Therefore, the above-described weights and / or combinations of each weight are included in the data structure including the weights of a neural network. The above-described data structures are merely examples and the present disclosure is not limited thereto.

[0170] A data structure including neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed and used. A computing device can serialize the data structure to transmit and receive data over a network. The serialized data structure including neural network weights can be reconstructed on the same or different computing devices through deserialization. The data structure including neural network weights is not limited to serialization. Furthermore, the data structure including neural network weights can include a data structure that increases computational efficiency while minimizing the use of computing device resources (e.g., a B-Tree, a Trie, an m-way search tree, an AVL tree, a Red-Black Tree in nonlinear data structures). The foregoing is merely an example, and the present disclosure is not limited thereto.

[0171] The data structure may include hyperparameters of a neural network. Furthermore, the data structure including the hyperparameters of the neural network may be stored on a computer-readable medium. The hyperparameters may be variables that can be varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle repetitions, weight initialization (e.g., setting a range of weight values ​​to be subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is merely an example, and the present disclosure is not limited thereto.

[0172]

[0173] FIG. 8 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0174] Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and / or other program modules that may be executed on one or more computers and / or as a combination of hardware and software.

[0175] Generally, program modules include routines, programs, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be implemented with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

[0176] The described embodiments of the present disclosure can also be practiced in distributed computing environments, where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0177] Computers typically include a variety of computer-readable media. Computer-readable media can be any media that can be accessed by a computer, and includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

[0178] Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

[0179] An exemplary environment (1100) implementing various aspects of the present disclosure is illustrated, including a computer (1102) comprising a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

[0180] The system bus (1108) may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0181] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA) - which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

[0182] These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of the computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art will appreciate that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

[0183] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and / or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0184] A user may enter commands and information into the computer (1102) via one or more wired / wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is connected to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

[0185] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0186] The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), although for simplicity, only the memory storage device (1150) is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) (1152) and / or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

[0187] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means of establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules or portions thereof described for the computer (1102) may be stored in a remote memory / storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

[0188] The computer (1102) operates to communicate with any wireless device or object that is arranged and operates via wireless communication, such as a printer, a scanner, a desktop and / or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

[0189] Wi-Fi (Wireless Fidelity) enables connections to the Internet and other devices without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data anywhere within the coverage area of ​​a base station, both indoors and outdoors, similar to cell phones. Wi-Fi networks use wireless technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual-band).

[0190] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0191] Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0192] The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0193] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

[0194] The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

[0195]

[0196] As described above, the relevant contents have been described in the best form for carrying out the invention.

Claims

1. A method for generating molecules based on a reinforcement learning model, performed by a computing device, A step of inputting target pocket information into a molecular generation model; A step of performing a reinforcement learning process based on the target pocket information by utilizing the above molecular generation model; and A step of generating a final molecule corresponding to the target pocket information based on the reinforcement learning process by utilizing the above molecule generation model. Including, The above reinforcement learning process utilizes actions or states associated with partially generated molecules. method.

2. In paragraph 1, The above molecular generation model is a model that generates the final molecule in an atom-autoregressive manner based on the target pocket information of the target protein. method.

3. In paragraph 1, The above state of the above reinforcement learning process is, including the partially generated molecule and state information related to the target pocket, method.

4. In paragraph 3, The state information related to the partially generated molecule and the target pocket includes graph information in which the state information related to the partially generated molecule and the target pocket are jointly expressed, The nodes of the above graph correspond to atoms, In the above graph, the edge inside the partially generated molecule corresponds to a bond, In the above graph, the edge between the partially generated molecule and the target pocket corresponds to spatial proximity. method.

5. In paragraph 1, The above actions of the above reinforcement learning process are, Including an atomic add action, or a stop action, method.

6. In paragraph 5, The above atomic additional action is, Index information of the reference atom to which a new atom will be bonded among the atoms of the above partially generated molecule; Coordinate information of the above reference atom; Type information of the new atom; and Bond information between the above reference atom and the above new atom is performed based on, method.

7. In paragraph 5, An episode of the above reinforcement learning process is, An operation of sampling the above atomic addition behavior based on a softmax function and a Gaussian mixture distribution; and The above stop action is returned based on the sigmoid function. Obtained based on, method.

8. In paragraph 7, The parameters associated with the action of sampling the above atomic addition action or the action from which the above stop action is returned are: The learning process and inference process of the above molecular generation model are set differently. method.

9. In paragraph 8, The parameters that are set differently in the learning and inference processes of the above molecular generation model are: a threshold value associated with the sigmoid function, and a covariance value of the Gaussian mixture distribution, method.

10. In paragraph 7, The episode of the above reinforcement learning process is, A successful episode in which the above stopping action is returned and the final molecule corresponds to a valid molecule; A first type of failure episode in which the above stop action is returned but the final molecule corresponds to an invalid molecule; A second type of failure episode in which the above atomic addition behavior is repeated beyond a predetermined threshold; or A third type of failure episode corresponding to the case where the intermediate molecule is detected to have a nonsensical structure. Containing at least one of, method.

11. In paragraph 10, The reward of the above reinforcement learning process is, Success reward for the above successful episode; and Failure penalty compensation for all types of failure episodes is determined based on, method.

12. In paragraph 11, The above success reward is, It is determined based on the properties of interest and the coefficients for each property, The above failure penalty compensation is, Determined based on the mean and standard deviation of success rewards, method.

13. A computer program stored in a computer-readable storage medium that causes a computing device to perform operations for generating molecules based on a reinforcement learning model, wherein the operations are: The act of inputting target pocket information into a molecular generation model; An operation of performing a reinforcement learning process based on the target pocket information by utilizing the above molecular generation model; and An operation of generating a final molecule corresponding to the target pocket information based on the reinforcement learning process by utilizing the above molecule generation model. Including, The above reinforcement learning process utilizes actions or states associated with partially generated molecules. A computer program stored on a computer-readable storage medium.

14. In paragraph 13, The above molecular generation model is a model that generates the final molecule in an atom-autoregressive manner based on the target pocket information of the target protein. A computer program stored on a computer-readable storage medium.

15. In paragraph 13, The above state of the above reinforcement learning process is, including the partially generated molecule and state information related to the target pocket, A computer program stored on a computer-readable storage medium.

16. In paragraph 13, The above actions of the above reinforcement learning process are, Includes an atomic add action, or a Stop action, The above atomic additional action is, Information about a reference atom to which a new atom will be bonded among the atoms of the partially generated molecule, and information about the new atom, A computer program stored on a computer-readable storage medium.

17. A computing device for generating molecules based on a reinforcement learning model, one or more processors; and memory; Including, One or more of the above processors: Enter target pocket information into the molecular generation model, By utilizing the above molecular generation model, a reinforcement learning process is performed based on the target pocket information, Using the above molecular generation model, the final molecule corresponding to the target pocket information is generated based on the reinforcement learning process. It is configured to do so, The above reinforcement learning process utilizes actions or states associated with partially generated molecules. Computing device.

18. In paragraph 17, The above molecular generation model is a model that generates the final molecule in an atom-autoregressive manner based on the target pocket information of the target protein. Computing device.

19. In paragraph 17, The above state of the above reinforcement learning process is, including the partially generated molecule and state information related to the target pocket, Computing device.

20. In paragraph 17, The above actions of the above reinforcement learning process are, Includes an atomic add action, or a Stop action, The above atomic additional action is, Information about a reference atom to which a new atom will be bonded among the atoms of the partially generated molecule, and information about the new atom, Computing device.

Images