3D Spatial Molecular Generation Method and Device Based on Multi-Task Pre-Trained Inverse Reinforcement Learning

Abstract

The present invention discloses a three-dimensional space molecular generation method and device based on multi-task pre-trained inverse reinforcement learning. The minimum spanning tree algorithm is used to construct a real molecular generation sequence according to chemical molecules; a policy network is used to generate an atom addition action based on the current molecular intermediate to construct a predicted molecular generation sequence; the real molecular generation sequence and the predicted molecular generation sequence generated based on the policy network are simultaneously input into a pre-trained Transformer model, and a reward function is constructed based on the regression prediction of the Transformer model. The policy network is trained through the reward function to improve the prediction robustness of the policy network. In addition, when constructing the reward function, the reward value of the real sample is maximized and the reward value of the predicted sample is minimized. During optimization, the agent takes actions similar to those of the expert behavior, alleviating the problem of sparse rewards and improving the parameter optimization efficiency at the same time.

Description

Technical Field

[0001] The present invention belongs to the field of chemical drug molecule generation, and particularly relates to a three-dimensional space molecule generation method and device based on multi-task pre-trained inverse reinforcement learning. Background Art

[0002] Compared with the one-dimensional SMILEs string representation and two-dimensional space graph representation of drug molecules, the three-dimensional space representation provides rich structural information. At present, more and more studies use deep learning methods to mine the structural features of three-dimensional space molecules to generate molecules with specific properties and accelerate drug research and development. For example, a graph model drug generation method, device and medium disclosed in patent document CN110459275A, and another example is a drug molecule generation method with controllable path based on reinforcement learning disclosed in patent document CN112331277A. The method based on reinforcement learning is flexible in generation and can effectively explore the chemical molecule space outside the training data. However, the performance of the reinforcement learning agent depends on a suitable reward function.

[0003] Existing reward function design methods rely on molecular property predictors or molecular dynamics simulations. The property predictors are usually trained on a complete molecular dataset with manual annotations and have poor generalization performance on the intermediate product representations of molecule generation. Although molecular dynamics simulations can provide intermediate rewards, the physical simulation runs slowly, which limits the sampling efficiency of the reinforcement learning algorithm. There are also some methods that only detect whether the final generated product meets the generation requirements and there is no reward value in the intermediate process. The algorithm may be difficult to discover valuable information from sparse rewards and explore too much irrelevant chemical space, resulting in low convergence efficiency.

[0004] The commonly used adversarial inverse reinforcement learning (AIRL) in inverse reinforcement learning usually uses a binary classification GAN as the reward function, which is prone to overfitting to the local noise differences between expert behaviors and apprentice behaviors. Summary of the Invention

[0005] In view of the above, the purpose of the present invention is to provide a three-dimensional space molecule generation and device based on multi-task pre-trained inverse reinforcement learning to improve the generation efficiency and accuracy of three-dimensional space molecules.

[0006] To achieve the above object of the invention, an embodiment of the present invention provides a three-dimensional space molecule generation method based on multi-task pre-trained inverse reinforcement learning, including the following steps:

[0007] Using the minimum spanning tree algorithm to construct a real molecule generation sequence according to chemical molecules;

[0008] Using a policy network to generate an atom addition action according to the current molecular intermediate product to construct a predicted molecule generation sequence;

[0009] Sample molecular intermediate-atom addition action pairs from the real molecular sequence and the predicted molecular generation sequence respectively as real samples and predicted samples, and perform vector encoding on the real samples and predicted samples to obtain vector representations;

[0010] After adding task prompts to the vector representations of the real samples and the predicted samples respectively, use them as pre-training data and input them into the pre-trained Transformer model to obtain the decoded data corresponding to the real samples and the predicted samples respectively;

[0011] Calculate the regression error based on the real samples, the predicted samples and the decoded data corresponding to them, and construct a reward function based on the regression error;

[0012] Optimize the Transformer model by maximizing the reward value of the real samples and minimizing the reward value of the predicted samples according to the expected value of the reward function, and use the reward function to optimize the parameters of the policy network. After the optimization is completed, the policy network with optimized parameters is used as the molecular generation model;

[0013] Generate three-dimensional space molecules using the molecular generation model.

[0014] Preferably, use the minimum spanning tree algorithm to construct the real molecular generation sequence based on chemical molecules, including:

[0015] Randomly select a carbon atom or a nitrogen atom from the chemical molecule as the initial atom, obtain the atom traversal order of the chemical molecule graph according to the minimum spanning tree algorithm, and construct a real molecular generation sequence composed of molecular intermediate-atom addition action pairs sorted by time step. The molecular intermediate-atom addition action pair is represented as (s,a), where s is the current molecular intermediate, and a is the atom addition action that should be taken at the current time step, in the form of P is an atom in the current molecular intermediate s, Q is the atom to be added to s, d is the bond length between P and Q, and the bond type is determined by the bond length. is the angle between the newly added edge and the existing edge in the molecule when the number of atoms in s is greater than or equal to 2. is the angle between the plane PQN1 formed by the newly added edge and the plane PN1N2 in s when the number of atoms in s is greater than or equal to 3, where N1 and N2 are the nearest neighbor atom and the second nearest neighbor atom of atom P respectively.

[0016] Preferably, the policy network is used to generate atom addition actions based on the current molecular intermediate graph to construct a predicted molecular generation sequence, including:

[0017] The policy network contains six sub-networks, and these six sub-networks respectively output that make up the atom addition action P is an atom in the current molecular intermediate product s, Q is the atom to be added to s, d is the bond length between P and Q, and the bond type is determined by the bond length. is the angle between the newly added edge and the existing edges in the molecule when the number of atoms in s is greater than or equal to 2. is the angle between the plane PQN1 formed by the newly added edge and the plane PN1N2 in s when the number of atoms in s is greater than or equal to 3, where N1 and N2 are the nearest neighbor atom and the second nearest neighbor atom of atom P respectively, and T represents whether the atom addition action terminates.

[0018] The current molecular intermediate product s and the atom addition action a form the molecular intermediate product - atom addition action (s, a) at the current time step. Atom Q will be added to the current molecular intermediate product s according to the atom addition action a to obtain the molecular intermediate product as the input for the next step. The molecular intermediate product - atom addition actions at each time step are sorted according to the time step to form a predicted molecular generation sequence.

[0019] Preferably, the vector encoding of the real sample and the predicted sample to obtain vector representations includes:

[0020] The real sample and the predicted sample adopt the same encoding method. When the real sample and the predicted sample are represented as the molecular intermediate product - atom addition action (s, a), first, each atomic node in the molecular intermediate product s constructs the vector representation of the atomic node according to the vector corresponding to when the atom is added to the molecular intermediate product, to obtain the initial vector representation of the molecular intermediate product, and then use a pluggable graph encoding model to encode based on the initial vector representation to obtain the vector representation of the molecular intermediate product.

[0021] For the atom addition action a, the in a is converted into a vector representation through spherical Fourier - Bessel basis functions. P and Q in a are one - hot encoded, where the length of the one - hot vector of P is the number of atoms in s, and the length of the one - hot vector of Q is the number of allowed atom types. The vectors of

[0022] are concatenated as the vector representation of a.

[0023] Preferably, according to the following formula, the in a is converted into a vector representation through spherical Fourier - Bessel basis functions

[0024]

[0025] Among them, c is the truncation distance, and z ln is the n-th root of the l-th order Bessel function, is the spherical harmonic function of degree m and order l, where l ∈ {0, …, N SHBF}, n ∈ {1, …, N SRBF}, m ∈ {-l, …, l}, and N SHBF , N SRBF are respectively the highest orders of the preset spherical harmonic function and spherical Bessel function.

[0026] Preferably, when using a pluggable graph encoding model to perform vector encoding on molecular intermediate products, during the optimization process, the parameters of the graph encoding model are optimized by maximizing the reward value of the real sample and minimizing the reward value of the predicted sample according to the expected value of the reward function.

[0027] Preferably, the constructed reward function is expressed as:

[0028]

[0029] δ = ||Dec(s k , a k ) - [s k , a k ||2

[0030] The objective function for optimizing the Transformer model parameters is:

[0031]

[0032] Among them, represents the expected value of the reward value of the test sample extracted from the predicted molecular generation sequence generated by the policy network π θ , represents the expected value of the reward value of the real sample extracted from the real molecular generation sequence D E , w represents the optimized parameter, r w (s, a) represents the reward value of the molecular intermediate product - adding atom action pair, δ represents the prediction error of the Transformer model, [s k , a k represents the predicted label of the k-th order sliding window, specifically the vector representation of the molecular intermediate product - adding atom action pair, Dec(s k , a k ) represents the decoded data output by the Transformer model, and ||·||2 represents the L2 loss function.

[0033] Preferably, each sub-network of the policy network adopts an MLP, the graph encoding model adopts schnet, and during optimization, a pair of molecular intermediate-adding atom actions with a k-order sliding window is used as the input to perform the decoding data prediction of the Transformer model.

[0034] Preferably, when applied to other downstream small molecule generation tasks, the small molecule pair molecule generation model is fine-tuned. During fine-tuning, the parameters of the Transformer model and the graph encoding model are fixed, and only the constructed reward function is used to re-optimize the pair molecule generation model until convergence. The re-optimized pair molecule generation model is used for the generation of other small molecules.

[0035] To achieve the above-mentioned invention purpose, the embodiment also provides a three-dimensional space molecule generation device based on multi-task pre-training inverse reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned three-dimensional space molecule generation method based on multi-task pre-training inverse reinforcement learning is realized.

[0036] Compared with the prior art, the beneficial effects of the present invention at least include:

[0037] The real molecule generation sequence and the predicted molecule generation sequence generated based on the policy network are simultaneously input into the pre-trained Transformer model. The reward function is constructed based on the regression prediction of the Transformer model, and the policy network is trained through the reward function to improve the prediction robustness of the policy network. In addition, when constructing the reward function, the reward value of the real sample is maximized and the reward value of the predicted sample is minimized. During optimization, the agent takes actions similar to the expert behavior, alleviating the problem of sparse rewards and improving the parameter optimization efficiency at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 is the flowchart of the three-dimensional space molecule generation method based on multi-task pre-training inverse reinforcement learning provided by the embodiment;

[0040] Figure 2 is the overall framework of the three-dimensional space molecule generation method based on multi-task pre-training inverse reinforcement learning provided by the embodiment;

[0041] Figure 3It is the flowchart of the molecular generation sequence construction provided by the embodiment. Detailed implementation manners

[0042] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific implementation manners described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0043] Figure 1 It is the flowchart of the three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning provided by the embodiment. Figure 2 It is the overall framework of the three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning provided by the embodiment. As Figure 1 and Figure 2 shown, the three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning provided by the embodiment includes the following steps:

[0044] Step 1, use the minimum spanning tree algorithm to construct a real molecular generation sequence based on chemical molecules.

[0045] In the embodiment, based on chemical molecules, a real molecular generation sequence is constructed by the minimum spanning tree algorithm, and the real molecular generation sequence is used to guide the construction of the reward function of the policy network.

[0046] Specifically, a carbon atom or a nitrogen atom is randomly selected from the chemical molecules as the initial atom. By randomly selecting the initial atom multiple times, multiple real molecular generation sequences are constructed for each chemical molecule. Starting from the initial atom, according to the minimum spanning tree algorithm, the atom traversal order of the chemical molecule graph is obtained, and a real molecular generation sequence composed of molecule intermediate-atom addition action pairs sorted by time steps is constructed. The molecule intermediate-atom addition action pair is represented as (s,a), which describes the position information of the newly added atom in the three-dimensional space. Among them, s is the current molecule intermediate, a is the atom addition action that should be taken at the current time step, and the form is P is an atom in the current molecule intermediate s, Q is the atom to be added to s, d is the bond length between P and Q, and the bond type is determined by the bond length. is the included angle between the newly added edge and the existing edge in the molecule when the number of atoms in s is greater than or equal to 2. is the included angle between the plane PQN1 formed by the newly added edge and the plane PN1N2 in s when the number of atoms in s is greater than or equal to 3, where N1 and N2 are the nearest neighbor atom and the next nearest neighbor atom of atom P respectively, as Figure 3 shown.

[0047] Step 2, use the policy network to generate an atom addition action based on the current molecule intermediate to construct a predicted molecular generation sequence.

[0048] In the embodiment, the policy network includes six sub-networks, respectively denoted as and The sub-network can adopt a single-hidden-layer MLP, and respectively output according to the current molecular intermediate product s of the input T represents whether the atom addition action is terminated (Terminal). Specifically, the policy network first judges whether it is terminated and outputs the predicted molecule. If it is not terminated, the atom addition action continues to be executed. Among them, the sub-network scores each atom in the current molecular intermediate product s, and randomly selects one of the nodes exceeding the threshold as the output. constitutes the atom addition action P is an atom in the current molecular intermediate product s, Q is the atom to be added to s, d is the bond length between P and Q, and the bond type is determined by the bond length. is the included angle between the newly added edge and the existing edge in the molecule when the number of atoms in s is greater than or equal to 2. is the included angle between the plane PQN1 formed by the newly added edge and the plane PN1N2 in s when the number of atoms in s is greater than or equal to 3, where N1 and N2 are the nearest neighbor atom and the next nearest neighbor atom of atom P respectively.

[0049] The current molecular intermediate product s and the atom addition action a form the molecular intermediate product-atom addition action (s, a) at the current time step. Atom Q will be added to the current molecular intermediate product s according to the atom addition action a to obtain the molecular intermediate product as the input for the next step. The molecular intermediate product-atom addition action at each time step is sorted according to the time step to form a predicted molecule generation sequence.

[0050] Step 3: Sample the molecular intermediate product-atom addition action pairs from the real molecule sequence and the predicted molecule generation sequence as real samples and predicted samples respectively, and perform vector encoding on the real samples and predicted samples to obtain vector representations.

[0051] In the embodiment, the real samples and the predicted samples adopt the same encoding method. When the real samples and the predicted samples are represented as the molecular intermediate product-atom addition action (s, a), first, each atom node in the molecular intermediate product s constructs the vector representation of the atom node according to the corresponding vector when the atom is added to the molecular intermediate product, to obtain the initial vector representation of the molecular intermediate product, and then use the pluggable graph encoding model to encode based on the initial vector representation to obtain the vector representation of the molecular intermediate product. Among them, the graph encoding model can adopt some three-dimensional molecular graph encoding models such as schnet and spherenet, and general representation models such as MLP and VAE.

[0052] For adding atomic action a, in a It is converted into a vector representation through spherical Fourier Bessel basis functions. P and Q in a adopt one-hot encoding, where the length of the one-hot vector of P is the number of atoms in s, and the length of the one-hot vector of Q is the number of allowed atomic types. The vector concatenation of... is used as the vector representation of a;

[0053] Specifically, according to the following formula, in a It is converted into a vector representation through spherical Fourier Bessel basis functions

[0054]

[0055] where c is the cut-off distance, z ln is the nth root of the lth order Bessel function, is the spherical harmonic function of the lth order and mth degree, l ∈ {0, …, N SHBF}, n ∈ {1, …, N SRBF}, m ∈ {-l, …, l}, N SHBF , N SRBF are respectively the highest orders of the pre-set spherical harmonic function and spherical Bessel function.

[0056] The vector representation of the molecular intermediate product s and the vector representation of the added atomic action a are concatenated as the vector representation of the molecular intermediate product - added atomic action (s, a).

[0057] Step 4, After adding the task prompt to the vector representations of the real sample and the predicted sample respectively, they are used as pre-training data and input into the Transformer model to obtain the decoded data corresponding to the real sample and the predicted sample respectively.

[0058] In the embodiment, multi-task datasets are used for joint training, and different tasks are distinguished by task prompts to improve the generalization ability of the Transformer model. Specifically, the task prompt is concatenated with the vector representation of the state-action pair, and the concatenated vector is used as the input to the pre-trained Transformer model.

[0059] Step 5, Calculate the regression error based on the real sample, the predicted sample and their corresponding decoded data, and construct a reward function based on the regression error.

[0060] In the embodiment, the constructed reward function is expressed as:

[0061]

[0062] δ = ||Dec(s k , a k ) - [sk , a k || 2

[0063] Among them, w represents the optimized policy network parameters, r w (s, a) represents the reward value for the molecular intermediate-atom addition action pair, δ represents the prediction error of the Transformer model, [s k , a k represents the predicted label of the k-order sliding window, specifically the vector representation of the molecular intermediate-atom addition action pair, Dec(s k , a k ) represents the decoded data output by the Transformer model, ||·||2 represents the L2 loss function.

[0064] Step 6, optimize the parameters of the policy network using the reward function, and optimize the Transformer model and the graph encoding model by maximizing the reward value of the real sample and minimizing the reward value of the predicted sample according to the expected value of the reward function. After the optimization is completed, the policy network with optimized parameters is used as the molecular generation model.

[0065] In the embodiment, the Transformer model and the graph encoding model are optimized by maximizing the reward value of the real sample and minimizing the reward value of the predicted sample according to the expected value of the reward function. The specific optimization objective is:

[0066]

[0067] represents the expected value of the reward value of the test sample extracted from the predicted molecular generation sequence generated by the policy network π θ represents the expected value of the reward value of the real sample extracted from the real molecular generation sequence D E

[0068] In the embodiment, the parameters of the policy network are also optimized according to the reward function, and the optimization objective is to maximize the cumulative reward value of the molecular generation sequence:

[0069]

[0070] Among them, T is the length of the prediction sequence. When the number of atoms reaches the set maximum number or the policy network actively terminates the generation, the predicted molecule generation is completed and output.

[0071] ​During specific optimization, sample a batch of (s, a) from the predicted molecule generation sequence and a batch of (s, a) from the true molecule generation sequence. Combine the two batches into one batch and use a k - order sliding window to predict a single time step (s, a). Optimize the parameters of the Transformer model and the graph encoding model by differentiating the prediction errors between the true samples and the predicted samples. Then, optimize the policy network through the cumulative reward function.

[0072] In the embodiment, the algorithm for optimizing the policy network can specifically adopt a reinforcement learning policy optimization algorithm. This reinforcement learning policy optimization algorithm includes PPO, PRTO, DDPG, etc. After optimization, use the policy network with optimized parameters as the molecule generation model.

[0073] Step 7, use the molecule generation model to generate molecules in three - dimensional space.

[0074] In the embodiment, when using the molecule generation model to generate molecules in three - dimensional space, take a single atom as the current molecule intermediate product, and use the molecule generation model to generate an atom - adding action based on the current molecule intermediate product. Construct the next - step molecule intermediate product according to the atom - adding action, and then use the molecule generation model to generate the next - two - step molecule intermediate products based on the next - step molecule intermediate product, and so on, to continuously update the molecule intermediate product to construct molecules in three - dimensional space.

[0075] In the embodiment, when it is necessary to apply it to other downstream small - molecule generation tasks, fine - tune the molecule generation model with small molecules. During fine - tuning, fix the parameters of the Transformer model and the graph encoding model, and only optimize the molecule generation model using the constructed reward function until convergence to obtain a policy network suitable for a specific task. Use the re - optimized policy network as the molecule generation model for the generation of other small molecules to generate new molecules that meet specific properties.

[0076] Based on the same inventive concept, the embodiment also provides a three - dimensional space molecule generation device based on multi - task pre - trained inverse reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above - mentioned Step 1 - Step 7.

[0077] In specific applications, the computer memory can be a volatile memory at the proximal end, such as RAM, or a non - volatile memory, such as ROM, FLASH, floppy disk, mechanical hard disk, etc., or a remote storage cloud. The computer processor can be a central processing unit (CPU), a micro - processing unit (MPU), a digital signal processor (DSP), or a field - programmable gate array (FPGA), that is, the three - dimensional space molecule generation device based on multi - task pre - trained inverse reinforcement learning can be implemented through these processors.

[0078] The three-dimensional space molecular generation method and device provided by the above embodiments construct a reward function for molecular intermediate products based on inverse reinforcement learning, alleviating the problem of sparse rewards in generative learning, and at the same time establishing a connection between the molecular generation behavior and the molecular properties to be optimized.

[0079] The three-dimensional space molecular generation method and device provided by the above embodiments are the first to use the prediction results of a pre-trained Transformer model to construct a reward function for adversarial inverse reinforcement learning behavior discrimination. Different from the existing binary classifier reward functions that may overfit the local differences in single-step actions, the behavior discrimination method based on sequence modeling can effectively discriminate the global dynamic behavior of sequences and avoid overfitting.

[0080] In the three-dimensional space molecular generation method and device provided by the above embodiments, a multi-task pre-trained Transformer model is constructed, and shared knowledge of molecular generation is obtained through molecular generation tasks on multiple data sets to achieve knowledge transfer, so as to better learn the structural knowledge and domain knowledge contained in the three-dimensional space molecular graph.

[0081] Different from the existing property predictors based on complete molecules that are difficult to generalize to molecular generation intermediates and the physical simulator reward functions with high running costs, in the three-dimensional space molecular generation method and device provided by the above embodiments, a reward function for molecular generation behavior is proposed, using the generation step sequence of real molecules as expert guidance to learn more guiding intermediate rewards and improve learning efficiency.

[0082] The above specific embodiments have detailed the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the scope of the principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning, characterized in that It includes the following steps: Using the minimum spanning tree algorithm to construct a real molecule generation sequence based on a chemical molecule, including: randomly selecting a carbon atom or a nitrogen atom from the chemical molecule as the initial atom, obtaining the atom traversal order of the chemical molecule graph according to the minimum spanning tree algorithm, and constructing a real molecule generation sequence composed of molecule intermediate-atom addition action pairs sorted by time steps; Using a policy network to generate an atom addition action based on the current molecule intermediate to construct a predicted molecule generation sequence; Sampling molecule intermediate-atom addition action pairs from the real molecule sequence and the predicted molecule generation sequence respectively as real samples and predicted samples, and performing vector encoding on the real samples and the predicted samples to obtain vector representations; Adding task prompts to the vector representations of the real samples and the predicted samples respectively and using them as pre-training data to input into the Transformer model to obtain the decoded data corresponding to the real samples and the predicted samples respectively; Calculating the regression error based on the real samples, the predicted samples and the decoded data corresponding to both, and constructing a reward function based on the regression error; Optimizing the Transformer model by maximizing the reward value of the real samples and minimizing the reward value of the predicted samples according to the expected value of the reward function, and using the reward function to optimize the parameters of the policy network. After the optimization is completed, the policy network with optimized parameters is used as the molecule generation model; Using the molecule generation model to generate molecules in three-dimensional space.

2. The three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning according to claim 1, wherein, The molecular intermediate-adding atom action pair is represented as (s, a), where s is the current molecular intermediate and a is the atom-adding action to be taken at the current time step, in the form of P is an atom in the current molecular intermediate s, Q is the atom to be added to s, d is the bond length between P and Q, and the bond type is determined by the bond length. is the angle between the newly added edge and the existing edges in the molecule when the number of atoms in s is greater than or equal to 2. is the angle between the plane PQN1 formed by the newly added edges and the plane PN1N2 in s when the number of atoms in s is greater than or equal to 3, where N1 and N2 are the nearest neighbor atom and the next nearest neighbor atom of atom P, respectively.

3. The three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning according to claim 1, wherein The step of using a policy network to generate an atom addition action based on the current molecule intermediate graph to construct a predicted molecule generation sequence includes: The policy network contains six sub-networks, which respectively output P, Q, d, T based on the input current molecular intermediate s, to form an atom addition action P is an atom in the current molecular intermediate s, Q is the atom to be added to s, d is the bond length between P and Q, and the bond type is determined by the bond length. is the angle between the newly added edge and the existing edge in the molecule when the number of atoms in s is greater than or equal to 2. is the angle between the plane PQN1 formed by the newly added edge and the plane PN1N2 in s when the number of atoms in s is greater than or equal to 3, where N1 and N2 are the nearest neighbor atom and the second nearest neighbor atom of atom P respectively, and T represents whether the atom addition action terminates. The current molecule intermediate s and the atom addition action a form a molecule intermediate-atom addition action (s, a) at the current time step. Adding an atom Q to the current molecule intermediate s according to the atom addition action a to obtain the molecule intermediate as the input for the next step. The molecule intermediate-atom addition actions at each time step are sorted by time steps to form a predicted molecule generation sequence.

4. The three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning according to claim 2 or 3, characterized in that The step of performing vector encoding on the real samples and the predicted samples to obtain vector representations includes: The real samples and the predicted samples adopt the same encoding method. When the real samples and the predicted samples are represented as molecular intermediate - adding atom actions (s, a), first, for each atom node in the molecular intermediate s, a vector representation of the atom node is constructed according to the corresponding d when the atom is added to the molecular intermediate, obtaining the initial vector representation of the molecular intermediate. Then, a pluggable graph encoding model is used to encode based on the initial vector representation to obtain the vector representation of the molecular intermediate; of the vector to construct the vector representation of the atom node, obtaining the initial vector representation of the molecular intermediate, and then using a pluggable graph encoding model to encode based on the initial vector representation to obtain the vector representation of the molecular intermediate; For adding atomic action a, d in a, it is converted into a vector representation by spherical Fourier Bessel basis functions. P and Q in a are one-hot encoded. The length of the one-hot vector of P is the number of atoms in s, and the length of the one-hot vector of Q is the number of allowed atomic types. The vector concatenation of P, Q, d, is used as the vector representation of a; Concatenating the vector representation of the molecule intermediate s and the vector representation of the atom addition action a as the vector representation of the molecule intermediate-atom addition action (s, a).

5. The three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning according to claim 4, wherein Convert d in a according to the following formula, to a vector representation by spherical Fourier Bessel basis functions where c is the truncation distance, z ln is the n-th root of the l-th order Bessel function, Y l m is the spherical harmonic function of degree m and order l, l ∈ {0, …, N SHBF}, n ∈ {1, …, N SRBF}, m ∈ {-l, …, l}, N SHBF , N SRBF are the preset maximum orders of the spherical harmonic function and the spherical Bessel function, respectively.

6. The three-dimensional space molecule generation method based on multi-task pre-trained inverse reinforcement learning according to claim 4, characterized in that When using a pluggable graph encoding model to perform vector encoding on the molecule intermediate, during the optimization process, the parameters of the graph encoding model are optimized by maximizing the reward value of the real samples and minimizing the reward value of the predicted samples according to the expected value of the reward function.

7. The three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning according to claim 1, characterized in that The constructed reward function is expressed as: δ = ||Dec(s k , a k ) - [s k , a k ||₂ The objective function for optimizing the parameters of the Transformer model is: Among them, represents the expected reward value of the test sample extracted from the predicted molecular generation sequence generated by the policy network π θ , and represents the expected reward value of the real sample extracted from the real molecular generation sequence D E . w represents the optimized parameter, and r w (s,a) represents the reward value of the molecular intermediate-atom addition action pair. δ represents the prediction error of the Transformer model. [s k ,a k represents the predicted label of the k-order sliding window, specifically the vector representation of the molecular intermediate-atom addition action pair. Dec(s k ,a k ) represents the decoded data output by the Transformer model. ||·||2 represents the 2-loss function.

8. The three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning according to claim 4, wherein Each sub-network of the policy network uses an MLP, and the graph encoding model uses schnet. During the optimization, a k-order sliding window is used for the molecule intermediate-atom addition action pairs as the input to predict the decoded data of the Transformer model.

9. The three-dimensional space molecular generation method based on multi-task pre-trained inverse reinforcement learning according to claim 4, characterized in that When applied to other downstream small molecule generation tasks, the small molecule pair molecular generation model is fine-tuned. During fine-tuning, the parameters of the Transformer model and the graph encoding model are fixed, and only the constructed reward function is used to re-optimize the molecular generation model until convergence. The re-optimized molecular generation model is used for the generation of other small molecules.

10. A three-dimensional space molecular generation device based on multi-task pre-trained inverse reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional space molecular generation method based on multi-task pre-training inverse reinforcement learning according to any one of claims 1-9.

Images