A Method and System for Generating Non-Peptide Macrocyclic Molecules Based on Curriculum-Based Reinforced Learning and Multi-Objective Optimization

Abstract

The application discloses a non-peptide macrocyclic molecule generation method and system based on course reinforcement learning and multi-objective optimization. The application adopts a character string sequence representing a molecular structure with chemical effectiveness guarantee, and pre-trains and constructs a reinforcement learning strategy network by using an autoregressive generation model; course reinforcement learning is adopted, and constraints such as molecular effectiveness, drug-likeness, macrocyclic structure, synthetic accessibility, molecular flexibility and target binding capacity are introduced in stages from low to high in terms of task difficulty, so that the generation model is migrated from a general small molecule chemical space to a non-peptide macrocyclic molecule chemical space; the multi-objective properties are scalarized and evaluated by using a Pareto dominance relationship to obtain a reinforcement learning reward, the strategy network is updated to obtain an optimized generation model, and then candidate non-peptide macrocyclic molecules are generated and property evaluation is performed. The application can be applied to macrocyclic molecule virtual library construction, target-oriented molecule optimization and lead compound discovery.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence drug design, computer-aided drug development, molecular generation, reinforcement learning, and molecular multi-objective optimization, and particularly to a method and system for generating non-peptide macrocyclic molecules based on curriculum reinforcement learning and multi-objective optimization. Background Technology

[0002] In artificial intelligence-based drug design and computer-aided drug development, non-peptide macrocyclic molecules possess significant application value for "difficult-to-drug" targets such as protein-protein interactions, kinases, and membrane proteins due to their combination of the pharmacokinetic advantages of small molecule drugs and the high binding selectivity of biomolecules. Compared to traditional small molecules, non-peptide macrocyclic molecules typically have larger molecular scales, more complex cyclic topologies, and stronger conformational constraints, enabling them to enhance target binding capacity through conformational pre-organization and multi-point interactions. However, the complex trade-offs between the closed-loop constraints of macrocyclic structures, molecular flexibility, synthetic accessibility, and drug-likeness make their rational design and optimization challenging. Traditional methods based on empirical modification, rule enumeration, or local search usually rely on existing scaffolds or linear precursors, allowing only local exploration within a limited chemical space, making it difficult to achieve global optimization and de novo generation of non-peptide macrocyclic molecules.

[0003] In related technologies, deep learning molecular generation models have been widely used in small molecule design. Common methods include generation models based on SMILES sequences, molecular graphs, or 3D structures, which can be combined with reinforcement learning to optimize properties such as QED, synthetic accessibility, and docking scoring. However, these methods still have significant shortcomings when directly applied to the generation of non-peptide macrocyclic molecules: on the one hand, traditional SMILES representation is prone to problems such as ring closure errors, illegal valence bonds, or unresolved molecules during macrocyclic generation, leading to a decrease in generation efficiency; on the other hand, existing models are mostly trained on traditional small molecule data, and the generation distribution tends to be biased towards structurally simple small molecule regions, making it difficult to stably migrate to sparse macrocyclic chemical spaces.

[0004] Furthermore, macrocyclic molecule design involves multiple conflicting objectives, such as drug-likeness, synthetic accessibility, ring size, molecular weight, polarity, flexibility, and target binding ability. Single-objective optimization or statically weighted multi-objective optimization can easily lead to reward bias, strategy degradation, or pattern collapse, making it difficult to obtain candidate molecules that possess both macrocyclic structural characteristics and drug development potential. Therefore, there is an urgent need for a macrocyclic molecule generation method that can improve chemical efficiency, stably explore the chemical space of non-peptide macrocycles, and coordinate the optimization of multiple objectives. Summary of the Invention

[0005] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims. This application aims to at least address one of the technical problems existing in the prior art. To this end, this application provides a method and system for generating non-peptide macrocyclic molecules based on curriculum reinforcement learning and multi-objective optimization. This application uses SELFIES as a unified molecular representation, utilizes an autoregressive Transformer chemical language model for pre-training, and introduces curriculum reinforcement learning and Pareto multi-objective optimization strategies on the basis of the pre-trained model to guide the generation model from the general small molecule distribution to the chemical space of non-peptide macrocyclic molecules in stages, achieving stable, efficient, and controllable generation of non-peptide macrocyclic molecules.

[0006] The method for generating non-peptide macrocyclic molecules based on curriculum reinforcement learning and multi-objective optimization in this invention includes the following steps:

[0007] S1, Obtain the molecular dataset for training, and perform standardization, deduplication, validity filtering, and screening of non-peptide macrocyclic molecules on the molecular dataset to obtain the training molecule set;

[0008] S2, convert the molecular structures in the training molecule set into string sequence representations with guaranteed chemical validity, and construct the corresponding token vocabulary;

[0009] S3, input the string sequence into the autoregressive generation model for pre-training, so that the autoregressive generation model learns the statistical prior distribution, local combination rules and long-range dependencies of molecular structures;

[0010] S4, a reinforcement learning policy network is constructed based on the pre-trained autoregressive generative model to model the molecular generation process as a sequential decision-making process;

[0011] S5. A course-based reinforcement learning approach is adopted, introducing multiple optimization objectives in stages according to the order of task difficulty from low to high, so that the reinforcement learning strategy network gradually migrates from the general small molecule chemical space to the non-peptide macrocyclic molecule chemical space; the multiple optimization objectives include at least molecular effectiveness constraints, drug-likeness constraints, macrocyclic structure constraints, synthetic accessibility constraints, and molecular flexibility constraints.

[0012] S6 utilizes Pareto dominance to perform scalar evaluation of the multi-objective properties of the generated molecules and obtains reinforcement learning rewards.

[0013] S7. Update the reinforcement learning policy network according to the reinforcement learning reward to obtain the optimized non-peptide macrocyclic molecule generation model;

[0014] S8. Candidate non-peptide macrocyclic molecules are generated using the optimized non-peptide macrocyclic molecule generation model.

[0015] The non-peptide macrocyclic molecule generation system based on curriculum reinforcement learning and multi-objective optimization in this invention includes:

[0016] The data preparation module is configured to acquire a molecular dataset for training, and to perform standardization, deduplication, validity filtering, and screening of non-peptide macrocyclic molecules on the molecular dataset to obtain a training molecule set.

[0017] The molecular representation module is configured to convert the molecular structures in the training molecule set into string sequence representations with guaranteed chemical validity, and to construct a corresponding token vocabulary.

[0018] The pre-training module is configured to input string sequences into the autoregressive generative model for pre-training, so that the autoregressive generative model learns the statistical prior distribution, local combinatorial rules and long-range dependencies of molecular structures.

[0019] The reinforcement learning modeling module is configured to build a reinforcement learning policy network based on a pre-trained autoregressive generative model, and to model the molecular generation process as a sequential decision-making process.

[0020] The course reinforcement learning optimization module is configured to adopt a course reinforcement learning approach, introducing multiple optimization objectives in stages according to the order of task difficulty from low to high, so that the reinforcement learning strategy network gradually migrates from the general small molecule chemical space to the non-peptide macrocyclic molecule chemical space; the multiple optimization objectives include at least molecular effectiveness constraints, drug-likeness constraints, macrocyclic structure constraints, synthetic accessibility constraints, and molecular flexibility constraints.

[0021] The multi-objective reward calculation module is configured to use Pareto dominance to perform scalar evaluation of the multi-objective properties of the generated molecules and obtain reinforcement learning rewards.

[0022] The policy update module is configured to update the reinforcement learning policy network according to the reinforcement learning reward to obtain an optimized non-peptide macrocyclic molecule generation model.

[0023] The molecule generation module is configured to generate candidate non-peptide macrocyclic molecules using the optimized non-peptide macrocyclic molecule generation model.

[0024] The present invention has at least the following beneficial effects:

[0025] First, this invention uses SELFIES to represent molecular structures, which, compared to SMILES, can reduce ring closure errors and illegal valence bond problems at the representation level, and improve the chemical efficiency of the generated molecules. It is especially suitable for generating non-peptide macrocyclic molecules with complex ring closure constraints.

[0026] Second, this invention adopts a course-based reinforcement learning strategy, which decomposes the task of generating non-peptide macrocyclic molecules into multiple stages with increasing difficulty, and gradually introduces constraints such as molecular effectiveness, drug-likeness, macrocyclic structure, synthetic accessibility, flexibility and target binding ability, so as to avoid directly applying reinforcement constraints that would lead to unstable model training or degeneration of generated molecules.

[0027] Third, this invention uses the Pareto multi-objective optimization method to dynamically coordinate multiple objectives, without relying on fixed weights and linear weighting, which can alleviate the conflict between drug-likeness, synthetic accessibility, macrocyclic structure and target binding ability.

[0028] Fourth, this invention can significantly increase the proportion of non-peptide macrocyclic molecules generated. In the fifth stage, the proportion of macrocyclic molecules reached 64.80%±3.2%, the average ring size was 12.87±0.61, while maintaining a high molecular diversity of 0.89±0.01.

[0029] Fifth, this invention exhibits superior technical performance compared to single-objective optimization and static weighted multi-objective optimization methods. In comparative experiments, the macrocyclic ratios in single-objective optimization and static weighted multi-objective optimization were 32.30%±6.3% and 32.50%±3.7%, respectively, while this invention achieved 64.80%±3.2%. This demonstrates that this invention is not a simple combination of existing technologies, but rather produces significant and unexpected technical effects in non-peptide macrocyclic molecule generation tasks.

[0030] Sixth, the present invention can be further extended to target-guided molecule generation. In one embodiment, JAK2 is used as the target protein, and a QuickVina2-based docking score reward is introduced in the sixth stage. The molecules generated in the sixth stage show a better JAK2 docking score distribution than those in the fifth stage, with the proportion of molecules exceeding the fenzotinib reference threshold being approximately five times that of the fifth stage. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the overall architecture of a non-peptide macrocyclic molecule generation framework based on course reinforcement learning and multi-objective optimization, provided in an embodiment of this application.

[0032] Figure 2a The first statistical distribution plot of molecular weight, cLogP (oil-water partition coefficient), QED (drug-likeness), SAScore (syntheticability), SMILES length, and ring size in the training dataset provided for embodiments of this application.

[0033] Figure 2bThe second statistical distribution plot of molecular weight, cLogP (oil-water partition coefficient), QED (drug-likeness), SAScore (syntheticability), SMILES length, and ring size in the training dataset provided for embodiments of this application.

[0034] Figure 3 UMAP chemical spatial distribution diagrams of molecules generated at different course reinforcement learning stages provided in the embodiments of this application.

[0035] Figure 4 The property distribution and scaffold diversity analysis diagram of molecules generated at different course reinforcement learning stages provided in the embodiments of this application.

[0036] Figure 5 Pearson correlation analysis diagram of the properties of generated molecules provided in the embodiments of this application.

[0037] Figure 6a The first comparison result diagram of the method of the present invention provided in the embodiments of this application with single-objective optimization and static weighted multi-objective optimization methods.

[0038] Figure 6b The second comparison result diagram shows the method of the present invention provided in the embodiments of this application with single-objective optimization and static weighted multi-objective optimization methods.

[0039] Figure 6c The third comparison result diagram shows the method of the present invention provided in the embodiments of this application with single-objective optimization and static weighted multi-objective optimization methods.

[0040] Figure 6d The fourth comparison result diagram shows the method of the present invention provided in the embodiments of this application with single-objective optimization and static weighted multi-objective optimization methods.

[0041] Figure 7a Property distribution diagrams of molecules generated in the fifth and sixth stages provided in embodiments of this application.

[0042] Figure 7b JAK2 docking score distribution diagrams of molecules generated in the fifth and sixth stages provided in the embodiments of this application.

[0043] Figure 7c The following are representative candidate molecular structures provided for embodiments of this application.

[0044] Figure 7d The binding conformation diagrams of representative candidate molecular structures provided in the embodiments of this application are shown. Detailed Implementation

[0045] The technical solution of this application will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. It should be understood that the described embodiments are only used to explain this application and are not intended to limit the scope of protection of this application. In the absence of conflict, the embodiments and technical features in the embodiments of this application can be combined with each other.

[0046] In this application, "multiple" refers to two or more. Terms such as "first" and "second" are used only to distinguish technical features and do not indicate relative importance, quantity limitations, or order of priority.

[0047] In this application, the term "non-peptide macrocyclic molecule" refers to an organic molecule whose molecular structure contains a macrocyclic structure composed of 12 or more atoms and whose main backbone feature is not continuous peptide bond repeating units.

[0048] In this application, "course reinforcement learning" refers to a training strategy that introduces optimization objectives such as molecular effectiveness, drug-likeness, macrocyclic structure constraints, synthetic accessibility, molecular flexibility, and target binding ability in stages during the reinforcement learning training process, in order of increasing task difficulty.

[0049] In this application, the "Pareto multi-objective optimization" refers to judging the overall merit of candidate molecules based on the dominance relationship between multiple objectives, rather than using fixed weights to simply linearly weight multiple objectives.

[0050] like Figure 1 As shown, this application provides a method for generating non-peptide macrocyclic molecules based on curriculum reinforcement learning and multi-objective optimization, comprising the following steps:

[0051] S1, Obtain the molecular dataset for training, and perform standardization, deduplication, validity filtering, and screening of non-peptide macrocyclic molecules on the molecular dataset to obtain the training molecule set;

[0052] S2, convert the molecular structures in the training molecule set into SELFIES sequence representations and construct a SELFIES token vocabulary;

[0053] S3, input the SELFIES sequence into the autoregressive Transformer generative model for pre-training, so that the model learns the statistical prior distribution, local combinatorial rules and long-range dependencies of molecular structures;

[0054] S4. A reinforcement learning policy network is constructed based on the pre-trained autoregressive Transformer generative model, and the policy network is fine-tuned using the proximal policy optimization algorithm (PPO).

[0055] S5 adopts a course-based learning approach to introduce molecular effectiveness constraints, drug-likeness constraints, macrocyclic structure constraints, synthetic accessibility constraints, molecular flexibility constraints, and target binding score constraints in stages, so that the generative model gradually migrates from the general small molecule chemical space to the non-peptide macrocyclic molecule chemical space.

[0056] S6 utilizes Pareto dominance to perform scalar evaluation of the multi-objective properties of the generated molecules and obtains reinforcement learning rewards.

[0057] S7. Based on the reinforcement learning reward update generation strategy, the optimized non-peptide macrocyclic molecule generation model is obtained.

[0058] S8. Using the optimized non-peptide macrocyclic molecule generation model, candidate non-peptide macrocyclic molecules are generated, and the generated molecules are subjected to property calculations, molecular diversity analysis, and target binding ability assessment.

[0059] In one embodiment, the macrocyclic molecule dataset used for model pre-training is derived from the ChEMBL and Macrocycle-DB databases.

[0060] First, 38,342 candidate macrocyclic compounds represented by SMILES were collected from the aforementioned database. Then, the raw data was systematically cleaned and filtered, and finally 25,633 molecules were retained for model pre-training.

[0061] Specifically, the data cleaning and filtering steps include: removing natural products and their direct derivatives to make the training set more focused on synthetically accessible and rationally designed non-peptide macrocyclic chemical spaces; retaining molecules with a maximum ring size of 12 to 18 atoms to cover representative medium to large-sized macrocyclic structures in medicinal chemistry; removing molecules with SMILES sequence lengths exceeding 200 tokens to reduce the impact of overly complex structures on sequence modeling and training stability; using RDKit to perform validity checks on molecules, removing molecules that cannot be resolved, have abnormal valence states, or have irregular structures; and performing deduplication, standardization, and normalization on the molecules.

[0062] In one embodiment, statistical analysis was performed on the training set. The average molecular weight of the molecules in the training set was 621.9 Da, the average cLogP was 3.58, the average QED was 0.33, the average SAScore was 5.22, the SMILES sequence length ranged from 20 to 250, the average length was 76.2, and the average ring size was 14.2. These statistical results indicate that the training set covers the chemical space of non-peptide macrocyclic molecules with high molecular weight, strong structural complexity, and typical macrocyclic structural features. Figure 2a .

[0063] Furthermore, the molecular properties in the training set exhibit a clear trend with varying molecular weight and ring size, see [see details]. Figure 2b As molecular weight increases, QED systematically decreases while SAScore increases; conversely, as ring size increases, QED decreases while SAScore increases. This indicates an inherent conflict between structural scale, drug-likeness, and synthetic accessibility in the design of non-peptide macrocyclic molecules, and also underscores the need for multi-objective optimization methods rather than single-objective optimization methods.

[0064] In one embodiment, the pretreated molecule is converted from SMILES representation to SELFIES sequence representation. SELFIES is a molecular string representation with strong chemical validity guarantees, which can reduce the ring closure errors and illegal valence bond problems that are prone to occur when generating macrocyclic molecules using traditional SMILES.

[0065] Specifically, each normalized SMILES molecule is encoded using SELFIES to obtain a sequence consisting of multiple SELFIES tokens. A token vocabulary is then constructed based on all SELFIES sequences in the training set. This vocabulary includes basic SELFIES tokens, start tokens, end tokens, padding tokens, unknown tokens, or masked tokens.

[0066] During the generation process, the model starts with the initial token and autoregressively generates the next SELFIES token until a termination token is generated or the maximum sequence length is reached. The generated SELFIES sequences can be further converted back to SMILES to be compatible with downstream processes such as RDKit property calculations, Morgan fingerprint calculations, molecular docking, and candidate molecule screening.

[0067] By employing SELFIES representation, this application improves chemical efficiency at the action space level of molecular generation, thereby providing more stable initial conditions for subsequent reinforcement learning optimization.

[0068] In one embodiment, the generative model employs an autoregressive chemical language model based on a Transformer decoder architecture. In a preferred embodiment, the model is a GPT-2 style causal language model used to predict the probability distribution of the next token based on the generated SELFIES token sequence.

[0069] The autoregressive Transformer generative model comprises a token embedding layer, a position encoding layer, a multi-head self-attention module, a feed-forward neural network module, a residual connection module, a layer normalization module, an output linear layer and a softmax probability output layer.

[0070] Let the generated SELFIES sequence be x=(x₁,x₂,...,x_T), wherein x_t represents the t-th token, and T represents the sequence length. The autoregressive model decomposes the generation probability of the molecular sequence into: P_θ(x)=∏_{t=1}^T P_θ(x_t|x_{<t}). Wherein, x_{<t} represents the token sequence before the current position, and θ represents the model parameters.

[0071] In the pre-training stage, a teacher forcing mechanism and maximum likelihood estimation are used to train the model. The pre-training objective is to minimize the masked cross-entropy loss: L_CE=-∑_{t=1}^T log P_θ(x_t|x_{<t}). Wherein, padding positions do not participate in loss calculation.

[0072] Through pre-training, the model learns the statistical rules in non-peptidic macrocyclic molecular sequences, the combination rules of atoms and structural fragments, long-range dependencies and basic chemical priors. The pre-trained model is used as the initial policy network for the subsequent curriculum reinforcement learning stage.

[0073] In the reinforcement learning stage, the SELFIES molecular generation process is modeled as a sequential decision-making process.

[0074] At time step t, the state s_t is the partial SELFIES sequence that has been currently generated, that is, s_t=(x₁,x₂,...,x_{t-1}); the action a_t is to select the next token from the SELFIES vocabulary, that is, a_t=x_t; the policy network π_θ(a_t|s_t) represents the probability of selecting each token under the current state.

[0075] When the model generates a termination token or reaches the maximum sequence length, the generation process ends and a complete SELFIES sequence is obtained. Then the SELFIES sequence is converted to SMILES, and RDKit is used to judge whether the molecule is valid. If the conversion fails, RDKit parsing fails, there is an illegal structure or a forbidden symbol is included, a penalty is given; if the molecule is valid, the molecular properties are further calculated and a reward is obtained.

[0076] In one embodiment, the reinforcement learning policy network adds a value function head based on the pre-trained Transformer language model to estimate the state value, thereby forming a joint policy-value network structure.

[0077] In one embodiment, this application employs the Proximal Policy Optimization (PPO) algorithm to update the generated policy. PPO improves the stability of reinforcement learning training by limiting the update magnitude between the old and new policies, preventing model crashes caused by excessive policy updates.

[0078] The objective function for PPO pruning is:

[0079] LPPO(θ)=Et[min(ρt(θ)Ât, clip(ρt(θ),1-ε',1+ε')Ât)].

[0080] Where ρt(θ) = πθ(at|st) / πθold(at|st), πθ represents the current policy network, πθold represents the old policy network, Ât represents the dominance function, and ε' represents the PPO pruning parameter.

[0081] To prevent the generated policy from deviating excessively from the pre-training distribution during reinforcement learning, this embodiment further introduces a KL divergence regularization term: L(θ) = LPPO(θ) - βEt[DKL(πθ(·|st)||πref(·|st))]. Here, πref represents the reference policy network, preferably the initial generated model after pre-training; β represents the KL constraint coefficient; and DKL represents the KL divergence.

[0082] By using the KL regularization term, this application is able to optimize the target properties while maintaining the chemical rationality of the generation distribution, reducing the risk of reward speculation and structural degradation.

[0083] In one embodiment, the curriculum reinforcement learning comprises five consecutive training phases. After each phase completes a fixed number of reinforcement learning update steps, a batch of molecules is independently sampled for property evaluation and statistical analysis.

[0084] In one specific embodiment, 1,000 molecules are sampled after each stage for property statistics and stage comparison.

[0085] Phase 1: Optimization of Molecular Resolvability and Structural Effectiveness

[0086] The goal of Phase 1 is to guide the model to generate valid molecules that can be decoded by SELFIES, transformed by SMILES, and parsed by RDKit. The reward function for this phase includes a molecule validity reward R1 = Rvalid. Here, Rvalid = 1 if the generated molecule can be correctly parsed by RDKit, and 0 or -1 otherwise. Phase 1 primarily serves to stabilize the initial reinforcement learning policy, allowing the model to explore within the distribution of valid molecules.

[0087] Phase Two: Basic Property Optimization Phase

[0088] Phase Two, while maintaining molecular effectiveness, introduces fundamental drug-likeness constraints, such as QED and basic physicochemical properties. The reward function can be expressed as R² = Rvalid + λQEDRQED, where RQED can be directly calculated using the QED value obtained from RDKit, and λQED is the QED reward weight. The average QED of the molecules generated in Phase Two increased to 0.45 ± 0.09, and the SAScore was 4.51 ± 0.33, but the proportion of macrocyclic molecules decreased to 5.90% ± 0.63%, indicating that simple drug-likeness optimization can easily bias the model towards traditional small molecule regions.

[0089] Phase Three: Enhanced Drug-like Properties and Preliminary Structural Regulation

[0090] Phase 3 further optimized drug-like properties and introduced preliminary structural constraints, allowing the model to continue optimizing drug-related properties within the effective molecular space. The average QED of the molecules generated in this phase reached 0.55±0.11, the SAScore was 4.31±0.19, and the molecular diversity was 0.88±0.01. However, the proportion of macrocyclic molecules remained only 10.20%±1.5%, with an average ring size of 7.97±0.45. These results indicate that, in the absence of explicit macrocyclic structural constraints, reinforcement learning still struggles to automatically transfer to non-peptide macrocyclic chemical spaces.

[0091] Phase Four: Introduction of Large-Ring Structure Constraints

[0092] The four-stage explicit introduction of macrocyclic structure constraints and macrocyclic-specific physicochemical property constraints allows the generative model to shift from traditional small molecule regions to non-peptide macrocyclic regions. (See...) Figure 3 The reward function can be expressed as: R4 = Rvalid + λmacroRmacro + λringRring + λMWRMW + λlogPRlogP + λTPSARTPSA + λHBDRHBD + λFlexRFlex.

[0093] After introducing macrocyclic structure constraints in stage four, the proportion of macrocycles in the generated molecules significantly increased to 63.90% ± 4.4%, the average ring size increased to 14.10 ± 0.52, and the average molecular weight increased to 659.99 ± 41 Da. These results demonstrate that explicit macrocyclic constraints can effectively drive the generative model to migrate to the macrocyclic chemical space.

[0094] Phase 5: Multi-objective balance optimization phase

[0095] Phase 5 further optimizes QED, SAScore, macrocyclic structure constraints, physicochemical properties, and molecular flexibility based on Phase 4, enabling the model to achieve a more refined multi-objective equilibrium within the non-peptide macrocyclic chemical space. (See...) Figure 4 and Figure 5 .

[0096] The reward function can be expressed as:

[0097] R5=R4+λQEDRQED+λSARS A+λdivRdiv-λinvalidPinvalid-λdupPdup.

[0098] The results of stage five showed that the macrocyclic ratio was 64.80%±3.2%, the average ring size was 12.87±0.61, the average molecular weight was 517.79±31 Da, the cLogP was 4.53±0.14, the QED was 0.42±0.06, the SAScore was 6.34±0.24, the TPSA was 128.00±17 Ų, the HBD was 2.53±0.11, the Flex was 0.17±0.03, and the molecular diversity was 0.89±0.01.

[0099] Compared to Phase 4, Phase 5, while maintaining a high proportion of macrocyclic structures, restored QED from 0.27±0.04 to 0.42±0.06 and improved SAScore from 6.53±0.21 to 6.34±0.24, indicating that Phase 5 achieved a better balance between macrocyclic structural complexity and drug-likeness.

[0100] In one embodiment, this application employs a macrocyclic-specific physicochemical property constraint system to jointly define and regulate the structural scale, physicochemical properties, and conformational flexibility of non-peptide macrocyclic molecules. The macrocyclic constraints include the contents shown in the table below.

[0101]

[0102] The aforementioned constraints were not arbitrarily set, but were determined based on a comprehensive analysis of training data statistics, the chemical characteristics of non-peptide macrocyclic molecules, and the results of generative experiments. The average molecular weight of the molecules in the training set is 621.9 Da, the average cLogP is 3.58, and the average ring size is 14.2, indicating that the dataset mainly covers molecular regions with medium to high molecular weights and typical macrocyclic structural characteristics.

[0103] Specifically, a maximum ring size of 12-20 is used to ensure that the generated molecule has macrocyclic characteristics and avoid conformational and synthetic instabilities caused by excessively large ring structures; a molecular weight of 500-900 Da is used to cover most macrocyclic molecular sizes with development potential; cLogP 3.0-5.0 is used to balance membrane permeability and solubility; TPSA 80-120 Ų and HBD 1-3 are used to control polarity and the number of hydrogen bond donors; and Flex 0.10-0.20 is used to avoid excessive molecular flexibility and improve the degree of conformational pre-organization.

[0104] For any continuous property pi, its interval reward function can be defined as follows: Ri = 1 when li ≤ pi(m) ≤ ui; otherwise, Ri = -ηi·d(pi(m), [li, ui]). Here, li and ui are the lower and upper limits of the target interval, respectively, d(·) represents the distance to the target interval, and ηi represents the penalty coefficient.

[0105] In other embodiments, the above numerical range may be appropriately adjusted according to specific targets, molecular library distribution or drug development needs, without affecting the core implementation of the technical solution of this application.

[0106] In the design of non-peptide macrocyclic molecules, there are significant conflicts among multiple optimization objectives. For example, larger ring size and molecular weight facilitate the formation of macrocyclic structures but may reduce the QED and increase the SAS score; stronger target binding ability may be accompanied by higher polarity, larger molecular weight, or more complex structures; improving synthetic accessibility may limit structural complexity. Therefore, it is difficult to effectively coordinate multiple objectives using a fixed-weight linear weighting method.

[0107] Therefore, this application employs Pareto dominance for multi-objective reward calculation. For a generated molecule m, its objective vector f(m) = [f1(m), f2(m), ..., fn(m)] is calculated. Objectives may include QED, SAScore, maximum ring size, molecular weight, cLogP, TPSA, HBD, Flex, molecular diversity, and docking score. For indices that need to be minimized, such as SAScore and docking score, they can be converted into maximization objectives by taking negative values ​​or normalization.

[0108] Establish a sliding window pool M for historical generated molecules. For the current generated molecule m and the historical molecule m′, if fi(m)≥fi(m′) for all targets i, and there exists at least one target j such that fj(m)>fj(m′), then m is considered to be Pareto dominant to m′.

[0109] Calculate the number of historical molecules dominated by the current molecule, Nwin, and the number of historical molecules dominated by the current molecule, Nloss, and then calculate the Pareto reward RPareto = (Nwin - Nloss) / (Nwin + Nloss + ε), where ε is a constant to prevent the denominator from being zero.

[0110] In another embodiment, the normalized form RPareto=Nwin / (|M|+ε) can also be used.

[0111] The final reward function can be expressed as:

[0112] R(m)=Rbase(m)+Rconstraint(m)+RPareto(m)-P(m).

[0113] Where Rbase(m) represents the basic reward, Rconstraint(m) represents the constrained reward, RPareto(m) represents the Pareto reward, and P(m) represents the penalty term.

[0114] By means of the above method, this application can dynamically evaluate the comprehensive advantages of candidate molecules based on the property distribution of historically generated molecules without relying on fixed weights, thereby alleviating the problems of scale difference and objective conflict in multi-objective optimization.

[0115] In one embodiment, to avoid reward speculation, duplicate generation, and pattern collapse during reinforcement learning, this application introduces robust control mechanisms during sampling and decoding, including illegal token masking, duplicate token suppression, sample deduplication, invalid molecule penalty, duplicate molecule penalty, scaffold diversity analysis, and Morgan fingerprint diversity calculation.

[0116] In one embodiment, molecular diversity is calculated based on Morgan fingerprints and Tanimoto similarity. Let the generated molecule set be G, where the Morgan fingerprint similarity between any two molecules is Sim(i,j). Then, diversity can be defined as Diversity = 1 - [2 / (|G|(|G|-1))]∑i <j Sim(i,j)。

[0117] In the experiment, the molecular diversity from the first to the fifth stage remained in the range of 0.86-0.89, and the molecular diversity in the fifth stage reached 0.89±0.01, indicating that the curriculum reinforcement learning method of this application significantly increased the proportion of macrocycles without causing serious pattern collapse.

[0118] To verify that this application is not a simple combination of existing technologies, in one embodiment, the method of this application is compared with a single-objective optimization method and a static weighted multi-objective optimization method.

[0119] Single-objective optimization methods use QED as the sole reward signal; statically weighted multi-objective optimization methods sum the QED, SAScore, and macrocyclic constraints according to preset fixed weights. See Figure 6a , Figure 6b , Figure 6c and Figure 6d The results are as follows:

[0120]

[0121] As the results above show, while single-objective optimization achieves a high QED, the generated molecules are mainly concentrated in traditional small molecule regions, with macrocyclic molecules accounting for only 32.30%. Static weighted multi-objective optimization only increases the macrocyclic molecule proportion to 32.50%, showing limited effectiveness. In contrast, this application increases the macrocyclic molecule proportion to 64.80%, the average ring size to 12.87, while maintaining the highest molecular diversity of 0.89.

[0122] This demonstrates that, through the synergistic effect of course reinforcement learning and Pareto multi-objective optimization, this application achieves a stable migration of the generative distribution from small molecule regions to non-peptide macrocyclic molecule regions. This effect cannot be directly predicted by any single technique, such as SELFIES, Transformer, PPO, or Pareto optimization.

[0123] Furthermore, when considering only the macrocyclic subset with a ring size of not less than 12, the macrocyclic molecules generated in this application have an average ring size of 15.77±0.33, a MW of 556.44±28 Da, a cLogP of 4.80±0.12, a TPSA of 104.68±23 Ų, a HBD of 2.68±0.09, a Flex of 0.13±0.02, and a diversity of 0.88±0.01. This indicates that this application can still maintain good structural diversity and macrocyclic characteristics even in a more stringent subset of macrocyclic molecules.

[0124] In one embodiment, to verify the scalability of this application in target-directed drug design, a sixth stage is further introduced after the fifth stage, using JAK2 kinase as the specific target protein for structure-directed macrocyclic molecule optimization, see [link to previous embodiment]. Figure 7a .

[0125] JAK2 is a kinase target with clear clinical significance, associated with a variety of hematologic malignancies and inflammatory diseases. Its binding pocket has certain superficial and flexible characteristics, which poses a challenge to the design of traditional small molecule inhibitors. Therefore, it is suitable as a target example for the design of non-peptide macrocyclic molecules.

[0126] In the sixth stage, the generation strategy obtained in the fifth stage is used as the initial strategy for further reinforcement learning optimization. Since the generation strategy in the fifth stage is already in a distribution region rich in macrocyclic molecules and has good drug-likeness and diversity, the target-oriented optimization in the sixth stage can be carried out on the basis of reasonable macrocyclic molecule priors, rather than directly optimizing docking scores in the unconstrained chemical space.

[0127] The reward function for the sixth stage can be expressed as: R6 = R5 + λdockRdock, where Rdock is the reward item obtained based on the JAK2 molecular docking score.

[0128] In one embodiment, QuickVina2 is used to calculate the docking score. Since a lower docking score generally indicates a more favorable docking, the docking score can be converted into a reward Rdock = -DockScore(m), or a normalized form can be used, see [link to documentation]. Figure 7b .

[0129] In the sixth stage, the macrocyclic structure constraint remains enabled to prevent the model from speculatively increasing the docking reward by unreasonably increasing molecular size, excessively increasing polarity, or generating chemically unreliable structures.

[0130] In one specific embodiment, the docking score of the marketed JAK2 inhibitor Fedratinib, at -9.8 kcal / mol under the same docking conditions, was used as a reference threshold. Experimental results show that, compared to stage five, the docking score distribution of molecules generated in stage six shifts towards a more favorable direction overall, and the proportion of molecules exceeding the Fedratinib reference threshold is approximately five times that of stage five.

[0131] Furthermore, several representative non-peptide macrocyclic candidate molecules were screened from the molecules generated in stage VI for binding mode analysis. These molecules formed hydrogen bonds, hydrophobic interactions, or aromatic interactions with residues such as ARG-980, ASN-981, LYS-857, LEU-932, LEU-855, SER-936, TYR-931, and GLY-935 in the JAK2 binding pocket. Among them, some candidate molecules were able to establish interactions with the key residue LEU-932 in the JAK2 kinase hinge region, simultaneously occupying both the hinge region and the hydrophobic back pocket region, exhibiting binding mode characteristics consistent with known kinase inhibitors.

[0132] It should be noted that the docking scores and binding pattern analyses described above are only used for calculating the screening and prioritization of candidate molecules, and are not directly equivalent to actual biological activity. The actual synthetic feasibility, biological activity, and pharmacokinetics of the generated molecules still need to be further verified through subsequent synthetic experiments, enzyme activity experiments, cell experiments, or other experimental methods.

[0133] In one embodiment, candidate molecules are generated by sampling using a trained non-peptide macrocyclic molecule generation model, see [link to article]. Figure 7c The following steps were followed for screening: The generated SELFIES sequences were converted to SMILES; molecule validity was assessed using RDKit; molecules containing illegal symbols, abnormal valence states, or unresolved molecules were removed; the maximum ring size was calculated, retaining molecules with a maximum ring size of at least 12; typical peptide macrocyclic molecules were removed based on peptide bond number, continuous amide bond characteristics, or predefined substructure rules; QED, SAScore, MW, cLogP, TPSA, HBD, and Flex were calculated; candidate molecules that met the macrocyclic physicochemical property constraints or were within acceptable ranges were retained; molecular novelty and diversity were calculated based on Morgan fingerprinting; molecular docking or target binding ability of candidate molecules was evaluated; and priority candidate molecules were screened based on multi-target properties, docking scores, and structural diversity.

[0134] Through the above screening process, this application can preferentially obtain non-peptide macrocyclic candidate molecules with macrocyclic structural features, reasonable physicochemical properties, good synthetic accessibility, and potential target binding ability from the generated molecular library. (See attached image) Figure 7d .

[0135] The non-peptide macrocyclic molecule generation method based on course reinforcement learning and multi-objective optimization provided in this application can be applied to scenarios such as artificial intelligence drug design, computer-aided drug development, construction of macrocyclic molecule virtual libraries, target-guided molecule optimization, lead compound discovery, and candidate molecule priority ranking.

[0136] Based on the same concept as the above method, this application also provides a non-peptide macrocyclic molecule generation system based on curriculum reinforcement learning and multi-objective optimization, including:

[0137] The data preparation module is configured to acquire a molecular dataset for training, and to perform standardization, deduplication, validity filtering, and screening of non-peptide macrocyclic molecules on the molecular dataset to obtain a training molecule set.

[0138] The molecular representation module is configured to convert the molecular structures in the training molecule set into string sequence representations with guaranteed chemical validity, and to construct a corresponding token vocabulary.

[0139] The pre-training module is configured to input string sequences into the autoregressive generative model for pre-training, so that the autoregressive generative model learns the statistical prior distribution, local combinatorial rules and long-range dependencies of molecular structures.

[0140] The reinforcement learning modeling module is configured to build a reinforcement learning policy network based on a pre-trained autoregressive generative model, and to model the molecular generation process as a sequential decision-making process.

[0141] The course reinforcement learning optimization module is configured to adopt a course reinforcement learning approach, introducing multiple optimization objectives in stages according to the order of task difficulty from low to high, so that the reinforcement learning strategy network gradually migrates from the general small molecule chemical space to the non-peptide macrocyclic molecule chemical space; the multiple optimization objectives include at least molecular effectiveness constraints, drug-likeness constraints, macrocyclic structure constraints, synthetic accessibility constraints, and molecular flexibility constraints.

[0142] The multi-objective reward calculation module is configured to use Pareto dominance to perform scalar evaluation of the multi-objective properties of the generated molecules and obtain reinforcement learning rewards.

[0143] The policy update module is configured to update the reinforcement learning policy network according to the reinforcement learning reward to obtain an optimized non-peptide macrocyclic molecule generation model.

[0144] The molecule generation module is configured to generate candidate non-peptide macrocyclic molecules using the optimized non-peptide macrocyclic molecule generation model.

[0145] This application can be implemented on computer devices, including but not limited to servers, workstations, GPU computing platforms, or cloud computing platforms. By inputting molecular training data and target optimization conditions, a generative model can be trained and output candidate non-peptide macrocyclic molecules. Therefore, this invention has clear industrial application value and feasibility.

[0146] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A method for generating non-peptide macrocyclic molecules based on curriculum reinforcement learning and multi-objective optimization, characterized in that... Includes the following steps: S1, Obtain the molecular dataset for training, and perform standardization, deduplication, validity filtering, and screening of non-peptide macrocyclic molecules on the molecular dataset to obtain the training molecule set; S2, convert the molecular structures in the training molecule set into string sequence representations with guaranteed chemical validity, and construct the corresponding token vocabulary; S3, input the string sequence into the autoregressive generation model for pre-training, so that the autoregressive generation model learns the statistical prior distribution, local combination rules and long-range dependencies of molecular structures; S4, a reinforcement learning policy network is constructed based on the pre-trained autoregressive generative model to model the molecular generation process as a sequential decision-making process; S5. A course-based reinforcement learning approach is adopted, introducing multiple optimization objectives in stages according to the order of task difficulty from low to high, so that the reinforcement learning strategy network gradually migrates from the general small molecule chemical space to the non-peptide macrocyclic molecule chemical space; the multiple optimization objectives include at least molecular effectiveness constraints, drug-likeness constraints, macrocyclic structure constraints, synthetic accessibility constraints, and molecular flexibility constraints. S6 utilizes Pareto dominance to perform scalar evaluation of the multi-objective properties of the generated molecules and obtains reinforcement learning rewards. S7. Update the reinforcement learning policy network according to the reinforcement learning reward to obtain the optimized non-peptide macrocyclic molecule generation model; S8. Candidate non-peptide macrocyclic molecules are generated using the optimized non-peptide macrocyclic molecule generation model.

2. The method according to claim 1, characterized in that, The string sequence with chemical validity guarantee mentioned in step S2 is represented as a SELFIES sequence representation; the SELFIES sequence representation is generated stepwise through an autoregressive method, and the generation probability of each token is determined by the conditions of the preceding token sequence.

3. The method according to claim 1 or 2, characterized in that, The autoregressive generative model described in step S3 is a causal language model based on the Transformer decoder structure, including a token embedding layer, a position encoding layer, a multi-head self-attention module, a feedforward neural network module, a residual connection module, a layer normalization module, and a probability output layer.

4. The method according to claim 1, characterized in that, In step S4, the reinforcement learning policy network adds a value function head to the pre-trained autoregressive generative model to form a joint policy-value network structure; in the sequential decision-making process, the state is the currently generated partial string sequence, and the action is to select the next token from the token vocabulary.

5. The method according to claim 1, characterized in that, The course reinforcement learning described in step S5 includes the following consecutive training phases: The first stage, the molecular effectiveness optimization stage, introduces a molecular effectiveness reward to guide the reinforcement learning strategy network to generate effective molecules that can be correctly resolved. The second stage, the basic drug-likeness optimization stage: while maintaining molecular effectiveness, introduces drug-likeness-related rewards; The third stage, the stage of enhancing drug-likeness and preliminary structural regulation: further optimizing drug-likeness and introducing preliminary structural constraints; The fourth stage, the macrocyclic structure constraint introduction stage: explicitly introduce macrocyclic structure constraints and macrocyclic-specific physicochemical property constraints to enable the generative model to migrate to non-peptide macrocyclic molecular regions; The fifth stage is the multi-objective equilibrium optimization stage: jointly optimizing drug-like properties, synthetic accessibility, macrocyclic structure constraints, physicochemical properties, and molecular flexibility to achieve multi-objective equilibrium within the non-peptide macrocyclic chemical space.

6. The method according to claim 1, characterized in that, The method for determining the Pareto dominance relationship in step S6 is as follows: for the current generated molecule and the historical generated molecule, if the current generated molecule is not inferior to the historical generated molecule in all targets, and is superior to the historical generated molecule in at least one target, then it is determined that the current generated molecule Pareto dominates the historical generated molecule.

7. The method according to claim 6, characterized in that, The reinforcement learning reward mentioned in step S6 is calculated as follows: the number of historically generated molecules dominated by the current generated molecule, Nwin, and the number of historically generated molecules dominated by the current generated molecule, Nloss, are counted, and the Pareto reward RPareto=(Nwin-Nloss) / (Nwin+Nloss+ε) is calculated, where ε is a constant to prevent the denominator from being zero.

8. The method according to claim 1, characterized in that, In step S7, the Proximal Policy Optimization (PPO) algorithm is used to update the reinforcement learning policy network, and the deviation between the current policy and the pre-trained reference policy is constrained by the KL divergence regularization term.

9. The method according to claim 1 or 8, characterized in that, Step S8 also includes performing property calculations, molecular diversity analysis, and target binding capacity assessment on the candidate non-peptide macrocyclic molecules.

10. A non-peptide macrocyclic molecule generation system based on curriculum reinforcement learning and multi-objective optimization, characterized in that, include: The data preparation module is configured to acquire a molecular dataset for training, and to perform standardization, deduplication, validity filtering, and screening of non-peptide macrocyclic molecules on the molecular dataset to obtain a training molecule set. The molecular representation module is configured to convert the molecular structures in the training molecule set into string sequence representations with guaranteed chemical validity, and to construct a corresponding token vocabulary. The pre-training module is configured to input string sequences into the autoregressive generative model for pre-training, so that the autoregressive generative model learns the statistical prior distribution, local combinatorial rules and long-range dependencies of molecular structures. The reinforcement learning modeling module is configured to build a reinforcement learning policy network based on a pre-trained autoregressive generative model, and to model the molecular generation process as a sequential decision-making process. The course reinforcement learning optimization module is configured to adopt a course reinforcement learning approach, introducing multiple optimization objectives in stages according to the order of task difficulty from low to high, so that the reinforcement learning strategy network gradually migrates from the general small molecule chemical space to the non-peptide macrocyclic molecule chemical space; the multiple optimization objectives include at least molecular effectiveness constraints, drug-likeness constraints, macrocyclic structure constraints, synthetic accessibility constraints, and molecular flexibility constraints. The multi-objective reward calculation module is configured to use Pareto dominance to perform scalar evaluation of the multi-objective properties of the generated molecules and obtain reinforcement learning rewards. The policy update module is configured to update the reinforcement learning policy network according to the reinforcement learning reward to obtain an optimized non-peptide macrocyclic molecule generation model. The molecule generation module is configured to generate candidate non-peptide macrocyclic molecules using the optimized non-peptide macrocyclic molecule generation model.

Images