Computer-aided drug dosage optimization method

Abstract

The invention discloses a computer-aided drug dosage optimization method. The method comprises the following steps: constructing and optimizing a QSP model of a drug and a patient; defining a state space and an action space, and initializing a bidirectional model; performing bidirectional deduction on the trained bidirectional model to obtain a virtual track; forming integrated data based on the real data, the output of the QSP model and the virtual track; constructing a learning relationship between the hierarchies; a learner and a guider are trained based on the integrated data and a PPO algorithm. According to the application, PBPK / PD modeling of the medicine in bodies of different crowds is carried out through a quantitative systematic pharmacology (QSP) model, and the physiological status of the patient after taking the medicine is calculated and predicted through a formula. According to the method, the PPO algorithm is used for strategy optimization, the amplitude of strategy updating is limited, the problem that strategy updating is too large in a traditional strategy gradient method is avoided, and the efficiency of the algorithm is improved while the strategy updating stability is guaranteed.

Description

Technical Field

[0001] The present application belongs to the technical field of precise drug administration clinical decision making, and specifically relates to a computer-assisted drug dosage optimization method. Background Art

[0002] In recent years, drug dose optimization methods have gradually shifted from in vivo experiments to in vitro experiments, which can be divided into model-driven methods, such as physiological pharmacokinetic / pharmacodynamic models (PBPK / PD), and data-driven methods, such as reinforcement learning (RL) algorithms. PBPK models usually include detailed descriptions of multiple organs and tissues, and use physiological parameters such as volume, blood flow, and drug-tissue partition coefficient for modeling. PD models describe the effects of drugs in target tissues or organs, that is, the relationship between drug concentration and efficacy, usually including the drug's mechanism of action, dose-response relationship, and time-response relationship. Combining PBPK and PD models can more comprehensively predict the behavior and effects of drugs in vivo, thereby optimizing drug dose design, improving therapeutic effects, and reducing side effects. RL algorithms can select the optimal solution in a complex environment by learning and optimizing decision-making strategies, and are suitable for individualized dose optimization.

[0003] Although PBPK / PD models have been widely used in drug development, there are still challenges in relying solely on PBPK / PD models for dose optimization, such as high model complexity, difficulty in extrapolation, and physiological parameters that vary from person to person and are difficult to accurately estimate. The applicability and accuracy of the model under different individuals and conditions need to be further verified. On the other hand, although RL algorithms have achieved remarkable success in the fields of games and robot control, their application in drug dose optimization is still in its early stages. The main challenges include: the lack of medical data makes it difficult to support the high requirements of reinforcement learning for data quantity and quality, and it is difficult to balance the scope of strategy exploration and the stability of the optimization process.

[0004] In summary, existing drug dosage optimization methods are difficult to predict the optimal dosage that suits the different physiological conditions of different patients using limited medical data. In the medical field, inventing a method that combines the advantages of the QSP model and the reinforcement learning algorithm can maximize the role of technology by leveraging its strengths and avoiding its weaknesses, improve the model's prediction accuracy and generalization ability, assist in diagnosis and treatment, and help medical staff provide more effective dosing plans. Summary of the invention

[0005] The present application provides a computer-assisted drug dose optimization method to solve the problems of data shortage and excessive strategy updates that are out of touch with clinical practice in drug dose optimization, and to improve data utilization and the accuracy and generalization ability of the model.

[0006] In order to solve the above technical problems, a technical solution adopted in this application is: a computer-assisted drug dosage optimization method, comprising the following steps:

[0007] Based on the patient's physiological data and drug information, build and optimize the QSP model of drugs and patient populations;

[0008] Define the state space and action space, and initialize the bidirectional model, where the bidirectional model includes a forward model and a reverse model;

[0009] Train the bidirectional model, and perform bidirectional deduction on the trained bidirectional model to obtain a virtual trajectory;

[0010] Based on real data, the output of QSP model and virtual trajectory, integrated data is constructed;

[0011] Based on the learning goals and level division of learners and guides, build learning relationships between levels;

[0012] Based on the integrated data and PPO algorithm, the learner and the guide are trained.

[0013] Furthermore, based on the patient's physiological data and drug information, the QSP model of drugs and patient populations is constructed and optimized, including:

[0014] Define the research objectives and collect experimental data, select the type of model and set the dosing regimen;

[0015] Based on the model type, create simulations and run QSP models, and analyze simulation results;

[0016] Verify and optimize the QSP model based on simulation results;

[0017] Based on the QSP model, the changes in index parameters of different patients after medication are predicted.

[0018] Furthermore, the state space and action space are defined, and the method of initializing the bidirectional model includes:

[0019] Physiological data and drug information are summarized, the number of optimal clusters is determined by the silhouette coefficient and placed in the state space, and the different doses of each administration are recorded and placed in the action space;

[0020] The forward model is a prediction model. The forward model receives the current state and the action taken as input and predicts the next state and reward.

[0021] The reverse model is a reverse dynamic model, which is used to predict given the current state and action;

[0022] The forward model and the reverse model are set as full network link models, and the average value of the dose is selected as the mean of the Gaussian distribution when the network weight is initialized. The standard deviation of the dose is selected to set the standard deviation, and the softmax activation function is used to output the decision of the dosage.

[0023] Set the policy network of the forward model and the policy network of the reverse model as a full network link model, and initialize the input layer, hidden layer, and output layer.

[0024] Furthermore, a method for training a bidirectional model and bidirectionally deducing the trained bidirectional model to obtain a virtual trajectory includes:

[0025] The policy network of the forward model and the policy network of the reverse model are updated through maximum likelihood estimation to make the output action probability distribution consistent with the true action distribution;

[0026] Based on minimizing the loss function and the gradient descent algorithm, the policy parameters of the policy network of the forward model and the policy network of the reverse model are updated.

[0027] Furthermore, based on the learning objectives and level division of learners and guides, methods for constructing learning relationships between levels include:

[0028] Define the facilitator as the high-level layer corresponding to the long-term treatment goals, and the learner as the low-level layer corresponding to the short-term goals;

[0029] Based on the disease treatment process stratification corresponding to the patient's clinical test indicators, the instructor sets short-term goals and passes them to the learner, so that the learner resets the learning reward function according to the short-term goals and achieves the short-term goals;

[0030] The learning reward function is fed back to the leader to obtain the leader's guidance reward function.

[0031] Furthermore, based on the integrated data PPO algorithm, the method of training the learner and the guide includes:

[0032] Get the loss function of the value network / strategy network in the learner and the guide;

[0033] Based on the learner's loss function, obtain the first weight of the value network / strategy network in the learner;

[0034] Based on the first weight and the initial first weight, updating the first weight of the value network / strategy network in the learner;

[0035] Based on the loss function of the leader, obtain the second weight of the value network / strategy network in the leader;

[0036] Based on the second weight and the initial second weight, the second weight of the value network / strategy network in the leader is updated.

[0037] The beneficial effects of this application are:

[0038] 1. This application uses the quantitative systems pharmacology (QSP) model to model the PBPK / PD of drugs in different populations and predicts the physiological state of patients after taking the medicine through formula calculation, which is then input into RL as the main data source, forming the main part of RL strategy learning, helping the intelligent agent to learn the general law of changes in physiological indicators of the medication population with dosage, and solving the problem of low utilization rate of reinforcement learning data.

[0039] 2. This application inputs data on changes in physiological indicators of real patients after taking medication over time and uses two-way deduction to amplify the data. It takes advantage of the data-driven model's ability to capture potential correlations between data and retains detailed features of drug efficacy affected by covariates.

[0040] 3. This application stratifies patients according to the disease treatment process corresponding to their clinical test indicators, which is more in line with clinical treatment practice and enhances the accuracy of model prediction.

[0041] 4. This application uses the PPO algorithm for policy optimization, limits the amplitude of policy updates, avoids the problem of excessive policy updates in traditional policy gradient methods, ensures the stability of policy updates, and improves the efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flowchart of an embodiment of a computer-assisted drug dosage optimization method of the present application;

[0043] Figure 2 It is a structural framework diagram of an embodiment of a computer-assisted drug dosage optimization method of the present application;

[0044] Figure 3 for Figure 1 A schematic diagram of a flow chart of an embodiment of step S1 in FIG.

[0045] Figure 4 for Figure 1 A schematic diagram of a flow chart of an embodiment of step S2 in FIG.

[0046] Figure 5 for Figure 1 A schematic diagram of a flow chart of an embodiment of step S3;

[0047] Figure 6 for Figure 1 A schematic diagram of a flow chart of an embodiment of step S5;

[0048] Figure 7 for Figure 1A schematic diagram of a flow chart of an embodiment of step S6;

[0049] Figure 8 for Figure 1 FIG. 1 is a structural block diagram of an embodiment of step S6 in FIG. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0052] See also Figure 1-2 , Figure 1 : is a flow chart of an embodiment of a computer-assisted drug dosage optimization method of the present application, the method comprising the following steps:

[0053] Step S1. Based on the patient's physiological data and drug information, the QSP model of drugs and patient population is constructed and optimized.

[0054] Specifically, the QSP model is composed of a physiological pharmacokinetic model (PBPK) and a pharmacodynamic model (PD). The PBPK model data is extracted through the PD model, and the formula connecting blood drug concentration and effect is as follows:

[0055]

[0056] Among them, the relevant parameters include E min (effect baseline without drug), E max (theoretical maximum value of drug effect), EC 50 (the concentration of the drug at which it achieves half of its maximal effect) and the Hill coefficient n.

[0057] The QSP model predicts the behavior and effects of drugs in the body by simulating the absorption, distribution, metabolism and excretion of drugs in the human body to generate more data that conforms to the real situation. The QSP model and bidirectional deduction are used to expand the data, and deep reinforcement learning is trained based on the disease treatment process layered according to the patient's clinical test indicators.

[0058] See also Figure 3 , Figure 3 for Figure 1 The flowchart of step S1 in the embodiment is as follows; step S1 comprises:

[0059] Step S11. Clarify the research objectives and collect experimental data, select the type of model and set the dosing regimen.

[0060] Specifically, predict the changes in the blood concentration of aspirin in different patient groups over time; collect experimental data. Collect the physical and chemical information of the drug aspirin, collect relevant population data, and related physiological parameters.

[0061] Among them, the physicochemical information of aspirin includes lipophilicity, the fraction of drugs not bound to plasma proteins or other blood components, relative molecular mass, dissociation constant, solubility, permeability, and the logarithm of the oil-water partition coefficient.

[0062] The relevant population data include basic data such as age, gender ratio, and body weight. The relevant physiological parameters include liver and kidney weight, liver total clearance rate, and kidney plasma clearance rate.

[0063] The specific model type can be selected from the physiological pharmacokinetic model (PBPK) and the pharmacodynamic model (PD). In this embodiment, the PBPK model is selected in PK-sim and the parameters of the model are set. The relevant parameters of the drug aspirin mentioned above are input. The metabolic pathway of aspirin is set, the main enzyme (CYP2C9) in the aspirin metabolism process is added, and the initial concentration and clearance rate of CYP2C9 are input by searching for data.

[0064] Enter the dosage and mode of administration. The mode of administration in this model is oral administration, single administration. Then define the time interval and number of administrations. In this example, the time interval for administration is 24 hours, and the number of administrations is 30 times.

[0065] In the Individuals section, define the characteristics of individuals within the selected age range, including age, weight, height, and gender ratio.

[0066] Step S12: Based on the type of model, create a simulation and run the QSP model, and analyze the results.

[0067] Specifically, add the above settings to the simulation. PK-sim will simulate the pharmacokinetic process of the drug in the body based on the input parameters and dosing regimen.

[0068] After the simulation, the concentration-time curve of the drug in the body can be obtained.

[0069] Step S13: Based on the simulation results, verify and optimize the QSP model.

[0070] Specifically, the experimental observation data or the data found in the literature are compared with the data predicted by the model, and the consistency of the observed data (Observed Data) and the predicted data (Predicted Data) is compared to verify the model. The literature is consulted to find the parameters and physiological indicators that have an impact on or are related to the pharmacokinetic process of aspirin in the body, and they are added to the "Parameter Identification" section for parameter optimization, and then the optimized value "Transfer to Simulation" is applied to the model. Among them, the parameters and physiological indicators that have an impact on or are related to the pharmacokinetic process of aspirin in the body include lipophilicity, permeability, total liver clearance, renal plasma clearance, dosage, and body weight.

[0071] In the population section, create a population based on the age range of the corresponding individuals, build a population PBPK model, and verify the absorption distribution of the drug in the population. Create a sensitivity analysis and add all parameters to the sensitivity analysis to determine which parameters have a greater impact on the output of the QSP model. By inputting a large amount of clinical data or experimental data, the QSP model is continuously calibrated and optimized to make the QSP model more accurate.

[0072] Step S14: Based on the QSP model, predict the changes in index parameters of different patients after taking the medicine.

[0073] Specifically, predict the changes in platelet aggregation rate and other parameters in different patient groups after oral administration of low-dose aspirin

[0074] In this embodiment, step S14 includes:

[0075] Step S141. Based on the relationship between platelet aggregation rate, TXA2 and COX-1 concentration, obtain the COX-1 concentration change.

[0076] Specifically, the platelet aggregation rate change that needs to be obtained is quantitatively converted into the COX-1 concentration change, and then the COX-1 concentration change is obtained through the PK and PD models. The PK model built in PK-sim is exported in .json format, imported into MATLAB, and the read binary data is converted into a character vector to obtain a data structure that MATLAB can process.

[0077] Among them, the quantitative conversion is based on the formula:

[0078] Agg = β*α*cox_1;

[0079] Among them, α and β are proportional constants obtained from experimental data.

[0080] Step S142: Select a PD model based on relevant parameter data.

[0081] Specifically, the required relevant parameter values ​​are extracted from the processed data. According to the way and result of aspirin's pharmacodynamic effect in the body, a suitable PD model is selected. In this embodiment, the indirect connection S-type E max Model.

[0082] Step S143. Based on the extracted blood drug concentration value and the PD model, set initial guess values ​​for relevant parameters in the PD model;

[0083] Specifically, E is defined by adding the extracted blood drug concentration value to the corresponding effect value. max Model, set initial guess values ​​for relevant parameters in the model.

[0084] Step S143. Set the boundaries of relevant parameters, perform model fitting through the lsqcurvefit function, and output the parameter values ​​obtained by fitting.

[0085] Specifically, set the parameter boundaries to ensure that the parameter values ​​are within a reasonable range during the fitting process. Use the lsqcurvefit function in MATLAB to fit the model, minimize the square difference between the model prediction value and the actual observation value, and output the parameter value obtained by fitting.

[0086] Step S144: Based on the parameter values ​​obtained by output fitting and related parameter values, obtain the change of COX-1 activity with blood drug concentration.

[0087] Specifically, the parameter value obtained by the output fitting is added to the value of the blood drug concentration extracted in step S133 to establish the S-type E max The model shows that COX-1 activity changes with blood drug concentration. The specific formula is:

[0088]

[0089] Then, combined with the law of COX-1 activity changing over time, the curve of COX-1 activity changing over time was converted. The curve of COX-1 activity changing over time was drawn, and the X-axis, Y-axis labels and image title were set, and the grid was enabled for easy observation. The peak and minimum values ​​in each dosing cycle were calculated, and the peak and minimum values ​​were marked and displayed in the image.

[0090] Step S2: define the state space and action space, and initialize the bidirectional model, wherein the bidirectional model includes a forward model and a reverse model.

[0091] Specifically, define the state space and action space; initialize the forward model, the reverse model, the policy network of the forward model, and the policy network of the reverse model.

[0092] See also Figure 4 , Figure 4 for Figure 1 The flowchart of step S2 in the embodiment of FIG. 1 is as follows; step S2 includes:

[0093] Step S21. Summarize the physiological data and drug information, determine the number of optimal clusters through the silhouette coefficient and put them into the state space, record the different doses of each medication and put them into the action space.

[0094] Specifically, the patient's height, weight, age, gender, blood sugar, platelet aggregation rate, and blood lipid indicators are fixed between [0, 1] using the maximum and minimum normalization method, and clustered using a hierarchical clustering algorithm. The optimal number of clusters is determined by the silhouette coefficient and placed in the state space; the different doses of the drug each time are recorded and placed in the action space.

[0095] Step S22. The forward model is a prediction model. The forward model receives the current state and the action taken as input, and predicts the next state and reward;

[0096] Specifically, the forward model refers to a prediction model that receives the current state and the action taken as input and predicts the next state and possible rewards. It is built using a set of parameters parameterized by multiple layers of neural networks. Each probabilistic neural network outputs a Gaussian distribution with a diagonal covariance, representing the distribution of the next state:

[0097] p φ’ (S t+1 |S t,a )=N(μ φ’ (S t ,a t ),∑ φ’ S t ,a t ));

[0098] Among them, p φ , is the forward model, S t is the current state, a t Current action, S t+1 is the next state.

[0099] The forward model loss function is defined as the actual state S t+1 With the model prediction state μ θ (S t , a t ), plus the logarithmic determinant of the covariance matrix, the specific formula is:

[0100]

[0101] Among them, θ is the neural network parameter, L f(θ) is the loss value.

[0102] Step S23. The reverse model is a reverse dynamic model used to predict a given current state and action.

[0103] Specifically, the reverse model is established, similar to the forward model, using a multi-layer neural network with parameters θ′, and outputting a Gaussian prediction. The reverse model is:

[0104] q φ” (S t |S t+1 , a t )=N(μ φ” (S t+1 , a t ),∑ φ (S t+1 , a t ));

[0105] Among them, q φ″ For the reverse model, S t is the current state, a t Current action, S t+1 is the next state.

[0106] Where S t is the current state, a t Current action, S t+1 is the next state.

[0107] The reverse loss function is defined as the actual state S at time t-1 t+1 And the model predicts the state μ θ (S t , a t-1 ), plus the logarithmic determinant of the covariance matrix, the specific formula is:

[0108]

[0109] Among them, θ' is the neural network parameter, L f (θ') is the loss value.

[0110] Step S24. The forward model and the reverse model are set as full network link models, and when initializing the network weights, the average value of the dose is selected as the mean of the Gaussian distribution, the standard deviation of the dose is selected to set the standard deviation, and the softmax activation function is used to output the decision of the dosage;

[0111] Step S25. Set the policy network of the forward model and the policy network of the reverse model as a full network link model, and initialize the input layer, hidden layer and output layer.

[0112] Among them, the policy network selects the drug dosage according to the current state, which can be expressed as the policy network of the forward model and reverse model strategies

[0113] Step S3: Train the bidirectional model and perform bidirectional deduction on the trained bidirectional model to obtain a virtual trajectory.

[0114] For details, see Figure 5 , Figure 5 for Figure 1 Schematic diagram of the process of an embodiment of step S3; step S3 includes:

[0115] Step S31. The policy network of the forward model and the policy network of the reverse model are updated by maximum likelihood estimation so that the output action probability distribution is consistent with the real action distribution. The formula of maximum likelihood estimation is as follows:

[0116]

[0117]

[0118] The parameters of the forward policy network and the reverse policy network are updated using the gradient ascent algorithm:

[0119]

[0120]

[0121] Among them, φ' is the forward policy neural network parameter, L(φ') is the forward policy network loss value, and α is the learning rate; φ" is the reverse policy neural network parameter, L(φ") is the reverse policy network loss value, and α is the learning rate.

[0122] Step S32: Based on minimizing the loss function and the gradient descent algorithm, update the policy parameters of the policy network of the forward model and the policy network of the reverse model.

[0123] Specifically, the parameters of the bidirectional model are updated using the gradient descent algorithm:

[0124]

[0125]

[0126] Among them, θ is the forward model neural network parameter, L(θ) is the loss value, and α is the learning rate.

[0127] Where θ' is the reverse model neural network parameter, L(θ') reverse model is the loss value, and α is the learning rate.

[0128] Step S4: Based on the real data, the output of the QSP model and the virtual trajectory, the integrated data is constructed.

[0129] Specifically, the output of the QSP model refers to converting the intrinsic experimental data of the patient predicted by the QSP model into clinical test data through calculation, which helps the hierarchical deep reinforcement learning to explore the relationship between drug dosage and clinical test data. The integration of real data, QSP model output and virtual trajectory together constitutes integrated data.

[0130] Step S5. Based on the learning objectives and level divisions of the learners and guides, construct learning relationships between levels.

[0131] For details, see Figure 6 , Figure 6 for Figure 1 Schematic diagram of the process of step S5 in an embodiment; step S5 includes:

[0132] Step S51. Define the guide as the high-level layer corresponding to the long-term treatment goals, and the learner as the low-level layer corresponding to the short-term goals.

[0133] Specifically, the facilitator is at a high level, corresponding to long-term treatment goals, and the learner is at a low level, corresponding to short-term goals.

[0134] Among them, the guidance reward function of the guide is:

[0135]

[0136] in It is a physiological indicator in a healthy state. is the physiological index in the current state, and the parameter α is selected as the weight coefficient; the learner's learning reward function is set as:

[0137]

[0138] in It is a physiological indicator in a healthy state. is the physiological index in the current state; parameter a is the weight coefficient.

[0139] Step S52. Based on the disease treatment process stratification corresponding to the patient's clinical test indicators, the instructor formulates short-term goals and passes them to the learners, so that the learners reset the learning reward function according to the short-term goals to achieve the short-term goals.

[0140] Specifically, the instructor stratifies the disease treatment process according to the patient's clinical test indicators and formulates short-term goals and passes them to the learner. The learner resets the learning reward function according to the short-term goals to achieve the short-term goals.

[0141] Step S53: Feedback the learning reward function to the leader to obtain the leader's guidance reward function.

[0142] Specifically, the results of the short-term goals are fed back to the guide, and the guide evaluates the learning reward function of the short-term goals for the long-term goals through feedback.

[0143] In one embodiment, data on cardiovascular events in real data and virtual data derived from a two-way model are selected for statistics, and the mean values ​​of platelet, blood sugar, and blood lipid indicators are calculated.

[0144]

[0145] in is the platelet aggregation rate in the state of cardiovascular events, is the platelet aggregation rate in the current state; Blood glucose in the state of cardiovascular events, is the blood sugar in the current state; Blood lipids in the event of cardiovascular events, This embodiment selects parameter α as the weight coefficient of platelet aggregation rate, selects β as the weight coefficient of blood glucose, and selects λ as the weight coefficient of blood lipid.

[0146] The reward function is set using real data, QSP model output, and integrated data derived from the two-way model.

[0147]

[0148] in, is the platelet aggregation rate under the target state, is the platelet aggregation rate in the current state; a is the adjustment parameter.

[0149] The neural networks used by the guide and learner in this embodiment are all fully connected neural networks. The network weights are initialized by selecting the average value of aspirin dose as the mean of the Gaussian distribution, and the standard deviation of aspirin dose as the standard deviation.

[0150] (1) Initialize the guide:

[0151] Strategy layer:

[0152] The platelet aggregation rate is divided into five levels: too high, relatively high, normal, relatively low, and too low, which are recorded as C+, B+, A, B-, and C- respectively.

[0153] Value Network and Target Value Network:

[0154] Input layer: 7 neurons, hidden layer activated by ReLU; Output layer: 1 neuron, representing the expected reward of the current state. This value can be used to evaluate the goodness of taking a specific action in this state.

[0155] (2) Initialize the learner:

[0156] Policy Network:

[0157] Input layer: 7 neurons; hidden layer through ReLU activation function; output layer: 5 neurons using Softmax activation function output: 0mg, 75mg, 100mg, 125mg, 150mg.

[0158] Value network and target network:

[0159] Input layer: 7 neurons; hidden layer activated by ReLU; output layer: 1 neuron, using a linear activation function, represents the expected reward of the current state. This value can be used to evaluate the goodness of taking a specific action in this state.

[0160] According to the five levels of platelet aggregation rate, the parameters of the reward function are set:

[0161]

[0162] The guide divides the treatment stages of the disease into levels by setting short-term goals to achieve the purpose of stratified treatment. The guide passes the short-term goals to the learner, and the learner resets the reward function based on the short-term goals. When the learner completes the short-term goals, the guide evaluates the reward value of the short-term goals for the long-term goals through r'.

[0163] Step S6: Based on the integrated data PPO algorithm, train the learner and the guide.

[0164] For details, see Figure 7-8 , Figure 7 for Figure 1 Schematic diagram of the process of an embodiment of step S6; step S6 includes:

[0165] Step S61. Obtain the loss function of the value network / strategy network in the learner and the guide.

[0166] Specifically, the value network loss values ​​of the learner and the guide are as follows:

[0167]

[0168] in, represents the value network loss function; Loss represents the loss value of the value network; γ represents the discount coefficient; t represents time, t' represents the time after time t; r represents the reward obtained through the reward function in the preset batch of interaction data; T represents the preset batch number of interaction data; ∑ t'>t γ (t′-t) r t' Represents a reward that decays over time.

[0169] The strategy network loss functions of the learner and the guide are as follows:

[0170]

[0171] Among them, j(ω) represents the policy network loss function; Loss j(ω) represents the loss value of the policy network; t represents time; A t represents the advantage function; π ω represents the new strategy; π old represents the old strategy; s is the current state of the agent at the current time t, and a is the action taken by the agent (i.e., the guide and learner) in the current state s. π ω (a t |s t ) represents the new strategy π ω The probability of the action taken by the agent in the current state s under the distribution; π old (a t |s t ) represents the old policy π old The probability of the action taken by the agent in the current state s under the distribution; is the probability ratio of actions under the new and old strategy distributions; ε represents a fixed parameter, and clip means limiting the probability ratio between 1-ε and 1+ε.

[0172] Step S62: Based on the loss function of the learner, obtain the first weight of the value network / strategy network in the learner.

[0173] Specifically, based on the learner's loss value, the partial derivative of the value network is calculated, and the learning rate is used to obtain the first weight of the value;

[0174] Step S63: Based on the first weight and the initial first weight, update the first weight of the value network / strategy network in the learner.

[0175] Specifically, the first weight is added to the initial first weight to update the new first weight of the value network, and the target network is updated by soft updating.

[0176] The method of weight optimization is: use the gradient ascent algorithm to optimize the first weight of the value network, and the formula is:

[0177]

[0178] Among them, θ is the first weight parameter of the current value network, α is the learning rate, and L(θ) is the calculated loss value. It is to find the partial derivative of the first weight parameter θ of the value network, and finally complete the update of the first weight parameter of the value network.

[0179] The soft update target networks are as follows:

[0180] θ'←τθ'+(1-τ)θ;

[0181] Among them, θ' is the target network parameter, τ is the soft update parameter, and θ is the first weight parameter of the value network, which finally completes the update of the target value network parameters.

[0182] Step S64: Based on the loss function of the leader, obtain the second weight of the value network / strategy network in the leader.

[0183] Specifically, based on the loss value of the guide, the partial derivative of the value network is calculated, and the learning rate is used to obtain the first weight of the value.

[0184] Step S65: Based on the second weight and the initial second weight, update the second weight of the value network / strategy network in the leader.

[0185] Specifically, the first weight is added to the initial first weight to update the new first weight of the value network, and the target network is updated by soft updating.

[0186] The second weight parameter of the policy network is optimized using the gradient ascent algorithm using the following formula:

[0187]

[0188] Among them, θ is the second weight parameter of the current policy network, α is the learning rate, and L(θ) is the calculated loss value. It is to find the partial derivative of the second weight parameter θ of the strategy network, and finally complete the update of the second weight parameter of the strategy network.

[0189] This application uses a quantitative systems pharmacology (QSP) model to model the PBPK / PD of drugs in different populations and predicts the physiological state of patients after taking the medicine through formula calculations, which is then input into RL as the main data source, forming the main part of RL strategy learning, helping the intelligent agent to learn the general rules of changes in physiological indicators of the medication population with dosage, and solving the problem of low utilization of reinforcement learning data.

[0190] This application inputs data on how real patients’ physiological indicators change over time after taking medication and uses two-way deduction to amplify the data, taking advantage of the data-driven model’s ability to capture potential correlations between data and retaining detailed features of drug efficacy affected by covariates. This application stratifies patients according to the disease treatment process corresponding to their clinical test indicators, which is more in line with clinical treatment practice and enhances the accuracy of model predictions. This application uses the PPO algorithm for strategy optimization, limits the magnitude of strategy updates, avoids the problem of excessive strategy updates in traditional policy gradient methods, ensures the stability of strategy updates, and improves the efficiency of the algorithm.

[0191] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A computer-assisted drug dosage optimization method, characterized in that: The following steps are involved: Based on the patient's physiological data and drug information, build and optimize the QSP model of drugs and patient populations; Define a state space and an action space, and initialize a bidirectional model, wherein the bidirectional model includes a forward model and a reverse model; Training the bidirectional model, and bidirectionally deducing the trained bidirectional model to obtain a virtual trajectory; Based on real data, the output of the QSP model and the virtual trajectory, forming integrated data; Based on the learning goals and level division of learners and guides, build learning relationships between levels; The learner and the guide are trained based on the integrated data and the PPO algorithm.

2. The method according to claim 1, characterized in that The method for constructing and optimizing the QSP model of drugs and patient population based on the physiological data and drug information of the patient includes: Define the research objectives and collect experimental data, select the type of model and set the dosing regimen; Based on the type of model described, create simulations and run the QSP model, and analyze the simulation results; Based on the simulation results, verify and optimize the QSP model; Based on the QSP model, the changes in index parameters of different patients after taking the medicine are predicted.

3. The method according to claim 1, characterized in that: The method of defining the state space and the action space and initializing the bidirectional model includes: Summarizing the physiological data and the drug information, determining the number of optimal clusters by using the silhouette coefficient and placing them in the state space, and recording different doses of each administration and placing them in the action space; The forward model is a prediction model that receives the current state and the action taken as input and predicts the next state and reward; The reverse model is a reverse dynamic model, which is used to predict a given current state and action; The forward model and the reverse model are set as full network link models, and when initializing the network weights, the average value of the dose is selected as the mean of the Gaussian distribution, the standard deviation of the dose is selected to set the standard deviation, and a softmax activation function is used to output the decision of the administration dose; The policy network of the forward model and the policy network of the reverse model are set as a full network link model, and the input layer, hidden layer and output layer are initialized.

4. The method according to claim 3, characterized in that The method of training the bidirectional model and performing bidirectional deduction on the trained bidirectional model to obtain a virtual trajectory includes: The policy network of the forward model and the policy network of the reverse model are updated by maximum likelihood estimation so that the output action probability distribution is consistent with the true action distribution; Based on minimizing the loss function and the gradient descent algorithm, the policy parameters of the policy network of the forward model and the policy network of the reverse model are updated.

5. The method according to claim 1, characterized in that The method of constructing a learning relationship between levels based on the learning objectives and level division of the learners and guides includes: The guide is defined as a high-level layer corresponding to long-term treatment goals, and the learner is defined as a low-level layer corresponding to short-term goals; Based on the disease treatment process stratification corresponding to the patient's clinical detection indicators, the instructor formulates short-term goals and transmits them to the learner, so that the learner resets the learning reward function according to the short-term goals to achieve the short-term goals; The learning reward function is fed back to the guide to obtain the guide reward function of the guide.

6. The method according to claim 5, characterized in that The method for training learners and guides based on the integrated data PPO algorithm includes: Obtaining the loss function of the value network / strategy network in the learner and the guide; Based on the loss function of the learner, obtaining a first weight of a value network / strategy network in the learner; Based on the first weight and the initial first weight, updating the first weight of the value network / strategy network in the learner; Based on the loss function of the leader, obtaining a second weight of the value network / strategy network in the leader; Based on the second weight and the initial second weight, the second weight of the value network / strategy network in the leader is updated.

Images