A blood sugar closed-loop control method and device

Abstract

The present invention provides a blood sugar closed-loop control method and device. The method includes: constructing a deep deterministic policy gradient model for predicting the patient's initial insulin dose; constructing an auxiliary learning model for optimizing the initial insulin dose, thereby obtaining an optimized insulin correction dose; the insulin pump infuses according to the insulin correction dose, and obtains the patient's blood sugar data at the next time step; repeatedly performing the above steps to achieve automatic insulin infusion, thereby achieving closed-loop control of blood sugar; within a preset time period, the percentage of all blood sugar data of the patient within the normal range is counted to evaluate the performance of the method. The method improves the accuracy of predicting insulin and ultimately improves the effect of blood sugar control.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence medical applications, and in particular to a blood sugar closed-loop control method and device. Background Art

[0002] Diabetes mellitus (DM) is a series of metabolic disorders caused by genetic factors, immune dysfunction or other pathogenic factors that cause pancreatic islet dysfunction or even loss. Clinically, high blood sugar is the main feature. Current medical treatments cannot completely cure diabetes, but can only control and alleviate the symptoms of high blood sugar to a certain extent. The goal of diabetes treatment is to maintain blood sugar within the normal range (70-180 mg / dL) as much as possible.

[0003] The continuous glucose monitoring system (CGMS) is a system tool that can continuously monitor blood sugar fluctuations that has emerged in recent years. The advancement of CGMS technology and the development of artificial intelligence have promoted the development of artificial pancreas (AP) systems. The AP system uses CGMS to monitor the patient's glucose concentration in real time and regularly transmits the monitoring results to the blood sugar closed-loop control system. The blood sugar closed-loop control algorithm uses this information to output the insulin dose that needs to be infused. However, the current blood sugar closed-loop control algorithm has poor blood sugar control effect. Summary of the invention

[0004] The present invention provides a blood sugar closed-loop control method and device, which are used to solve the defect of poor blood sugar control in the prior art.

[0005] A blood sugar closed-loop control method, comprising:

[0006] Step 1: construct a computer simulation model of a type 1 diabetes patient; the type 1 diabetes patient model is used to simulate and output the patient's blood sugar data;

[0007] Step 2: constructing a deep deterministic policy gradient model based on the blood glucose data output by the type 1 diabetes patient model, wherein the deep deterministic policy gradient model is used to predict the initial insulin dose of the patient model;

[0008] Step 3: constructing a dosage optimization model; the dosage optimization model is used to optimize the initial insulin dosage output by the deep deterministic policy gradient model, thereby obtaining an optimized insulin correction dosage;

[0009] Step 4: determining the blood glucose data of the patient model at the next time step according to the optimized insulin correction dose;

[0010] Step 5: repeatedly obtaining the patient's blood glucose data at the next time step within a preset time period, and determining the TIR based on all the blood glucose data at the next time step within the preset time period;

[0011] Step six: determining the final insulin dosage of the patient model according to the TIR corresponding to the patient model in the preset time period.

[0012] Furthermore, in the above-mentioned closed-loop blood sugar control method, step one comprises:

[0013] Step 11: Obtain patient data; the patient data includes: three meal times and the amount of carbohydrates corresponding to each time point;

[0014] Step 12: Based on the patient data, a computer simulation model of a type 1 diabetes patient is constructed using the UVA / Padova simulator.

[0015] Furthermore, in the blood sugar closed-loop control method as described above, the deep deterministic policy gradient model in step 2 includes: an Actor network and a Critic network;

[0016] The Actor network is used to output the corresponding insulin dosage decision according to the blood sugar data output by the patient model; the Critic network is used to judge the quality of the insulin dosage decision given by the Actor network through the action-value function Q value; the two networks interact with each other to finally obtain the initial insulin dosage of the patient model.

[0017] Furthermore, in the blood glucose closed-loop control method as described above, the experience data samples generated by the interaction between the Actor network and the environment are stored in the experience replay pool; the experience data samples include: current blood glucose data St, current insulin dose at, current cumulative reward value rt, and next time step blood glucose data St+1;

[0018] The Critic network extracts some data from the experience replay pool to evaluate the quality of the Actor network;

[0019] The Actor network updates its own parameters according to the evaluation result; and uses the Actor network with updated parameters to output its corresponding insulin dosage decision.

[0020] Furthermore, in the blood sugar closed-loop control method as described above, the Actor network and the Critic network respectively include: an online network and a target network;

[0021] Through the soft update method, the parameters of the target network are slowly brought closer to the parameters of the online network.

[0022] Furthermore, in the above-mentioned closed-loop blood sugar control method, step three comprises:

[0023] Step 31: using the type 1 diabetes patient model, obtaining historical blood glucose data of the patient model;

[0024] Step 32: using the deep deterministic policy gradient model to obtain the insulin dosages corresponding to the historical blood sugar data;

[0025] Step 33: constructing a blood glucose prediction model based on the historical blood glucose data and its corresponding insulin dosage, wherein the blood glucose prediction model is used to predict the blood glucose prediction value of the patient model at the next time step;

[0026] Step 34: Based on the blood glucose prediction value of the patient model at the next time step, the current blood glucose data of the patient model, and the insulin dose corresponding to the current blood glucose data predicted by the deep deterministic policy gradient model, an auxiliary learning model is constructed, and the insulin dose output by the deep deterministic policy gradient model is optimized through the auxiliary learning model to obtain an optimized insulin correction dose.

[0027] Furthermore, in the blood glucose closed-loop control method as described above, the blood glucose prediction model includes: an input layer, a first LSTM layer, a first Dropout layer, a second LSTM layer, a second Dropout layer, a fully connected layer, and an output layer.

[0028] A blood sugar closed-loop control device, comprising:

[0029] A construction unit, used for constructing a computer simulation model of a type 1 diabetes patient; the type 1 diabetes patient model is used for simulating and outputting the patient's blood sugar data;

[0030] The construction unit is further used to construct a deep deterministic policy gradient model based on the blood glucose data output by the type 1 diabetes patient model, wherein the deep deterministic policy gradient model is used to predict the initial insulin dose of the patient model;

[0031] The construction unit is further used to construct a dosage optimization model, wherein the dosage optimization model is used to optimize the initial insulin dosage output by the deep deterministic policy gradient model, thereby obtaining an optimized insulin correction dosage;

[0032] A determination unit, used to determine the blood glucose data of the patient model at the next time step according to the optimized insulin correction dose;

[0033] An acquisition unit, used to repeatedly acquire the patient's blood sugar data for the next time step within a preset time period;

[0034] The determination unit is further configured to determine the TIR based on all blood glucose data of the next time step within the preset time period;

[0035] The determination unit is further configured to determine a final insulin dose of the patient model according to the TIR corresponding to the patient model in the preset time period.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a blood glucose closed-loop control method as described above is implemented.

[0037] The blood glucose closed-loop control method provided by the present invention predicts the patient's initial insulin dose by constructing a deep deterministic policy gradient model, and optimizes the initial insulin dose through an auxiliary learning model to obtain an optimized insulin correction dose. This method improves the accuracy of predicting insulin and ultimately improves the effect of blood glucose control. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of the blood sugar closed-loop control method provided by the present invention;

[0040] Figure 2 A schematic diagram of a blood sugar closed-loop control structure provided by the present invention;

[0041] Figure 3 A structural diagram of the blood sugar prediction model provided by the present invention;

[0042] Figure 4 The 24-hour blood sugar fluctuation graphs of the best performing adult patient 4 and the worst performing adult patient 3 in the adult group are shown in the embodiment;

[0043] Figure 5 The 24-hour blood sugar fluctuation graphs of adolescent patient 1 with the best performance and adolescent patient 5 with the worst performance in the adolescent group of the embodiment are shown in FIG. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Since the blood sugar regulation system is a complex nonlinear dynamic regulation system, there is a high degree of inter-individual and intra-individual variability, as well as uncertainty and interference related to diet, exercise and other daily events. In addition, the delayed action of insulin caused by the subcutaneous administration route, changes in the patient's insulin sensitivity and sensor hysteresis delay the action of insulin, resulting in large fluctuations in postprandial blood sugar, which increases the complexity of blood sugar control. Therefore, the present invention effectively controls the patient's blood sugar by constructing a deep deterministic strategy gradient model and a dosage optimization model.

[0046] Figure 1 The flow chart of the blood sugar closed-loop control method provided by the present invention is as follows: Figure 1 As shown, the method includes:

[0047] Step 1: construct a computer simulation model of a type 1 diabetes patient; the type 1 diabetes patient model is used to simulate and output the patient's blood sugar data.

[0048] Specifically, patient data were generated using the UVA / Padova T1D Glucose Dynamics Model, a software that simulates the human metabolic system using a dietary model of glucose-insulin dynamics and is intended to replace preclinical tests for the development and testing of T1D control algorithms. The UVA / Padova simulator allows the generation of in silico data from both ideal (without any errors) and real sensors such as GuardianRT and Dexcom. The choice of using GuardianRT sensors to generate data was made to make the simulated patient data as similar as possible to real patient data in order to take into account deviations from real sensors.

[0049] The information that needs to be specified for the virtual patient environment setting includes the carbohydrate content to be consumed in each meal and the corresponding time of the meal in a day. Each virtual patient ate three meals in the predefined scenario, consuming 40 grams, 80 grams, and 60 grams of carbohydrates at 7 am, 12 pm, and 19 pm respectively. Data samples of 6 patients in the adult group and 6 patients in the adolescent group were collected for 10 days, and three random seeds were trained using each patient's data.

[0050] Step 2: Based on the blood glucose data output by the type 1 diabetes patient model, a deep deterministic policy gradient model is constructed, and the deep deterministic policy gradient model is used to predict the initial insulin dose of the patient model.

[0051] Specifically, the deep deterministic policy gradient model is used to predict the insulin dosage corresponding to the virtual patient under the current blood sugar data according to the blood sugar data output by the type 1 diabetes patient model. However, since the deep deterministic policy gradient model is a model-free method, it faces challenges such as poor generalization ability and sensitivity to hyperparameters. Since there is no environmental model to guide the learning process, the model-free method usually requires more samples to learn a good strategy, and the performance is limited in long-term decision-making tasks, resulting in the insulin dosage predicted by the deep deterministic policy gradient model being not accurate enough. Therefore, in order to accelerate the learning process and improve the generalization ability and stability of the algorithm, the present invention introduces a dosage optimization model, which improves the insulin dosage strategy output by the deep deterministic policy gradient model through the dosage optimization model.

[0052] Step three: construct a dosage optimization model; the dosage optimization model is used to optimize the initial insulin dosage output by the deep deterministic policy gradient model, thereby obtaining an optimized insulin correction dosage.

[0053] Specifically, the present invention combines the advantages of model-free methods and model learning by combining the deep deterministic policy gradient model and the dosage optimization model, so as to more accurately guide the adjustment of insulin pump dosage and further optimize the performance of the algorithm. The entire algorithm is designed to fully automate the blood sugar control of diabetic patients, eliminate the manual estimation of CHO, and improve the blood sugar control effect of patients.

[0054] Step 4: determining the blood glucose data of the patient model at the next time step according to the optimized insulin correction dose;

[0055] Step 5: repeatedly obtaining the patient's blood glucose data at the next time step within a preset time period, and determining the TIR based on all the blood glucose data at the next time step within the preset time period;

[0056] Specifically, the main goal of blood sugar control is to maintain blood sugar levels within the normal range (70-180 mg / dL) and minimize the time that blood sugar is in the risk zone. The percentage time in the normal range (TIR) ​​is an intuitive indicator that represents the time that the subject's blood sugar level is in the normal blood sugar zone. The present invention can calculate the patient's blood sugar risk index by counting the percentage of time that the patient's blood sugar is maintained in different blood sugar concentration areas. The blood sugar risk index includes overall blood sugar risk (RI), high blood sugar risk (HGBI), and low blood sugar risk (LGBI). Table 1 summarizes the classification of the degree of danger of blood sugar levels.

[0057] Table 1 Risk classification of blood sugar levels

[0058]

[0059] Step six: determining the final insulin dosage of the patient model according to the TIR corresponding to the patient model in the preset time period.

[0060] The method provided by the present invention predicts the patient's initial insulin dose by constructing a deep deterministic policy gradient model, and optimizes the initial insulin dose through an auxiliary learning model to obtain an optimized insulin correction dose. This method improves the accuracy of predicting insulin and ultimately improves the effect of blood sugar control.

[0061] Further, Figure 2 The schematic diagram of the blood sugar closed-loop control structure provided by the present invention is as follows: Figure 2 As shown, the deep deterministic policy gradient model includes: an Actor network and a Critic network; the Actor network is used to output the corresponding insulin dosage decision according to the blood sugar data output by the patient model; the Critic network is used to judge the quality of the insulin dosage decision given by the Actor network through the action-value function Q value; the two networks interact with each other to finally obtain the initial insulin dosage of the patient model.

[0062] Specifically, constructing a deep deterministic policy gradient model includes: building a dual neural network architecture: an actor-critic network, the actor network is used to learn the deterministic policy function π, and is responsible for outputting the insulin dosage decision; the critic network is used to learn the state action function value Q, and is responsible for evaluating the quality of the current policy. The experience data samples generated by the interaction between the actor network and the environment are stored in the experience buffer pool, and then a small batch of data samples are extracted from it each time for training; the so-called strategy refers to a deterministic action selection function, and the embodiment of the present invention is to output the insulin dosage decision. The critic network is used to evaluate the reward of the strategy given in the actor network. The actor network is used to output the initial insulin dosage decision based on the patient's current blood sugar data after a meal; the critic network is used to evaluate the reward of the decision output by the actor network; the parameters of the actor network are updated by maximizing the Q value of the critic network, so that the insulin dosage decision output by the actor network can maximize the long-term reward; at the same time, the parameters of the critic network are updated by minimizing the loss function of the critic network to more accurately evaluate the policy performance of the actor network.

[0063] The present invention uses a dual neural network architecture to learn the strategy network and the Q network, and uses an online network and a target network for both the strategy function and the Q function, so that the learning process of the algorithm is more stable and the convergence speed is accelerated, thereby improving the efficiency of blood glucose concentration prediction.

[0064] Furthermore, in the process of constructing the deep deterministic policy gradient model, the experience data samples generated by the interaction between the Actor network and the environment need to be stored in the experience replay pool; the experience data samples include: current blood sugar data St, current insulin dosage at, current cumulative reward value rt, and next time step blood sugar data St+1; the Critic network extracts part of the data from the experience replay pool to evaluate the quality of the Actor network; the Actor network updates its own parameters according to the evaluation results; and uses the Actor network with updated parameters to output its corresponding insulin dosage decision.

[0065] Specifically, during the training phase of the deep deterministic policy gradient model, the Actor online network uses experience samples from the experience replay cache to update its weights according to the following formula:

[0066]

[0067] The Actor online network relies on backpropagation of policy gradients to obtain updates. In addition, the Critic online network can be trained by minimizing the loss function, which is calculated as follows:

[0068]

[0069] Here, y i is the target Q value, y i is obtained by calculating the current reward r i and the action value function Q(s i ,a i |θ Q ) is obtained by weighted sum, y i The calculation process of is as follows, where γ is the discount factor.

[0070] y i =r i +γ(Q'(s i+1 ,π'(s i+1 |θ π′ )|θ Q′ )

[0071] Finally, the target policy network parameter θ is guaranteed by the soft update mechanism. π and the target value network parameter θ Q Slow Update:

[0072] θ π′ ←ρθ π +(1-ρ)θ π′

[0073] θ Q′ ←ρθ Q +(1-ρ)θ Q′

[0074] The present invention selects the deep deterministic policy gradient model as the basic model for the following reasons: first, the deep deterministic policy gradient model is suitable for processing decision-making problems in continuous action space. In the process of blood glucose closed-loop control, it is necessary to adjust the dosage of the insulin pump according to the patient's real-time blood glucose level, which is a problem in a continuous action space. Secondly, the experience replay and soft update mechanism in the deep deterministic policy gradient algorithm can train the model more stably and reduce the instability during the training process. Compared with other deep reinforcement learning algorithms, it has better convergence and stability, and performs better in processing closed-loop control problems. Finally, the deep deterministic policy gradient model has a strong learning ability and can gradually optimize the control strategy according to continuous feedback and adjustment, making the closed-loop control system more intelligent and adaptable. However, like other model-free methods, the deep deterministic policy gradient model also faces challenges such as poor generalization ability and sensitivity to hyperparameters. Since there is no environmental model to guide the learning process, the model-free method usually requires more samples to learn a good strategy, and its performance is limited in long-term decision-making tasks. Therefore, in order to accelerate the learning process and improve the generalization ability and stability of the algorithm, the present invention introduces an auxiliary learning model for learning the patient's glucose dynamics and predicting blood sugar levels, and adjusts the insulin dosage decision of the Actor network in combination with the next step blood sugar prediction value output by the auxiliary learning model to improve the learned strategy. This two-step learning framework combines the advantages of model-free methods and model learning. By combining the strategy learned by the Actor network with the output of the blood sugar prediction model, it can more accurately guide the adjustment of the insulin pump dosage and further optimize the performance of the algorithm. The purpose of the present invention is to fully automate the blood sugar control of diabetic patients, eliminate the manual estimation of CHO, and improve the quality of life and health level of patients.

[0075] Furthermore, the Actor network and the Critic network respectively include an online network and a target network, that is, a current network and a target network. Through a soft update method, the parameters of the target network are slowly brought closer to the parameters of the online network.

[0076] Specifically, the target network parameters will copy parameters from the online network at a certain update frequency (such as every certain number of steps) to maintain consistency between the target network and the online network. The target network is used to calculate the target Q value or target strategy. If the parameters of the target network are directly copied as the parameters of the online network, the parameters of the target network may fluctuate greatly, affecting the convergence of the algorithm. The present invention can balance the update speed between the target network and the behavior network through a soft update method, so that the model converges to the optimal solution more easily. By gradually adjusting the parameters of the target network, the algorithm can find the optimal strategy faster, so that the parameters of the target network slowly move closer to the parameters of the online network, thereby maintaining the stability of the target network. The target network is more stable, which can improve the prediction efficiency of insulin dosage and ultimately improve the efficiency of blood glucose concentration prediction.

[0077] In the stage of building the insulin dosage recommendation model based on deep deterministic policy gradient, a dual deep neural network structure is used to implement the Actor network and the Critic network. The four networks have similar neural network structures, all consisting of an LSTM network and three fully connected layers. The average value and standard deviation of the predicted insulin dosage are output in the last layer of the network. The output of the Actor network is usually the mean of the action, while the standard deviation is an additional output used to represent the uncertainty of the action. By taking the standard deviation as the output, the Agent can take into account the variance of the action when learning the strategy, thereby better exploring the environment and improving the learning efficiency. The introduction of the standard deviation can make the Agent more robust in the learning process, help deal with uncertainty and noise in the environment, thereby improving the accuracy of predicting insulin dosage and ultimately improving the effect of controlling blood sugar. The hyperparameters of the online network and the target network include: network learning rate, batch size, number of hidden units, experience replay cache, and discount factor.

[0078] The selection of hyperparameters plays a crucial role in the performance and effect of the model. Good hyperparameter settings can significantly improve the prediction accuracy and generalization ability of the model. Therefore, when building a blood glucose closed-loop model, it is necessary to comprehensively consider factors such as model complexity, data characteristics, and accuracy to achieve the best performance and effect.

[0079] In the stage of building the insulin dosage recommendation model based on deep deterministic policy gradient, the specific hyperparameter settings are shown in Table 2.

[0080] Table 2 Parameter configuration based on deep deterministic policy gradient model

[0081]

[0082] The method provided by the present invention can more accurately guide the adjustment of insulin pump dosage and achieve more precise blood sugar control by combining the strategy learned by the Actor network with the output of the auxiliary learning model.

[0083] Further, how to construct a dose optimization model is described in detail below. The method specifically includes the following steps:

[0084] Step 31: Using the type 1 diabetes patient model, obtain the historical blood glucose data of the patient model; Step 32: Using the deep deterministic policy gradient model, obtain the insulin doses corresponding to the historical blood glucose data; Step 33: Based on the historical blood glucose data and the corresponding insulin doses, construct a blood glucose prediction model, and the blood glucose prediction model is used to predict the blood glucose prediction value of the patient model at the next time step; Step 34: Based on the blood glucose prediction value of the patient model at the next time step, the current blood glucose data of the patient model, and the insulin dose corresponding to the current blood glucose data predicted by the deep deterministic policy gradient model, construct an auxiliary learning model, and optimize the insulin dose output by the deep deterministic policy gradient model through the auxiliary learning model, so as to obtain the optimized insulin correction dose.

[0085] Specifically, the hyperparameters of the blood glucose prediction model include: network learning rate, batch size, number of hidden units, experience replay cache, and discount factor, as shown in Table 3.

[0086] Table 3 Hyperparameter configuration of blood glucose prediction model

[0087]

[0088] Figure 3 The structure diagram of the blood sugar prediction model provided by the present invention is as follows: Figure 3 As shown, the model includes:

[0089] Input layer: input historical blood glucose data and insulin dosage decisions; convert the raw data into a format acceptable to the neural network while maintaining the structure and characteristics of the data;

[0090] The first LSTM layer: is used to capture long-term dependencies and complex nonlinear patterns in time series data. This layer is responsible for modeling the input sequence and learning the temporal correlation in the data;

[0091] First Dropout layer: After the first LSTM layer, it is used to reduce the risk of overfitting and improve the generalization ability of the model. By randomly discarding the output of some neurons during the training process, the dropout layer helps prevent the model from overfitting the training data.

[0092] Second LSTM layer: further improves the model's ability to understand the intrinsic structure of the data. By stacking the second LSTM layer, the model can better capture the complex relationships and patterns in the data. Compared with a single-layer LSTM network, a stacked LSTM network has stronger feature learning and expression capabilities, can better adapt to complex time series data features, and is easier to train and converge.

[0093] Second Dropout layer: After the second LSTM layer, it is used again to reduce the risk of overfitting and ensure the robustness and generalization performance of the model. This layer helps improve the performance of the model on unseen data.

[0094] Fully connected layer: responsible for converting the output of the LSTM layer into a feature representation suitable for prediction. The fully connected layer combines and weights the features output by the LSTM layer to provide input for the final prediction task.

[0095] Output layer: Use appropriate activation functions to map the output of the model into the required blood glucose prediction range. Output the blood glucose prediction results for the next time step for subsequent analysis and application.

[0096] The present invention embeds the auxiliary learning model into the Actor network of the insulin dosage decision model based on deep deterministic policy gradient. The specific steps are as follows:

[0097] The blood glucose prediction model predicts the patient's next time step blood glucose prediction value g t+1 , and the patient's current blood sugar data t , Current insulin dosea t Stored in the auxiliary buffer; once the auxiliary buffer is filled, the auxiliary learning model starts learning, minimizing the deviation between the learned strategy and the optimal strategy after the deep deterministic policy gradient model is updated, and introducing a hyperparameter β to regulate the penalty. The goal is to minimize L defined in the following formula P (θ π ):

[0098]

[0099] Among them, -log(P(g t+1 |s t ,a t ) represents the negative log-likelihood loss of the blood glucose prediction model for the next blood glucose value, which measures the difference between the predicted value and the actual value. The Kullback-Leibler divergence (d KL ) to measure the DDPG policy network π DDPG and the optimized strategy π * The difference between them is multiplied by the hyperparameter β for penalty.

[0100] The present invention adjusts the insulin dosage decision given by the deep deterministic policy gradient model in combination with the next step blood glucose prediction value output by the blood glucose prediction model to improve the learned strategy. If the prediction accuracy of the prediction model P reaches RMSE < e target (18mg / dL), the auxiliary learning model improves the learned strategy by integrating short-term optimization goals and fine-tuning the learned strategy, that is, strategy updating. t ) Best Practices for Association Fine-tune the strategy. The goal is to minimize L defined by update (θ π ):

[0101]

[0102] Indicates that in state s t Best Practices The negative log-likelihood loss is used to measure the difference between the policy network’s prediction at the target state and the actual best action. By minimizing L update (θ π ) Minimize the loss as much as possible and optimize the policy network so that it better predicts the best action to perform in the target state, thereby improving the learned policy;

[0103] Implement the improved strategy and continue iterating.

[0104] The method provided by the present invention trains the auxiliary learning model by adopting the experience replay method. Randomly sampling samples from the buffer pool to train the auxiliary learning model can break the correlation between data, reduce the deviation in the training process, and improve the effect of the algorithm.

[0105] The present invention realizes that in reinforcement learning, training data is usually highly correlated, and the same sample needs to be used multiple times during the training process. Storing the samples in a buffer pool can avoid data waste and improve the utilization efficiency of the samples.

[0106] In summary, the blood glucose closed-loop control method provided by the present invention has the following advantages:

[0107] 1. The present invention models the blood sugar closed-loop control problem as a reinforcement learning problem. The state is defined as the current blood sugar level, insulin infusion and other related information, and the action is defined as the size of adjusting the insulin infusion. The goal is to maintain the blood sugar level within a reasonable range by adjusting the insulin infusion. A deep neural network is used to model a deterministic strategy, which is a function that maps the state to the action. Usually, an Actor network can be used to represent the strategy, and the output is the insulin infusion. At the same time, a Critic network can also be used to evaluate the strategy performance of the Actor network, and the output is the state-action value function (Q value). In order to improve the stability and convergence speed of the algorithm, by introducing the experience replay mechanism, the interaction data between the Agent and the environment is stored in the experience replay buffer, and random sampling is performed therefrom for training, reducing the correlation of the data and improving the training efficiency.

[0108] 2. In order to reduce the instability of the algorithm, the present invention uses a target network to stabilize the training process. The target networks of Actor and Critic are established respectively, and the parameters of the target network are updated regularly to gradually approach the parameters of the current network.

[0109] 3. Update the parameters of the Actor network by maximizing the Q value of the Critic network, so that the action output by the Actor network can maximize the long-term reward. At the same time, update the parameters of the Critic network by minimizing the loss function of the Critic network to more accurately evaluate the policy performance of the Actor network.

[0110] 4. Through continuous iterative training, the parameters of the Actor and Critic networks are optimized so that the Agent can learn an effective strategy to control blood sugar levels.

[0111] Experimental example:

[0112] The cumulative rewards of the ten patients are shown in Table 4. The average cumulative reward of the adult group is slightly higher than that of the adolescent group. The average cumulative reward of the adult group is 273.71±6.96, while the average cumulative reward of the adolescent group is 265.98±14.63. This shows that the proposed algorithm has a relatively good effect on blood sugar control in the adult group, and the data of the adult group is relatively more stable and less volatile. In the adult group, adult 4 obtained the highest cumulative reward of 285.03±2.94, while adult 3 obtained the lowest cumulative reward of 262.41±24.07. In the adolescent group, adolescent 5 obtained the highest total reward of 264.38±5.94, while adolescent 1 obtained the lowest total reward of 272.20±1.49. The method provided by the present invention has statistically significant cumulative rewards for 11 of the 6 adult and 6 adolescent subjects.

[0113] Table 4 Total rewards for all patients in the proposed algorithm

[0114]

[0115] The clinical performance test results of the method provided by the present invention in all simulated patients are shown in Table 5. It can be seen that the performance of the algorithm proposed by us in the adult group and the adolescent group shows good blood sugar control effect. The adult group performed well in terms of the percentage of time within the target blood sugar range and the risk of hypoglycemia, with an average TIR of 71.38%, showing good blood sugar control. In addition, we can observe that there are some differences in blood sugar control between the adult group and the adolescent group. In contrast, the TIR of the adolescent group was slightly lower than that of the adult group, with an average TIR of 66.42%.

[0116] Table 5 Comparison of clinical performance of all patients (%)

[0117]

[0118] In order to more intuitively demonstrate the blood sugar control effect of the algorithm proposed in the present invention on patients, the present invention displays the 24-hour blood sugar fluctuation graphs of the patients with the best blood sugar control effect and the patients with the worst control effect in the adult group and the adolescent group, respectively. Figure 4 (a) is the patient with the best blood sugar control in the adult group 4. Figure 4 (b) shows patient 3 who has the worst control effect. Adult patient 4 has the best blood sugar control effect, with a TIR of 74.58%, showing a lower incidence of hypoglycemia. Adult patient 3 has the worst blood sugar control effect, with a TIR of 67.78%, and spends a longer time in the high blood sugar range. Figure 5 (a) is the best performing patient among adolescent patients 1. Figure 5 The figure in (b) is the worst performing patient 5. Adolescent 1 had the best blood sugar control, with a TIR of 79.28%, while adolescent 5 had the worst blood sugar control, with a TIR of 55.27%. Overall, the blood sugar control in the adult group was more stable, while the blood sugar fluctuations and risks in the adolescent group were relatively large, but still within an acceptable range.

[0119] The present invention also provides a blood sugar closed-loop control device, comprising:

[0120] A construction unit, used for constructing a computer simulation model of a type 1 diabetes patient; the type 1 diabetes patient model is used for simulating and outputting the patient's blood sugar data;

[0121] The construction unit is further used to construct a deep deterministic policy gradient model based on the blood glucose data output by the type 1 diabetes patient model, wherein the deep deterministic policy gradient model is used to predict the initial insulin dose of the patient model;

[0122] The construction unit is further used to construct a dosage optimization model, wherein the dosage optimization model is used to optimize the initial insulin dosage output by the deep deterministic policy gradient model, thereby obtaining an optimized insulin correction dosage;

[0123] A determination unit, used to determine the blood glucose data of the patient model at the next time step according to the optimized insulin correction dose;

[0124] An acquisition unit, used to repeatedly acquire the patient's blood sugar data for the next time step within a preset time period;

[0125] The determination unit is further configured to determine the TIR based on all blood glucose data of the next time step within the preset time period;

[0126] The determination unit is further configured to determine a final insulin dose of the patient model according to the TIR corresponding to the patient model in the preset time period.

[0127] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program according to the blood glucose closed-loop control method described above.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A blood sugar closed-loop control device, characterized in that: include: A construction unit, used for constructing a computer simulation model of a type 1 diabetes patient; the type 1 diabetes patient model is used for simulating and outputting the patient's blood sugar data; The construction unit is further used to construct a deep deterministic policy gradient model based on the blood glucose data output by the type 1 diabetes patient model, wherein the deep deterministic policy gradient model is used to predict the initial insulin dose of the patient model; The construction unit is further used to construct a dosage optimization model, wherein the dosage optimization model is used to optimize the initial insulin dosage output by the deep deterministic policy gradient model, thereby obtaining an optimized insulin correction dosage; A determination unit, used to determine the blood glucose data of the patient model at the next time step according to the optimized insulin correction dose; An acquisition unit, used to repeatedly acquire the patient's blood sugar data for the next time step within a preset time period; The determination unit is further configured to determine the TIR based on all blood glucose data of the next time step within the preset time period; The determination unit is further configured to determine a final insulin dose of the patient model according to the TIR corresponding to the patient model in the preset time period; The blood sugar closed-loop control method of the blood sugar closed-loop control device comprises: Step 1: construct a computer simulation model of a type 1 diabetes patient; the type 1 diabetes patient model is used to simulate and output the patient's blood sugar data; Step 2: constructing a deep deterministic policy gradient model based on the blood glucose data output by the type 1 diabetes patient model, wherein the deep deterministic policy gradient model is used to predict the initial insulin dose of the patient model; Step 3: constructing a dosage optimization model; the dosage optimization model is used to optimize the initial insulin dosage output by the deep deterministic policy gradient model, thereby obtaining an optimized insulin correction dosage; The step three comprises: Step 31: using the type 1 diabetes patient model, obtaining historical blood glucose data of the patient model; Step 32: using the deep deterministic policy gradient model to obtain the insulin dosages corresponding to the historical blood sugar data; Step 33: constructing a blood glucose prediction model based on the historical blood glucose data and its corresponding insulin dosage, wherein the blood glucose prediction model is used to predict the blood glucose prediction value of the patient model at the next time step; Step 34: constructing an auxiliary learning model according to the predicted blood glucose value of the patient model at the next time step, the current blood glucose data of the patient model, and the insulin dose corresponding to the current blood glucose data predicted by the deep deterministic policy gradient model, and optimizing the insulin dose output by the deep deterministic policy gradient model through the auxiliary learning model, so as to obtain an optimized insulin correction dose; Step 4: determining the blood glucose data of the patient model at the next time step according to the optimized insulin correction dose; Step 5: repeatedly obtaining the patient's blood glucose data at the next time step within a preset time period, and determining the TIR based on all the blood glucose data at the next time step within the preset time period; Step six: determining the final insulin dosage of the patient model according to the TIR corresponding to the patient model in the preset time period.

2. The blood sugar closed-loop control device according to claim 1, characterized in that: The step one comprises: Step 11: Obtain patient data; the patient data includes: three meal times and the amount of carbohydrates corresponding to each time point; Step 12: Based on the patient data, a computer simulation model of a type 1 diabetes patient is constructed using the UVA / Padova simulator.

3. The blood sugar closed-loop control device according to claim 1, characterized in that: The deep deterministic policy gradient model described in step 2 includes: Actor network and Critic network; The Actor network is used to output the corresponding insulin dosage decision according to the blood sugar data output by the patient model; the Critic network is used to judge the quality of the insulin dosage decision given by the Actor network through the action-value function Q value; the Actor network and the Critic network interact with each other to finally obtain the initial insulin dosage of the patient model.

4. The blood sugar closed-loop control device according to claim 3, characterized in that: The experience data samples generated by the interaction between the Actor network and the environment are stored in the experience replay pool; The empirical data samples include: current blood sugar data S t 、Current insulin dose a t , the current cumulative reward value r t , blood sugar data S for the next time step t+1 ; The Critic network extracts some data from the experience replay pool to evaluate the quality of the Actor network; The Actor network updates its own parameters according to the evaluation result of the Critic network, and uses the Actor network with updated parameters to output its corresponding insulin dosage decision.

5. The blood sugar closed-loop control device according to claim 3, characterized in that: The Actor network includes an online network and a target network; The Critic network includes: an online network and a target network; Through the soft update method, the parameters of the target network are slowly brought closer to the parameters of the online network.

6. The blood sugar closed-loop control device according to claim 5, characterized in that: The blood sugar prediction model includes: an input layer, a first LSTM layer, a first Dropout layer, a second LSTM layer, a second Dropout layer, a fully connected layer, and an output layer; The hyperparameters of the blood sugar prediction model include: network learning rate, batch size, number of hidden units, experience replay cache, and discount factor.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the blood glucose closed-loop control method of the blood glucose closed-loop control device according to any one of claims 1 to 6 is implemented.

Images