Method and device for providing blood glucose control information

By combining fuzzy logic control and offline reinforcement learning algorithms, a personalized blood glucose control system was constructed, which solved the problem of instability in insulin dosage control for diabetic patients, and achieved stable blood glucose control for a longer period of time and reduced risk of hypoglycemia.

CN118969181APending Publication Date: 2024-11-15CHONGQING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410955550.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing blood glucose control methods struggle to achieve personalized and stable insulin dosage control in diabetic patients, leading to learning instability and potential dangers.

Method used

A fuzzy controller is constructed by combining a fuzzy logic control algorithm and a partially observable Markov decision process model with the offline reinforcement learning algorithm TD3-BC. The controller is embedded in the Actor module and generates personalized insulin dosage recommendations through fuzzy inference and action correction.

Benefits of technology

Under lower-risk conditions, it effectively controls blood glucose levels within the normal range for a longer period of time, avoids the risk of hypoglycemia, and improves the stability and safety of blood glucose control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118969181A_ABST
    Figure CN118969181A_ABST
Patent Text Reader

Abstract

The invention relates to a method and device for providing blood glucose control information. The method comprises the steps that monitoring data of a diabetic patient in a real scene is obtained and preprocessed; constructing a fuzzy controller by using a fuzzy logic control algorithm according to the monitoring data, and constructing a partially observable Markov decision process model according to the monitoring data; and embedding the fuzzy controller into an Actor module of an offline reinforcement learning algorithm TD3-BC, solving the partial observable Markov decision process model by using the offline reinforcement learning algorithm to obtain insulin dosage information required by the diabetic patient, and providing the insulin dosage information to a user. The blood glucose control information, namely the insulin dosage information required by the patient, can be accurately provided for the user, and the user can conveniently control the blood glucose according to the blood glucose control information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of diabetes blood sugar control, and in particular relates to a method and device for providing blood sugar control information. Background Art

[0002] People with diabetes either have insulin deficiency or are unable to produce enough insulin to overcome underlying insulin resistance or normalize glucose metabolism. Blood glucose levels in diabetic patients are affected by multiple complex factors, including diet, exercise, medication dosage, lifestyle, individual glucose metabolism and physical activity level, which are often nonlinear and dynamic in the system and difficult to accurately model using traditional mathematical models.

[0003] In glycemic control, reinforcement learning algorithms have demonstrated the ability to learn to develop complex and personalized control strategies for individual patients. These strategies often outperform traditional closed-loop control systems when trained and evaluated in glucose dynamics simulators; current approaches mainly use online RL algorithms that require interaction with patients or simulators during training to develop control strategies and learn through a trial-and-error-like process. These agents often have a poor understanding of their environment and are prone to learning instabilities, which can lead to dangerous insulin doses that threaten the patient's life. Summary of the invention

[0004] In order to solve the problems in the background technology, one aspect of the present invention provides a method for providing blood sugar control information, comprising:

[0005] S1: Obtain monitoring data of diabetic patients in real scenarios and perform preprocessing;

[0006] S2: construct a fuzzy controller using fuzzy logic control algorithm based on the monitoring data, and construct a partially observable Markov decision process model based on the monitoring data;

[0007] S3: Embed the fuzzy controller into the Actor module of the offline reinforcement learning algorithm TD3-BC, and use the offline reinforcement learning algorithm to solve the partially observable Markov decision process model to obtain the insulin dosage information required by the diabetic patient, and provide the insulin dosage information to the user.

[0008] Preferably, the monitoring data of the diabetic patient includes at least: the patient's blood sugar value, BMI value, age, the patient's basal insulin dose, the patient's insulin dose required to be injected, and carbohydrate activity.

[0009] Preferably, constructing a partially observable Markov decision process model based on monitoring data includes:

[0010] Parameters in the partially observable Markov decision process model are determined according to the monitoring data, wherein the parameters in the partially observable Markov decision process model include: state space, action space, state transition matrix, reward function, observation space and conditional observation probability.

[0011] Preferably, the states in the state space include:

[0012] s t =[g t ,g t-ΔT ,g t-2*ΔT ,...,g t-8*ΔT ,i t ,B t ,e t ,M t ]

[0013] Among them, s t represents the state at the current time t, g t represents the blood glucose level of a diabetic patient at the current time t; ΔT is 30 minutes; i t represents the basal insulin dose of a diabetic patient at the current time t; B t represents the BMI value of the diabetic patient at the current time t; e t represents the age of the diabetic patient at the current time t; M t represents the carbohydrate activity of the diabetic patient at the current time t.

[0014] Preferably, the reward function comprises:

[0015] r t =10·(3.5506·(log(g t ) 0.8353 -3.7932)) 2

[0016] Among them, g t represents the blood sugar value at time t; r t Reward value

[0017] Preferably, the input of the fuzzy controller is the status data of the diabetic patient, and the status data includes the patient's blood sugar value, BMI value, age, and the blood sugar difference value before and after 4 hours in the monitoring data. t -g t-8*T |, the output of the fuzzy controller is the insulin dosage required by the patient in the monitoring data; the fuzzy controller includes: a fuzzification module and a fuzzy reasoning module; the fuzzification module performs fuzzification processing on the input and output of the fuzzy controller based on a preset membership function; the fuzzy reasoning module performs reasoning on the fuzzified input based on a preset fuzzy rule base to obtain a reasoning result.

[0018] Preferably, the inference result of the fuzzy inference module is the output of the fuzzy controller, and its fuzzy set is set to small, moderate and large; the action in the action space is the insulin dose required by the diabetic patient; the action in the action space is obtained by the Actor module, and the Actor module includes: an Actor network and a fuzzy controller; the Actor network obtains the initial action according to the input state; the fuzzy controller obtains the inference result according to the input state; the initial action is corrected according to the inference result to obtain the action output by the Actor module, and the formula is as follows:

[0019] a t =(0.3*w less -0.4*w none +0.3*w more +1)*A original

[0020] Among them, a t Indicates the action output by the tActor module at the current moment, w less Indicates that the reasoning result of the fuzzy reasoning module is a small weight, w none Indicates that the reasoning result of the fuzzy reasoning module is a moderate weight, w more The reasoning result of the fuzzy reasoning module is represented by a large weight, A original Represents the initial action obtained by the tActor network at the current moment.

[0021] Preferably, the method of solving the partially observable Markov decision process model using an offline reinforcement learning algorithm to obtain the insulin dosage information required by the diabetic patient includes:

[0022] S41: Use UVA / PadovaT1D Simulator to generate offline data sets. Each data sample in the offline data set is represented by (s t , a t , r t ,s t+1 );

[0023] S42: Build the Actor module and convert s t Enter the Actor module to select action a t , then a t =u(s t |θ u ), where u represents the action selection strategy of the Actor module, θ u Represents the parameters of the Actor network in the Actor module;

[0024] S43: Using the Main Q Network and the target Q network Constructing an action evaluator using two Q networks and Estimate the current action a respectively t the value of

[0025] S44: Update the parameters of the main Q network by minimizing the Bellman error.

[0026] The calculation formula of the target value y is as follows:

[0027]

[0028] Among them, r t is the reward function, γ is the discount factor, Represents Q network or ε is a noise term used for smoothing; using the target value y and the predicted value of the main Q network Calculate the Bellman error loss L(θ1) and update the parameters of the main Q network through backpropagation:

[0029]

[0030] S45: Combined with the behavior cloning loss, update the parameters of the policy network u by the gradient ascent method, wherein the update gradient of the policy network u parameters includes:

[0031]

[0032] Where λ is the weight of behavior cloning loss; D represents the offline dataset; E represents the expectation; is the behavior cloning loss function, which represents the actual action a t The action output by the policy network u The mean square error between represents the policy network parameter θ u The updated gradient of represents the policy network parameter θ u The gradient of J(θ u ) represents the objective function of the policy network:

[0033] S46: After a certain number of training times, the parameters of the main Q network are soft-updated to the target Q network: θ2←τθ1+(1-τ)θ2, where θ1 and θ2 represent the Q network and Parameters; τ represents the step size parameter of soft update;

[0034] S47: Repeat the above steps until the model converges.

[0035] Another aspect of the present invention provides a device for providing blood sugar control information, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that a device for providing blood sugar control information executes a method for providing blood sugar control information.

[0036] Another aspect of the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the method for providing blood sugar control information.

[0037] The present invention has at least the following beneficial effects

[0038] The present invention consists of two modules: a fuzzy reasoning module and a blood sugar control module. The model can learn the fuzzy rules in blood sugar control through the treatment data of diabetic patients under the guidance of doctors or experts, and then integrate the fuzzy rules into the Actor network of TD3-BC in the form of fuzzy layers to obtain an improved fuzzy-TD3-BC algorithm, so as to control the patient's blood sugar level within the normal range for a longer time under the premise of lower risk, and try to control blood sugar not to fall to the range of low blood sugar level. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0040] Figure 2 It is a schematic diagram of a framework for solving a partially observable Markov decision process model in the present invention. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0042] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0043] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0044] See also Figure 1 and Figure 2 One aspect of the present invention provides a method for providing blood sugar control information, comprising:

[0045] S1: Obtain monitoring data of diabetic patients in real scenarios and perform preprocessing;

[0046] Preferably, the monitoring data of the diabetic patient includes: the patient's blood sugar value, BMI value, age, the patient's basal insulin dose, the patient's insulin dose required to be injected, and carbohydrate activity; wherein the blood sugar value and the patient's basal insulin dose are collected through the CGM blood sugar monitoring system, and the BMI value and age are given by the patient's basic information; the patient's insulin dose required to be injected is given by the patient's attending physician expert based on experience; the patient's basal insulin dose represents the insulin dose that existed in the patient's body at the previous moment.

[0047] Preferably, the preprocessing of the acquired monitoring data of the diabetic patient includes: standardization, missing value filling, deletion of incoherent data to remove invalid data, etc.

[0048] S2: construct a fuzzy controller using fuzzy logic control algorithm based on the monitoring data, and construct a partially observable Markov decision process model based on the monitoring data;

[0049] Preferably, the input of the fuzzy controller is the status data of the diabetic patient, and the status data includes the patient's blood sugar value, BMI value, age, and the blood sugar difference value before and after 4 hours in the monitoring data. t -g t-8*T |, g tRepresents the blood glucose level of a diabetic patient at time t; T is 30 minutes; the output of the fuzzy controller is the insulin dosage required by the patient in the monitoring data; the fuzzy controller includes: a fuzzification module and a fuzzy reasoning module; the fuzzification module fuzzifies the input and output of the fuzzy controller based on a preset membership function; the fuzzy reasoning module reasons the fuzzified input based on a preset fuzzy rule base to obtain a reasoning result.

[0050] In this embodiment, the fuzzy controller is constructed based on the historical treatment monitoring data of the diabetic patient as prior knowledge; the patient's status data: blood sugar value, BMI value, age, blood sugar difference before and after 4 hours |g t -g t-8*T |, and the insulin dosage required by the patient, respectively design different membership functions, obtain the fuzzy sets of the fuzzy controller input and output, and learn the corresponding fuzzy rules;

[0051] For example, using a trapezoidal membership function for blood glucose values, assuming that the blood glucose value at a certain moment is x mg / dL, blood glucose levels above 180 mg / dL are considered hyperglycemia, blood glucose levels between 70 mg / dL and 180 mg / dL are considered normal, and levels below 70 mg / dL are considered hypoglycemia. The fuzzy sets are set to hypoglycemia, normal blood glucose, and hyperglycemia, and the membership formulas are as follows:

[0052] Hypoglycemia membership formula:

[0053]

[0054] Normal blood sugar membership formula:

[0055]

[0056] Hyperglycemia membership formula:

[0057]

[0058] Using triangular membership functions for BMI, assuming the current BMI value is x, the normal BMI range is generally 18.5 kg / m 2 Up to 23.9kg / m 2 and BMI is less than 18.5 kg / m 2 Thin; BMI 24.0kg / m 2 Up to 27.9kg / m 2 If the BMI is greater than 28 kg / m 2 It is usually considered obese, and its fuzzy set is set as thin, normal, overweight, and obese, and the membership formulas are as follows:

[0059] The thin membership formula is:

[0060]

[0061] Normal membership formula:

[0062]

[0063] Overweight membership formula:

[0064]

[0065] Obesity membership formula:

[0066]

[0067] A trapezoidal membership function is used for age. Assuming the current age is x years old, people under 12 years old are considered children, people between 10 and 30 years old are considered young, people between 30 and 50 years old are considered middle-aged, and people over 50 years old are considered old. The fuzzy sets are set to children, young, middle-aged, and old. The membership formulas are as follows:

[0068] Children's membership formula:

[0069]

[0070] Young membership formula:

[0071]

[0072] Middle-aged membership formula:

[0073]

[0074] Elderly membership formula:

[0075]

[0076] The difference in blood sugar levels before and after 4 hours | g t -g t-8*T |, using the triangular membership function, assuming that the 4-hour blood sugar difference is xmg / dL; the 4-hour blood sugar difference is between 0-30mg / dL, which is a small difference; the 4-hour blood sugar difference is between 20-40, which is a normal difference; the 4-hour blood sugar difference is greater than 40, which is a large difference. The fuzzy sets are set to small, normal, and large, and the membership formulas are as follows:

[0077] Small membership formula:

[0078]

[0079] Normal membership formula:

[0080]

[0081] Large membership formula:

[0082]

[0083] A triangular membership function is used for the insulin dose required by diabetic patients. Assume that the insulin dose required by diabetic patients at a certain time t is xU; the insulin dose required by diabetic patients is small if it is between 0 and 10U; the insulin dose required by diabetic patients is moderate if it is between 10 and 20U; the insulin dose required by diabetic patients is large if it is between 20 and 25U. The fuzzy sets are set to small, moderate and large, and the membership formulas are as follows:

[0084] Small membership formula:

[0085]

[0086] Moderate membership formula:

[0087]

[0088] Large membership formula:

[0089]

[0090] For ease of understanding, this embodiment uses a sample to further illustrate the fuzzy controller under the above-mentioned conditions. For example, for a diabetic patient, the monitoring data at a certain moment is {blood sugar value 100mg / dL, BMI is 20kg / m 2 , age is 40 years old, the blood sugar difference before and after 4 hours is 50mg / dL, and the required insulin dose is 15U}; after processing through the above membership function, we get:

[0091] Blood glucose values ​​were {hypoglycemia (2 / 5), normoglycemia (3 / 5), hyperglycemia (0)};

[0092] BMI was {underweight (3 / 5), normal (2 / 5), overweight (0), obese (0)};

[0093] Age is {child(0), young(0), middle-aged(1), old(0)};

[0094] The 4-hour blood sugar difference is {small (0), normal (0), large (1)};

[0095] The insulin dose required by patients with diabetes {small (0), moderate (1), and large (0)};

[0096] Assume that there are two rules in the fuzzy rule base. For one fuzzy rule, if the blood sugar value is normal blood sugar, and the BIM is thin, and the age is middle-aged, and the 4-hour blood sugar difference is moderate, the diabetic patient needs a large dose of insulin; the weight of the fuzzy rule is 3 / 5×3 / 5×1×1×1=9 / 25; for the other rule, if the blood sugar value is hypoglycemia, and the BIM is thin, and the age is middle-aged, and the 4-hour blood sugar difference is moderate, the diabetic patient needs a large dose of insulin; the weight of the fuzzy rule is 2 / 5×3 / 5×1×1×1=6 / 25; the weight of the fuzzy controller outputting a large dose of insulin required by the diabetic patient is (9 / 25+6 / 25), and it is necessary to traverse all the rules in the rule base. Here is just an example of how to calculate the subsequent weights; similarly, the weights of the fuzzy controller outputting a small and moderate dose of insulin required by the diabetic patient can be calculated; the reasoning result of the fuzzy controller is {small (w less ), moderate (w none ), large (w more )}.

[0097] In an embodiment of the present application, a partially observable Markov decision process (POMDP) ​​is a generalized Markov decision process, in which the POMDP simulates the agent decision procedure by assuming that the system dynamics are determined by the MDP, but the agent cannot directly observe the state. On the contrary, it must infer the distribution of the state based on the global and partial regional observations of the model. The embodiment of the present application simulates the environment in which the diabetic patient is located and the action and state change process in this process through POMDP, and obtains the POMDP model.

[0098] Preferably, constructing a partially observable Markov decision process model based on monitoring data includes:

[0099] Determining parameters in a partially observable Markov decision process model according to the patient's monitoring data, wherein the parameters in the partially observable Markov decision process model include: a state space, an action space, a state transition matrix, a reward function, an observation space, and a conditional observation probability;

[0100] Specifically, the POMDP model can be described by a sextuple (S, A, Ω, Γ, R, O);

[0101] S represents the state space; the state s in the state space t ∈S includes:

[0102] s t =[g t ,g t-T ,g t-2T,...,g t-8T ,i t ,B t ,e t ,M t ]

[0103] Among them, s t ∈S represents the state at the current time t, g t represents the blood sugar level of the diabetic patient at the current time t; i t represents the basal insulin dose of a diabetic patient at the current time t; B t represents the BMI value of the diabetic patient at the current time t; e t represents the age of the diabetic patient at the current time t; M t represents the carbohydrate activity of the diabetic patient at the current time t;

[0104] In this embodiment, t represents the blood glucose value of the diabetic patient at the current time t, T represents the sampling time interval, and in this embodiment, T is set to 30 minutes, then g t-8T Expressed as the blood sugar value of diabetic patients 4 hours ago; |g t -g t-8T |Equal to the difference in blood sugar levels of diabetic patients before and after 4 hours;

[0105] A represents the action space; the action a in the action space t ∈A includes: the insulin dose that the patient needs to inject at time t;

[0106] Γ represents the state transfer matrix, which indicates that at time t the patient is in state s t ∈S, the agent is in the current state s t Take action a t ∈A transitions to state s t+1 , its state transition probability is P(s t+1 |s t ,a)=Γ(s t ,a,s t+1 ) indicates that the state s t Take action a t ∈A transitions to state s t+1 probability;

[0107] R represents the reward function, r t ∈R is given by the negative of the Magni hazard function, which models the clinical risk for a given blood glucose value; for blood glucose values ​​outside the physiologically feasible range (10 to 1000 mg / dl), an additional -1e 5Penalty; This modification is several orders of magnitude larger than the reward obtainable under the Magni risk function and acts as an incentive to prevent the agent from intentionally terminating the environment to avoid future negative rewards; The formula for the risk function is shown below:

[0108] r t =10·(3.5506·(log(g t ) 0.8353 -3.7932)) 2

[0109] Among them, g t represents the blood sugar value at time t; r t Represents the reward value.

[0110] Ω represents the observation space, o∈Ω represents the detected blood glucose value, the patient’s height, weight, age, and injected insulin dose;

[0111] In a POMDP, the agent cannot directly observe the next state s t+1 ∈S, but receives a signal with probability P(o t+1 |s t+1 ,a)=O(o t+1 ,s t+1 ,a) Determined observation value; o t+1 ∈Ω, which provides an observation about the next state.

[0112] In this embodiment, the basal insulin dose in the patient is t and carbohydrate active M t , given by the following formula:

[0113] The basal insulin dose in the patient t It is expressed as:

[0114]

[0115] Where N is the number of time steps before the considered time t; b t-t′ represents the insulin bolus dose at time step tt′, i t-t′ represents the basal insulin dose at time step tt′, c t-t′ Represents the amount of carbohydrates consumed at time step tt′.

[0116] S4: Embed the fuzzy controller into the Actor module of the offline reinforcement learning algorithm TD3-BC, and use the offline reinforcement learning algorithm to solve the partially observable Markov decision process model to obtain the insulin dosage information required by the diabetic patient, and provide the insulin dosage information to the user.

[0117] Preferably, the inference result of the fuzzy inference module is the output of the fuzzy controller, and its fuzzy set is set to small, moderate and large; the action in the action space is the insulin dose required by the diabetic patient; the action in the action space is obtained by the Actor module, and the Actor module includes: an Actor network and a fuzzy controller; the Actor network obtains the initial action according to the input state; the fuzzy controller obtains the inference result according to the input state; the initial action is corrected according to the inference result to obtain the action output by the Actor module, and the formula is as follows:

[0118] a t =(0.3*w less -0.4*w none +0.3*w more +1)*A original

[0119] Among them, a t Indicates the action output by the tActor module at the current moment, w less Indicates that the reasoning result of the fuzzy reasoning module is a small weight, w none Indicates that the reasoning result of the fuzzy reasoning module is a moderate weight, w more The reasoning result of the fuzzy reasoning module is represented by a large weight, A original It represents the initial action obtained by the Actor network at the current moment. less 、w none and w more They represent the reasoning results of the fuzzy controller. Taking the fuzzy set of the insulin dosage required by diabetic patients as {small, moderate, large} as an example, then w less The weight w represents the small insulin dose required by diabetic patients. none The insulin dose required by the blurred diabetic patient is a moderate weight, w more The insulin dose required by the blurred diabetic patient is a large weight; original Represents the insulin dosage required by a diabetic patient output by the Actor network in the Actor module.

[0120] In this embodiment, the Actor module includes: an Actor network and a fuzzy controller; the Actor network is composed of multiple serially connected linear layers, and its input is s t =[g t ,g t-T ,g t-2T ,...,g t-8T ,i t ,B t ,e t ,M t], the output is the initial action A original ; The input of the fuzzy controller is based on s t Extracted blood sugar value of diabetic patients g t , Age of diabetic patients t , BMI value of diabetic patients t The difference in blood sugar levels between the patients with diabetes and before and after 4 hours | g t -g t-8T |, after the fuzzy controller reasoning, the reasoning result {small (w less ), moderate (w none ), large (w more )}; The initial action is calibrated according to the inference result to obtain the final action.

[0121] In this embodiment, the offline reinforcement learning algorithm is used to solve the partially observable Markov decision process model, including:

[0122] S41: Use UVA / PadovaT1D Simulator to generate offline data sets. Each data sample in the offline data set is represented by (s t , a t , r t ,s t+1 );

[0123] S42: Build the Actor module and convert s t Enter the Actor module to select action a t , then a t =u(s t |θ u ), where u represents the action selection strategy of the Actor module, θ u Represents the parameters of the Actor network in the Actor module;

[0124] S43: Using the Main Q Network and the target Q network Constructing an action evaluator using two Q networks and Estimate the current action a respectively t The main Q network is used for actual Q value estimation and strategy update, and is the network directly involved in calculation and optimization during training. The target Q network is used to provide a stable target value y to calculate the loss of the main network, in order to slow down the change of the target value and increase the stability of training.

[0125] S44: Update the parameters of the main Q network by minimizing the Bellman error.

[0126] The calculation formula of the target value y is as follows:

[0127]

[0128] Among them, r t is the reward function, γ is the discount factor, Represents Q network or ε is a noise term used for smoothing; using the target value y and the predicted value of the main Q network Calculate the Bellman error loss L(θ1) and update the parameters of the main Q network through backpropagation:

[0129]

[0130] Among them, E stands for expectation.

[0131] S45: Combined with the behavior cloning loss, update the parameters of the policy network u by the gradient ascent method, wherein the update gradient of the policy network u parameters includes:

[0132]

[0133] Where λ is the weight of behavior cloning loss; D represents the offline dataset; E represents the expectation; is the behavior cloning loss function, which represents the actual action a t The action output by the policy network u The mean square error between represents the policy network parameter θ u The updated gradient of represents the policy network parameter θ u The gradient of J(θ u ) represents the objective function of the policy network:

[0134]

[0135] in, Indicates that the Q network is in state s t and the action output by the policy network The Q value below.

[0136] S46: After a certain number of training times, the parameters of the main Q network are soft-updated to the target Q network: θ2←τθ1+(1-τ)θ2, where θ1 and θ2 represent the Q network and Parameters; τ represents the step size parameter of soft update;

[0137] S47: Repeat the above steps until the model (strategy network u) converges.

[0138] Another aspect of the present invention provides a device for providing blood sugar control information, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that a device for providing blood sugar control information executes a method for providing blood sugar control information.

[0139] Another aspect of the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the method for providing blood sugar control information.

[0140] Next, we introduce the data set used in the experiment:

[0141] First, the electronic medical records and treatment record data of diabetic patients who are supervised by doctors and given treatment plan recommendations are used, which include the patient's physiological characteristics and treatment data. The Reduction Of Nocturnal Hypoglycemia By Using Predictive Algorithms And Pump Suspension (PSO1) dataset, Reduction of Nocturnal Hypoglycemia by Using Predictive Algorithms and Pump Suspension: An Outpatient Pilot Feasibility and Efficacy Study (PSO3) dataset, and Outpatient Reduction of Nocturnal Hypoglycemia by Using Predictive Algorithms and Pump Suspension in Children (PSO4) dataset from NIDDK are used here. The above three datasets are mainly used to extract the fuzzy rule base. In addition, the UVA / Padova simulator approved by the US Food and Drug Administration (FDA) is used to simulate the environment of T1D patients. The simulator combines processes related to glucose-insulin dynamics to simulate the effects of eating and insulin on blood glucose levels between AVPs. These processes include digestion of food, production of glucose in the liver, absorption of exogenous insulin, and sudden eating. The UVA / Padova simulator has three groups of AVPs with different glucose metabolism levels and insulin doses, including 3 adults, 3 adolescents, and 3 children. The training data volume of each virtual patient is 100,000, with attached Figure 2 v t and action a tMeaning the same.

[0142] In summary, the present invention consists of two modules: a fuzzy reasoning module and a blood sugar control module. The model can learn the fuzzy rules in blood sugar control through the treatment data of diabetic patients under the guidance of doctors or experts, and then integrate the fuzzy rules into the Actor network of TD3-BC in the form of fuzzy layers to obtain an improved fuzzy-TD3-BC algorithm, so as to achieve the goal of controlling the patient's blood sugar level within the normal range for a longer time under the premise of lower risks, and at the same time try to control blood sugar from falling to the range of low blood sugar levels.

[0143] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for providing blood sugar control information, characterized in that: include: S1: Obtain monitoring data of diabetic patients in real scenarios and perform preprocessing; S2: construct a fuzzy controller using fuzzy logic control algorithm based on the monitoring data, and construct a partially observable Markov decision process model based on the monitoring data; S3: Embed the fuzzy controller into the Actor module of the offline reinforcement learning algorithm TD3-BC, and use the offline reinforcement learning algorithm to solve the partially observable Markov decision process model to obtain the insulin dosage information required by the diabetic patient, and provide the insulin dosage information to the user.

2. A method for providing blood sugar control information according to claim 1, characterized in that: The monitoring data of the diabetic patient includes at least: the patient's blood sugar value, BMI value, age, the patient's basal insulin dose, the patient's insulin dose required to be injected, and carbohydrate activity.

3. A method for providing blood sugar control information according to claim 2, characterized in that: The construction of a partially observable Markov decision process model based on monitoring data includes: Parameters in the partially observable Markov decision process model are determined according to the monitoring data, wherein the parameters in the partially observable Markov decision process model include: state space, action space, state transition matrix, reward function, observation space and conditional observation probability.

4. A method for providing blood sugar control information according to claim 3, characterized in that: The states in the state space include: s t =[g t ,g t-ΔT ,g t-2*ΔT ,...,g t-8*ΔT ,i t ,B t ,e t ,M t ] Among them, s t represents the state at the current time t, g t represents the blood glucose level of a diabetic patient at the current time t; ΔT is 30 minutes; i t represents the basal insulin dose of a diabetic patient at the current time t; B t represents the BMI value of the diabetic patient at the current time t; e t represents the age of the diabetic patient at the current time t; M t represents the carbohydrate activity of the diabetic patient at the current time t.

5. A method for providing blood sugar control information according to claim 4, characterized in that: The reward function includes: r t =10·(3.5506·(log(g t ) 0.8353 -3.7932)) 2 Among them, g t represents the blood sugar value at time t; r t Represents the reward value.

6. A method for providing blood sugar control information according to claim 4, characterized in that: The input of the fuzzy controller is the status data of the diabetic patient, which includes the patient's blood sugar value, BMI value, age, and the blood sugar difference value before and after 4 hours in the monitoring data. t -g t-8*T |, the output of the fuzzy controller is the insulin dosage required by the patient in the monitoring data; The fuzzy controller includes: a fuzzification module and a fuzzy reasoning module; the fuzzification module performs fuzzification processing on the input and output of the fuzzy controller based on a preset membership function; the fuzzy reasoning module performs reasoning on the fuzzified input based on a preset fuzzy rule base to obtain a reasoning result.

7. A method for providing blood sugar control information according to claim 6, characterized in that: The inference result of the fuzzy inference module is the output of the fuzzy controller, and its fuzzy set is set to small, moderate and large; the action in the action space is the insulin dosage required by the diabetic patient; The action in the action space is obtained by the Actor module, which includes: an Actor network and a fuzzy controller; the Actor network obtains the initial action according to the input state; the fuzzy controller obtains the inference result according to the input state; the initial action is corrected according to the inference result to obtain the action output by the Actor module, and the formula is as follows: a t =(0.3*w less -0.4*w none +0.3*w more +1)*A original Among them, a t Indicates the action output by the tActor module at the current moment, w less Indicates that the reasoning result of the fuzzy reasoning module is a small weight, w none Indicates that the reasoning result of the fuzzy reasoning module is a moderate weight, w more The reasoning result of the fuzzy reasoning module is represented by a large weight, A original Represents the initial action obtained by the tActor network at the current moment.

8. A method for providing blood sugar control information according to claim 4, characterized in that: The method of using an offline reinforcement learning algorithm to solve the partially observable Markov decision process model to obtain the insulin dosage information required by the diabetic patient includes: S41: Use UVA / PadovaT1D Simulator to generate offline data sets. Each data sample in the offline data set is represented by (s t , a t , r t ,s t+1 ); S42: Build the Actor module and convert s t Enter the Actor module to select action a t , then a t =u(s t |θ u ), where u represents the action selection strategy of the Actor module, θ u Represents the parameters of the Actor network in the Actor module; S43: Using the Main Q Network and the target Q network Constructing an action evaluator using two Q networks and Estimate the current action a respectively t the value of S44: Update the parameters of the main Q network by minimizing the Bellman error. The calculation formula of the target value y is as follows: Among them, r t is the reward function, γ is the discount factor, Represents Q network or ε is a noise term used for smoothing; using the target value y and the predicted value of the main Q network Calculate the Bellman error loss L(θ1) and update the parameters of the main Q network through backpropagation: S45: Combined with the behavior cloning loss, update the parameters of the policy network u by the gradient ascent method, where the update gradient of the policy network u parameters includes: Where λ is the weight of behavior cloning loss; D represents the offline dataset; E represents the expectation; is the behavior cloning loss function, which represents the actual action a t The action output by the policy network u The mean square error between represents the policy network parameter θ u The updated gradient of represents the policy network parameter θ u The gradient of J(θ u ) represents the objective function of the policy network: S46: After a certain number of training times, the parameters of the main Q network are soft-updated to the target Q network: θ2←τθ1+(1-τ)θ2, where θ1 and θ2 represent the Q network and Parameters; τ represents the step size parameter of soft update; S47: Repeat the above steps until the model converges.

9. A device for providing blood sugar control information, characterized in that: It comprises a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the device for providing blood sugar control information executes a method for providing blood sugar control information as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, a method for providing blood sugar control information as described in any one of claims 1 to 6 is implemented.