Precise nutrition decision-making system and method based on offline reinforcement learning

Abstract

The invention relates to a precise nutrition decision-making system and method based on offline reinforcement learning, and belongs to the field of medical informatization. According to the method, patient medical record data are collected and preprocessed, and patient nutrition state indexes, a nutrition intervention scheme and nutrition state benefits after nutrition intervention are defined as a state space, an action space and a reward function respectively. And then, constructing a model by using an offline reinforcement learning algorithm, so that the intelligent agent learns to select an optimal nutrition intervention strategy under different nutrition conditions. And finally, evaluating the performance of the model in a test data set, and comparing the influence of different intervention schemes on the nutrition state of the patient. According to the invention, the nutrition intervention effect of the inpatient can be effectively improved, the reasonable distribution of medical resources is promoted, and the application value is high.

Description

Technical Field

[0001] The present invention belongs to the field of medical information technology and relates to an accurate nutrition decision-making system and method based on offline reinforcement learning. Background Art

[0002] Malnutrition is a common disease with a complex etiology, which can be caused by decreased nutrient intake or absorption, disease-related inflammation or other mechanisms. The high incidence of malnutrition in hospitalized patients can lead to adverse consequences such as decreased treatment efficacy, increased incidence of complications, reduced quality of life, and worse long-term prognosis. Therefore, early identification of the target population that can benefit and taking precise and individualized nutritional interventions for reversible factors in the development of malnutrition are very important for improving the prognosis of hospitalized patients.

[0003] Clinical nutritional intervention methods for hospitalized patients mainly include nutritional education, enteral nutrition and parenteral nutrition. Enteral and parenteral nutrition interventions using nutritional preparations are inherently complex, for example: 1) There are various intervention routes, including oral, tube feeding and parenteral nutrition; 2) Nutritional preparations have different dosage forms, doses and formulas; 3) During the treatment process, the intervention plan needs to be dynamically adjusted according to the patient's changing nutrition-related efficacy indicators; 4) Intervention plans need to be formulated separately for different phenotypes of malnutrition; 5) The potential limitations of existing nutritional risk screening or evaluation tools have led to some patients being considered unable to benefit from nutritional interventions, but actually can benefit; 6) Some patients actually cannot benefit but receive unnecessary nutritional interventions, resulting in a waste of medical resources and even negative treatment effects.

[0004] The selection of existing nutritional intervention programs is mainly based on the personal experience of health professionals, as well as nutritional intervention guidelines, standards and expert consensus. Since these methods mainly consider standardization at the group level, they sacrifice individual accuracy and cannot make more accurate intervention program selections based on the specific values ​​of the patient's core nutritional indicators. Conventional nutritional intervention programs led by health professionals cannot fully ensure benefits in core nutritional indicators such as weight. Therefore, a decision-making system that can intelligently and dynamically select nutritional intervention programs based on the different nutritional phenotypes of patients will provide greater flexibility, help improve treatment outcomes and promote the rational allocation of medical resources.

[0005] Reinforcement learning is one of the main branches of machine learning. In recent years, it has become a hot area in artificial intelligence research and has been gradually applied in the medical field. Because of its close to human learning mode, it is considered to be one of the most promising machine learning algorithms for achieving general artificial intelligence. Different from the characteristics of supervised learning that mainly solves intelligent cognition, the core idea of ​​reinforcement learning is that the agent takes behavioral actions in the environment and learns to improve the behavior strategy based on the reward information fed back by the environment to achieve the purpose of specific intelligent decision-making. This is similar to the classic scenario in nutritional intervention: the patient can be regarded as a system that conforms to Markov properties and exhibits a dynamically changing nutritional phenotype. Health professionals cause changes in the state by applying nutritional interventions, and adjust the intervention strategy according to the changes in the patient's detectable indicators, thereby maximizing medical benefits. Offline reinforcement learning is a form of reinforcement learning. The agent learns through the historical data that has been collected (usually collected by other strategies or manually) without real-time interaction with the environment. Previous studies have shown that intelligent systems based on offline reinforcement learning do show significant advantages comparable to or exceeding those of humans in disease treatment decision-making, such as the treatment of sepsis and tumors, but similar technologies are rarely used in the development of nutritional intervention decision-making support systems.

[0006] Using offline reinforcement learning to learn effective nutrition intervention strategies from a large amount of human experience and build a decision-making system can assist health professionals in selecting intervention plans. Based on the deployable and automated characteristics of machine learning models, it can also improve the efficiency of medical services and promote the rational allocation of medical resources, which has high application value. Therefore, the inventors have developed a precise nutrition decision-making system and method based on offline reinforcement learning. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide an accurate nutrition decision-making system and method based on offline reinforcement learning, so as to serve as an artificial intelligence-assisted decision-making tool to support health professionals in formulating and adjusting patients' nutrition intervention plans in real time.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] The precise nutrition decision-making method based on offline reinforcement learning includes the following steps:

[0010] Step 1: Collect medical records of hospitalized patients, including general information of patients, disease information, treatment information, nutrition-related anthropometric indicators, hematological indicators, changes in nutrition-related indicators before and after nutritional intervention, specific plans of nutritional intervention, and short-term and long-term clinical outcomes;

[0011] Step 2: Preprocess the collected medical record data, including deleting outliers, excluding samples with missing key information, interpolating missing data, performing unique-hot encoding on multi-classification data, standardizing continuous variables, deleting variables with the same value for all samples, and adding timestamp identification information based on whether the interaction segment between the agent and the environment has ended;

[0012] Step 3: Randomly divide the preprocessed medical record data into training set and test set according to a certain ratio;

[0013] Step 4: Define the patient's nutritional status index as the initial state space, and mark the index as the initial state S in the training and test data;

[0014] Step 5: Define the type of nutritional intervention plan actually implemented as the agent action space and mark it as action A in the training and test data;

[0015] Step 6: Define the nutritional status benefit after nutritional intervention as the environmental reward, and select one or more changes in nutrition-related indicators as the calculation basis of the environmental reward function. Express the reward result in numerical form and mark it as reward R in the training and test data;

[0016] Step 7: Take the same nutritional status index as the initial state space as the new state space and mark it as the new state S_new in the training and test data;

[0017] Step 8: Set the hyperparameters of offline reinforcement learning, including but not limited to learning rate, discount factor, greed coefficient, number of hidden layers of the neural network, loss function, and training rounds;

[0018] Step 9: Based on the training set data, according to the parameter settings of steps 4 to 8, an offline reinforcement learning algorithm is used to build a model, and the agent interacts with the environment according to the offline data in the training set until the model converges or reaches the preset number of iterations;

[0019] Step 10: Evaluate the model performance in the test data set, group the test data according to whether the nutritional intervention plan in the test data set is consistent with the model recommendation strategy, and compare whether there are differences in body weight, limb skeletal muscle index, nutritional risk screening score, nutritional assessment score and 30-day mortality rate indicators before and after nutritional intervention between the inconsistent group and the consistent group.

[0020] Furthermore, the offline reinforcement learning algorithm is Q learning, SARSA or batch constrained Q learning.

[0021] Furthermore, the patient's nutritional status indicators include but are not limited to body weight, limb skeletal muscle index, calf circumference, triceps skinfold thickness, inflammatory status score, nutritional risk screening score, nutritional assessment score, malnutrition diagnosis and severity grading.

[0022] Furthermore, the types of nutritional intervention plans actually implemented include but are not limited to parenteral nutritional intervention, enteral nutritional intervention, combined parenteral and enteral nutritional intervention, nutritional intervention not performed in accordance with guidelines, and no nutritional intervention at all.

[0023] Furthermore, the nutritional status benefit after the nutritional intervention is calculated based on the changes in one or more nutrition-related indicators before and after a certain length of intervention cycle, and the length of the intervention cycle includes but is not limited to 24 hours, one week, and one month.

[0024] Furthermore, the calculation method of the environmental reward is: R = post-intervention index - pre-intervention index. If the index increases, the reward is a positive number, if the index decreases, the reward is a negative number, and if there is no change, the reward is zero.

[0025] Furthermore, the medical record data collection of hospitalized patients in step one also includes the patient's surgical treatment information and drug treatment information.

[0026] Furthermore, in step 2, the missing data are interpolated using a multiple interpolation approach.

[0027] Furthermore, the evaluation methods used in step ten include independent sample t-test, rank sum test and chi-square test.

[0028] A precise nutrition decision-making system based on offline reinforcement learning, including:

[0029] Data collection module: used to collect electronic medical record data of patients, including general information of patients, disease information, treatment information, nutrition-related anthropometric indicators, hematological indicators, changes in nutrition-related indicators before and after nutritional intervention, specific plans of nutritional intervention, and short-term and long-term clinical outcomes;

[0030] Data analysis module: including a reinforcement learning modeling submodule and an individual recommendation data calculation submodule. The reinforcement learning modeling submodule is used to perform offline reinforcement learning-based modeling according to the data obtained by the data acquisition module, so that the intelligent agent can learn to obtain the best nutritional intervention strategy under different nutritional conditions. The individual recommendation data calculation submodule is used to find or calculate the best nutritional intervention plan under the current state according to the intervention strategy obtained by the reinforcement learning modeling submodule and the input individual recommendation data;

[0031] Strategy output module: includes a strategy display submodule and a personalized recommendation plan submodule. The strategy display submodule is used to display some nutritional intervention plans corresponding to different nutritional status on the screen according to the reinforcement learning modeling results of the reinforcement learning modeling submodule. The personalized recommendation plan submodule is used to display the input patient status and the obtained optimal nutritional intervention plan according to the calculation results of the individual recommendation data calculation submodule and the input patient status.

[0032] The beneficial effects of the present invention are:

[0033] (1) Compared with conventional nutritional intervention programs based on personal experience of health professionals, the present invention is based on human experience and has the characteristics of objectivity, a priori, and effect orientation; in addition, the offline reinforcement learning algorithm used in the present invention can use past data compared to online reinforcement learning, and does not need to collect new data in real time for modeling. Because there is a certain risk in the interaction between the intelligent agent and the environment in the medical scene, offline learning has good safety and cost savings.

[0034] (2) Compared with the nutrition intervention guidelines for specific populations that sacrifice a certain degree of accuracy, the present invention can theoretically set an infinitely divisible continuous state space, thereby achieving high-granularity division of the population and obtaining a more accurate nutrition intervention plan; the present invention can achieve different treatment goals in a targeted manner by replacing different nutrition status indicators and environmental rewards, so it has high flexibility; importantly, conventional patient clustering analysis methods may mask the role of a small number of potential intervention plans that are not in line with experience or guideline support but have significant therapeutic effects, and the present invention is expected to discover such effective intervention strategies.

[0035] (3) The present invention does not specify a specific offline reinforcement learning algorithm. Therefore, a specific algorithm can be reasonably selected according to different disease populations, disease stages, data types, and sample sizes. This improves the flexibility of the present invention in the application process and can meet the needs of saving computing costs under different hardware conditions. Therefore, the model can be easily applied to clinical health care scenarios with different conditions and disease complexity.

[0036] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 It is a technical flow chart of the present invention;

[0039] Figure 2 A schematic diagram of offline reinforcement learning modeling in Embodiment 1 of the present invention and a three-dimensional visualization diagram of a Q table generated according to the modeling results;

[0040] Figure 3 A bar graph showing changes in body mass index and limb skeletal muscle index before and after intervention and cross-grouping with strategy consistency in Example 1 of the present invention;

[0041] Figure 4 A schematic diagram of offline reinforcement learning modeling in the second embodiment of the present invention and a three-dimensional visualization diagram of a Q table generated according to the modeling results;

[0042] Figure 5 This is a bar graph of changes in body mass index and limb skeletal muscle index before and after intervention and cross-grouping with strategy consistency in Example 2 of the present invention. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0045] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0046] In a first aspect, the present invention provides a method for constructing a precise nutrition decision model based on offline reinforcement learning, comprising the following steps:

[0047] 1) Selection of modeling population

[0048] A population of inpatients was selected, and variable information of the modeling population was collected through sources such as medical electronic medical record systems, including but not limited to patient general information, disease information, surgical treatment information, drug treatment information, weight, height, limb skeletal muscle index, calf circumference, triceps skinfold thickness, serum albumin, serum transferrin, C-reactive protein, specific plan of nutritional intervention, type of nutritional preparation, specification of nutritional preparation, dosage of nutritional preparation, route of administration of nutritional preparation, weight change before and after one month of nutritional intervention, change of limb skeletal muscle index before and after one month of nutritional intervention, and short-term and long-term clinical outcomes as data set samples;

[0049] 2) Sample data preprocessing:

[0050] Preprocess the collected sample data sets, including but not limited to deleting outliers, excluding samples with missing key modeling information, interpolating missing data using multiple interpolation, one-hot encoding multi-classified data, standardizing continuous variables using the z-score method, deleting variables with the same value for all samples, and adding timestamp identification information based on whether the interaction segment between the agent and the environment has ended;

[0051] 3) Divide the training set and test set:

[0052] The sample data set preprocessed in step 2) is randomly divided into a training set and a test set according to a certain ratio. The training set occupies a larger proportion and is used to strengthen the learning strategy of the learning model, and the training set data occupies a smaller proportion and is used to evaluate the effectiveness of the strategy obtained by the training set;

[0053] 4) Define the value range of the initial state space (State, S):

[0054] Modeling the patient's state as an environmental state, specifically, modeling the patient's nutritional state as a partially observable environmental state, using an indicator, an indicator set or a state score that reflects the patient's nutritional state before the implementation of nutritional intervention as the value of the initial state space and annotating the variable name as the initial state S in the training and test data, and the selection of indicators includes but is not limited to body weight, limb skeletal muscle index, calf circumference, triceps skinfold thickness, inflammatory state score, nutritional risk screening score, nutritional assessment score, malnutrition diagnosis and severity classification;

[0055] 5) Define the value range of the agent action space (Action, A):

[0056] The doctor or nutritionist who performs the nutritional intervention is modeled as an intelligent agent, and the type of nutritional intervention program actually implemented is used as the value range of the action space. The variable name is marked as action A in the training and test data. For the algorithm that needs the next action to update the model parameters or Q table, the variable name is marked as A_new. The selection of action variables includes but is not limited to the coarse classification of parenteral enteral nutrition intervention, the daily dose of parenteral nutrition, the number of days of parenteral nutrition intervention, the daily dose of enteral nutrition corrected by kilograms of body weight, and the total dose of parenteral enteral nutrition calculated by individual energy protein consumption and actually implemented by the patient;

[0057] 6) Define the value range of environmental reward (Reward, R):

[0058] The nutritional status benefit of patients after nutritional intervention is modeled as an environmental reward, and the changes in one or more nutrition-related indicators before and after a certain length of intervention period are used as the calculation basis of the environmental reward function. The length of the intervention period includes but is not limited to 24 hours, one week, and one month; the selection of nutrition-related indicators depends on one or more indicators that the treatment hopes to improve. The reward result is expressed in numerical form, and the reward value is determined based on the improvement, unchanged or worsening of the patient's nutritional status after nutritional intervention, and the variable name is marked as reward in the training and test data; nutrition-related indicators include but are not limited to body weight, limb skeletal muscle index, nutritional risk screening score, and nutritional assessment score, and the variable name is marked as reward R in the training and test data;

[0059] 7) Define the value range of the new state space (New state, S_new):

[0060] The same indicator, indicator set or state score in step 4) that reflects the nutritional status of the patient after the nutritional intervention is implemented is used as the value of the new state space, and the variable name is marked as the new state S_new in the training and test data;

[0061] 8) Set model hyperparameters:

[0062] Pre-set the hyperparameters of offline reinforcement learning, including but not limited to the learning rate α, discount factor γ, greed coefficient ε, number of hidden layers of the neural network n_layer, loss function loss_function, and training round episode;

[0063] 9) Build an offline reinforcement learning model

[0064] Based on the training data of step 3), according to the parameter settings of steps 4) to 8), offline reinforcement learning is used to build the model. First, the agent behavior strategy is initialized, and then the interaction between the agent and the environment is started according to the offline data in the training set. Each interaction between the agent and the environment includes: sampling a patient instance from the data set and recording the initial state, the agent takes one step of action, and then the state is updated, and the reward is calculated and the model parameters or Q table state are updated according to the state update. Repeat the above interaction process until the agent behavior strategy is generated after the number of iterations predefined in step 8) is completed or the model converges; the selection of reinforcement learning algorithms includes but is not limited to Q-learning (Q-learning), SARSA (State-Action-Reward-State-Action) and batch constrained Q-learning (Batch Constrained Q-learning, BCQ);

[0065] 10) Evaluate model performance on the test dataset

[0066] According to the agent behavior strategy output in step 9), the model performance is evaluated using the test data in step 3), and the test indicators include but are not limited to: grouping patients according to whether the strategy adopted by the test data is the model recommended strategy, using independent sample t test or rank sum test to evaluate whether there is a significant difference in body weight before and after nutritional intervention between the inconsistent group and the consistent group, using independent sample t test or rank sum test to evaluate whether there is a significant difference in body weight before and after nutritional intervention within the inconsistent group and the consistent group, using independent sample t test or rank sum test to evaluate whether there is a significant difference in limb skeletal muscle index before and after nutritional intervention between the inconsistent group and the consistent group, using independent sample t test or rank sum test to evaluate whether there is a significant difference in limb skeletal muscle index before and after nutritional intervention between the inconsistent group and the consistent group, and using chi-square test to compare whether there is a difference in 30-day mortality between the inconsistent group and the consistent group;

[0067] In step 1), a specific disease subgroup population is selected for hierarchical modeling to reduce the amount of calculation and obtain a more targeted model;

[0068] In step 4) and step 7), the body mass index of the patient before and after the nutritional intervention is used as a continuous and discrete state space for modeling respectively;

[0069] In step 4) and step 7), the skeletal muscle index of the patient's limbs before and after nutritional intervention is used as a continuous and discrete state space to model respectively;

[0070] In step 4) and step 7), the quantitative or qualitative results of the nutritional screening, assessment or diagnosis of the patient before and after the nutritional intervention are used as a continuous or discrete state space for modeling;

[0071] In step 4) and step 7), cluster analysis is performed on multiple nutrition-related indicators of the patient, and the clustering results are used as discrete state space for modeling;

[0072] In step 5), only whether the nutritional intervention program is carried out in accordance with the guidelines is distinguished, thereby simplifying the action space value and accelerating the model convergence speed;

[0073] In step 5), a cluster analysis is performed on the complex intervention scheme, and the result of the cluster analysis is used as the value of the discrete action space, thereby accelerating the convergence speed of the model;

[0074] In step 6), if the patient's nutritional status changes from malnutrition to good nutrition, the positive reward value is multiplied by a coefficient greater than 1 to increase the reward effect on the agent;

[0075] In step 6), if the patient's nutritional status changes from well-nourished to malnourished, the negative reward value is multiplied by a coefficient greater than 1 to increase the penalty effect on the agent;

[0076] In the step 8), the model is optimized in parameters, wherein the parameter optimization is a grid parameter search in a predefined parameter space;

[0077] In step 9), the experience replay technique is used to sample the non-independent and identically distributed time series data in the training set to form an experience library containing several records, and the intelligent agent performs sampling learning from the experience library;

[0078] In step 10), the evaluation method used is to compare whether there is a difference in the patient's weight improvement when the human intervention scheme is the same or different from the reinforcement learning model strategy;

[0079] In the step 10), the evaluation method used is to compare whether there is a difference in the patient's hospital stay when the human intervention scheme is the same or different from the reinforcement learning model strategy;

[0080] In the second aspect, the present invention provides a precise nutrition decision-making system based on offline reinforcement learning, which mainly includes the following modules:

[0081] Data acquisition module:

[0082] 1) Model training data input submodule: used to collect patient medical electronic medical record data, including general information, disease information, treatment information, nutrition-related anthropometric indicators, hematological indicators, changes in nutrition-related indicators before and after nutritional intervention, specific plans for nutritional intervention, and short-term and long-term clinical outcomes as input data;

[0083] 2) Individual recommendation data input submodule: receiving single or batch patient-specific nutritional status indicator values ​​as input data;

[0084] Data Analysis Module:

[0085] 1) Reinforcement learning modeling submodule: Use the data obtained from the input module 1) to execute the offline reinforcement learning-based modeling, so that the intelligent agent learns to obtain the best nutritional intervention strategy under different nutritional conditions. This module uses offline reinforcement learning technology to simulate the process of doctors or nutritionists performing nutritional intervention on patients, so as to intelligently learn and improve nutritional intervention strategies. Based on the core indicators of clinical nutrition diagnosis and treatment, the nutritional intervention treatment process is modeled as a Markov decision process, the doctor is modeled as an intelligent agent, the patient's nutritional status is modeled as a partially observable environment, nutritional intervention is defined as an action, and the change of indicators that reflect the effect of nutritional intervention treatment is defined as a reward function. Through model learning on offline data collected during data collection, accurate nutritional intervention strategies that can maximize the therapeutic effect are obtained and a decision-making system is constructed;

[0086] 2) Individual recommendation data calculation submodule: According to the intervention strategy obtained by the data analysis module 1) reinforcement learning modeling submodule, the input data obtained by the input module 2) individual recommendation data input submodule are used to perform search or function calculation to obtain the best nutrition intervention plan in the current state;

[0087] Policy output module:

[0088] 1) Strategy display submodule: based on the reinforcement learning modeling results of the data analysis module 1), partial nutrition intervention plans corresponding to different nutrition states are displayed on the screen. If there are too many nutrition states to be listed completely, several examples are displayed;

[0089] Personalized recommendation program submodule: based on the calculation results of the data analysis module 2), the patient status input in the input module 2) and the optimal nutritional intervention program obtained by the data analysis module 2) are simultaneously displayed;

[0090] Embodiment 1:

[0091] In this embodiment, see Figure 1 , a method for constructing a precise nutrition decision model based on offline reinforcement learning, including the following steps:

[0092] 1) Establish a sample set:

[0093] Data of 1,210 hospitalized patients with chronic non-communicable diseases who met the research requirements were selected as the data set sample, including general information of the patient population, disease information, weight changes before and after nutritional intervention, and specific nutritional intervention plans;

[0094] 2) Sample data preprocessing:

[0095] Preprocess the collected sample data sets, including deleting outliers, excluding samples with missing key modeling information, and interpolating missing data;

[0096] 3) Divide the training set and test set:

[0097] The sample data set preprocessed in step 2) is randomly divided, 80% is saved as a training set in the experience replay pool, and the remaining 20% ​​is used as a test set, which are used for offline reinforcement learning model training and performance evaluation respectively;

[0098] 4) Define the value range of the initial state space (State, S):

[0099] See also Figure 2 The body mass index (BMI) of the patient before nutritional intervention is retained to one decimal place as the initial discrete state space S. The body mass index is calculated as: weight (kg) ÷ height (m) 2 ;

[0100] 5) Define the value of the agent action space (Action, A):

[0101] See also Figure 2 , the actual implemented nutrition intervention plan is used as the agent action space A, including the following categories: parenteral nutrition intervention according to the guidelines, enteral nutrition intervention according to the guidelines, combined parenteral and enteral nutrition intervention according to the guidelines, nutrition intervention not according to the guidelines, and no nutrition intervention;

[0102] 6) Define the value of environmental reward (Reward, R):

[0103] See also Figure 2 , the difference in body mass index before and after nutritional intervention is used as the reward value R, that is, R = BMI 干预后 -BMI 干预前 , if the body mass index increases, the reward is a positive number, if the body mass index decreases, the reward is a negative number, and if there is no change, the reward is zero;

[0104] 7) Define the value of the changed state space (New state, S_new):

[0105] See also Figure 2 , using the same method as in step 4) to calculate the body mass index after intervention as the discrete state space value S_new after the change;

[0106] 8) Set model hyperparameters:

[0107] Set the hyperparameters of offline reinforcement learning: learning rate α = 0.1, which means that at each update, the proportion of new information in the Q value update is 10%; discount factor γ = 0.9, which determines the influence of future rewards on current decisions; greed coefficient ε = 0.1, which indicates the probability of the agent randomly choosing an action; and learning iteration number episode = 50000, which indicates the total number of times the agent interacts with the environment in the environment;

[0108] 9) Build an offline reinforcement learning model;

[0109] Based on the training data of step 3), a Q-learning algorithm is used to build a model in the Python 3.9.11 environment, and the steps include: initializing the Q table using the NumPy library. The rows of the table represent different values ​​in the state space, and the columns represent different values ​​in the action space. The initial value of the table is set to 0. Use the ε-greedy strategy to select actions. Set the probability according to the greedy coefficient ε: if the random number between 0 and 1 is less than ε, randomly select an action (exploration). Otherwise, select the action with the maximum Q value corresponding to the current state in the Q table (utilization). Use the for loop statement to define the training rounds as 50,000, and use the if statement in each loop to control the agent to take action. In each round of training, the agent selects an action according to the current state, interacts with the environment, obtains rewards and updates the Q table; based on the current state and the reward R obtained after the action is executed, and the next state S_new. Calculate the update of Q value. For each state-action pair, use Bellman equation to update Q value, the formula is: Q[S,A] = Q[S,A] + α × (R + γ × argmax (Q[S_new]) - Q[S,A]). In each training round, the agent will move to the next state (S_new) according to the current action and environmental feedback, and update the round counter. Repeat this process until the end of the training round. After the training is completed, each row in the Q table represents the Q value of different actions in the current state. The optimal strategy of the agent is to select the action with the maximum Q value in each state, that is, select the column index corresponding to the maximum Q value in each row as the best nutritional intervention plan learned by the agent. See Figure 2 .

[0110] 10) Analyze in the test data set

[0111] According to the agent behavior strategy table output in step 9), the nutritional intervention effect analysis is performed on the test data set in step 3), and the test data is grouped according to whether the nutritional intervention plan in the test data set is consistent with the optimal plan in the Q table. The body mass index and limb skeletal muscle index before and after the intervention are compared between groups according to the grouping information, see Figure 3 .

[0112] The results show that the nutritional adjustment strategy obtained by combining offline data with Q-learning reinforcement learning agent learning in the present invention has achieved better results than conventional clinical strategies, significantly improved the weight gain of patients, and effectively improved the attenuation of skeletal muscle in the limbs, with statistical differences. Therefore, the present invention is expected to achieve dynamic intelligent adjustment of the nutritional intervention strategy for hospitalized patients, and is expected to significantly improve the weight of hospitalized patients and maintain and improve the skeletal muscle level, thereby improving the short-term and long-term clinical outcomes related to nutrition.

[0113] Embodiment 2:

[0114] This embodiment is substantially the same as the first embodiment. Figure 4 , but different state spaces and environmental rewards are defined, and different algorithms are used for modeling. The method for constructing a precise nutrition decision model based on offline reinforcement learning includes the following steps:

[0115] 1) Establish a sample set:

[0116] Data of 647 hospitalized patients with chronic non-communicable diseases who met the research requirements were selected as the data set sample, including general information of the patient population, disease information, changes in skeletal muscle index of the limbs before and after nutritional intervention, and specific nutritional intervention plans;

[0117] 2) Sample data preprocessing:

[0118] Preprocess the collected sample data sets, including deleting outliers, excluding samples with missing key modeling information, and interpolating missing data;

[0119] 3) Divide the training set and test set:

[0120] The sample data set preprocessed in step 2) is randomly divided into 80% as a training set and the remaining 20% ​​as a test set, which are used for training and performance evaluation of the offline reinforcement learning model respectively;

[0121] 4) Define the value of the initial state space (State, S):

[0122] See also Figure 4 The appendicular skeletal muscle mass index (ASMI) of the patient before nutritional intervention was retained as an integer as the initial discrete state space. The appendicular skeletal muscle mass index was calculated as follows: appendicular skeletal muscle mass (kg) obtained by bioelectrical impedance analysis ÷ height (m) 2 ;

[0123] 5) Define the value of the agent action space (Action, A):

[0124] See also Figure 4 , the actual implemented nutrition intervention plan is used as the agent action space A and the ε-greedy strategy selects the next action A_new. The values ​​of the action space include: parenteral nutrition intervention according to the guidelines, enteral nutrition intervention according to the guidelines, combined parenteral and enteral nutrition intervention according to the guidelines, nutrition intervention not according to the guidelines, and no nutrition intervention;

[0125] 6) Define the value of environmental reward (Reward, R):

[0126] See also Figure 4 , the difference in body mass index before and after nutritional intervention is used as the reward value R, that is, R = ASMI 干预后 -ASMI 干预前 , if the skeletal muscle index of the limbs increases, the reward is positive, if the skeletal muscle index of the limbs decreases, the reward is negative, and if there is no change, the reward is zero;

[0127] 7) Define the value of the changed state space (New state, S_new):

[0128] See also Figure 4 , using the same method as in step 4) to calculate the skeletal muscle index of the limbs after intervention as the discrete state space value S_new after the change;

[0129] 8) Set model hyperparameters

[0130] Set the hyperparameters of offline reinforcement learning: learning rate α = 0.1, which means that at each update, the proportion of new information in the Q value update is 10%; discount factor γ = 0.9, which determines the influence of future rewards on current decisions; greed coefficient ε = 0.1, which indicates the probability of the agent randomly choosing an action; and learning iteration number episode = 50000, which indicates the total number of times the agent interacts with the environment in the environment;

[0131] 9) Build an offline reinforcement learning model

[0132] Based on the training data of step 3), the SARSA (State-Action-Reward-State-Action) algorithm is used to build a model in the Python 3.9.11 environment, and the steps include: using the NumPy library to initialize the Q table. The rows of the table represent different values ​​of the state space, and the columns represent different values ​​of the action space. The initial value of the table is set to 0. Use the ε-greedy strategy to select actions. Set the probability according to the greedy coefficient ε: if the random number between 0 and 1 is less than ε, randomly select an action (exploration). Otherwise, select the action with the maximum Q value corresponding to the current state in the Q table (utilization). Use the for loop statement to define the training round as 50,000, and use the if statement in each loop to control the agent to take action. In each round of training, the agent selects an action according to the current state, interacts with the environment, obtains rewards and updates the Q table; according to the current state and the reward R obtained after the action is executed, and the next state S_new. Calculate the update of Q value. SARSA uses the Q value of the current state and the current action, as well as the Q value of the next state S_new and the next action A_new when calculating the Q value. The formula is: Q[S,A]=Q[S,A]+α×(R+γ×argmax(Q[S_new,A_new])-Q[S,A]). In each training round, the agent selects an action based on the current state and moves to the next state S_new after obtaining the reward. Then, in the new state, the agent selects the next action A_new according to the ε-greedy strategy. According to the above Q value update formula, update the corresponding value in the Q table. Repeat this process until the end of the training round. After the training is completed, each row in the Q table represents the Q value of different actions in the current state. The optimal strategy of the agent is to select the action with the largest Q value in each state, that is, select the column index corresponding to the maximum Q value in each row as the best nutritional intervention plan learned by the agent, see. Figure 4 .

[0133] 10) Analyze in the test data set

[0134] According to the agent behavior strategy table output in step 9), the nutritional intervention effect analysis is performed on the test data set in step 3), and the test data is grouped according to whether the nutritional intervention plan in the test data set is consistent with the optimal plan in the Q table. The body mass index and limb skeletal muscle index before and after the intervention are compared between groups according to the grouping information, see Figure 5 .

[0135] As can be seen from the results, the nutritional adjustment strategy obtained by the present invention using offline data combined with SARSA reinforcement learning agent learning has achieved better results than conventional clinical strategies, significantly improving the patient's weight gain and skeletal muscle benefit. Therefore, the present invention is expected to achieve dynamic intelligent adjustment of the nutritional intervention strategy for hospitalized patients, and is expected to significantly improve the weight of hospitalized patients and improve skeletal muscle levels, thereby improving nutrition-related short-term and long-term clinical outcomes. It is worth noting that by analyzing the results of the comparative example two, the present invention can achieve the benefits of different nutritional indicators by matching different state spaces and reward functions, has good flexibility, and can adapt to different clinical diagnosis and treatment purposes and scenarios.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An accurate nutrition decision-making method based on offline reinforcement learning, characterized by: The following steps are involved: Step 1: Collect medical records of hospitalized patients, including general information of patients, disease information, treatment information, nutrition-related anthropometric indicators, hematological indicators, changes in nutrition-related indicators before and after nutritional intervention, specific plans of nutritional intervention, and short-term and long-term clinical outcomes; Step 2: Preprocess the collected medical record data, including deleting outliers, excluding samples with missing key information, interpolating missing data, performing unique-hot encoding on multi-classification data, standardizing continuous variables, deleting variables with the same value for all samples, and adding timestamp identification information based on whether the interaction segment between the agent and the environment has ended; Step 3: Randomly divide the preprocessed medical record data into training set and test set according to a certain ratio; Step 4: Define the patient's nutritional status index as the initial state space, and mark the index as the initial state S in the training and test data; Step 5: Define the type of nutritional intervention plan actually implemented as the agent action space and mark it as action A in the training and test data; Step 6: Define the nutritional status benefit after nutritional intervention as the environmental reward, and select one or more changes in nutrition-related indicators as the calculation basis of the environmental reward function. Express the reward result in numerical form and mark it as reward R in the training and test data; Step 7: Take the same nutritional status index as the initial state space as the new state space and mark it as the new state S_new in the training and test data; Step 8: Set the hyperparameters of offline reinforcement learning, including but not limited to learning rate, discount factor, greed coefficient, number of hidden layers of the neural network, loss function, and training rounds; Step 9: Based on the training set data, according to the parameter settings of steps 4 to 8, an offline reinforcement learning algorithm is used to build a model, and the agent interacts with the environment according to the offline data in the training set until the model converges or reaches the preset number of iterations; Step 10: Evaluate the model performance in the test data set, group the test data according to whether the nutritional intervention plan in the test data set is consistent with the model recommendation strategy, and compare whether there are differences in body weight, limb skeletal muscle index, nutritional risk screening score, nutritional assessment score and 30-day mortality rate indicators before and after nutritional intervention between the inconsistent group and the consistent group.

2. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: The offline reinforcement learning algorithm is Q learning, SARSA or batch constrained Q learning.

3. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: The patient's nutritional status indicators include but are not limited to body weight, limb skeletal muscle index, calf circumference, triceps skinfold thickness, inflammatory status score, nutritional risk screening score, nutritional assessment score, malnutrition diagnosis and severity grading.

4. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: The types of nutritional intervention plans actually implemented include but are not limited to parenteral nutritional intervention, enteral nutritional intervention, combined parenteral and enteral nutritional intervention, nutritional intervention not performed in accordance with the guidelines, and no nutritional intervention at all.

5. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: The nutritional status benefit after the nutritional intervention is calculated based on the changes in one or more nutrition-related indicators before and after a certain length of intervention cycle, and the length of the intervention cycle includes but is not limited to 24 hours, one week, and one month.

6. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: The calculation method of the environmental reward is: R = post-intervention index - pre-intervention index. If the index increases, the reward is a positive number, if the index decreases, the reward is a negative number, and if there is no change, the reward is zero.

7. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: The medical record data collection of hospitalized patients in step 1 also includes patient surgical treatment information and drug treatment information.

8. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: In the step 2, the missing data are interpolated using a multiple interpolation method.

9. The precise nutrition decision-making method based on offline reinforcement learning according to claim 1, characterized in that: The evaluation methods used in step ten include independent sample t-test, rank sum test and chi-square test.

10. A precise nutrition decision-making system based on offline reinforcement learning, characterized by: include: Data collection module: used to collect patients' medical electronic record data, including patients' general information, disease information, treatment information, nutrition-related anthropometric indicators, hematological indicators, changes in nutrition-related indicators before and after nutritional intervention, specific plans for nutritional intervention, and short- and long-term clinical outcomes; Data analysis module: including a reinforcement learning modeling submodule and an individual recommendation data calculation submodule. The reinforcement learning modeling submodule is used to perform offline reinforcement learning-based modeling according to the data obtained by the data acquisition module, so that the intelligent agent can learn to obtain the best nutritional intervention strategy under different nutritional conditions. The individual recommendation data calculation submodule is used to find or calculate the best nutritional intervention plan under the current state according to the intervention strategy obtained by the reinforcement learning modeling submodule and the input individual recommendation data; Strategy output module: includes a strategy display submodule and a personalized recommendation plan submodule. The strategy display submodule is used to display some nutritional intervention plans corresponding to different nutritional status on the screen according to the reinforcement learning modeling results of the reinforcement learning modeling submodule. The personalized recommendation plan submodule is used to display the input patient status and the obtained optimal nutritional intervention plan according to the calculation results of the individual recommendation data calculation submodule and the input patient status.

Images