A sepsis treatment system based on dual-channel reinforcement learning

Abstract

The present invention relates to a sepsis treatment system based on dual-channel reinforcement learning, and belongs to the field of biomedical technology. The system combines a fully connected neural network and a long short-term memory neural network, wherein the fully connected neural network is used to capture the patient's current physiological state, and the long short-term memory neural network is used to represent the patient's treatment history, including state, action and reward; the system includes: a Markov decision module, a dual-channel data stream module, a deep reinforcement learning module, and an offline strategy evaluation module; the Markov decision module constructs the entire disease treatment process as a Markov decision process, selects the patient's historical treatment medical records to construct an input database of the reinforcement learning model, including state values, action values ​​and reward functions. Compared with the prior art, this system can better capture the dynamic changes of the patient's physiological characteristics by learning the patient's history and current physiological information, enrich the learning process of finding the optimal strategy, and has better performance.

Description

Technical Field

[0001] The invention belongs to the field of biomedical technology and relates to a sepsis treatment system based on dual-channel reinforcement learning. Background Art

[0002] Sepsis, also known as sepsis, is a life-threatening disease caused by infection that destroys human organs and is the leading cause of death in patients in the intensive care unit (ICU). It has become one of the deadliest diseases worldwide, with a mortality rate of up to 20%. Currently, intravenous fluids (IV) and maximum vasopressors (VP) are the main drugs for treating sepsis. Unfortunately, there is no universal consensus on how to deal with the timing, dosage or type of drugs. In addition, the different physical constitutions of different sepsis patients often lead to more significant differences in treatment. Even for the same sepsis patient, the therapeutic drugs given at different stages of sepsis are different. Therefore, it remains a major challenge for doctors, even those experienced ones, to develop the best treatment guidelines for sepsis patients.

[0003] To address this challenge, deep reinforcement learning has been applied to develop optimal treatment strategies for sepsis and has achieved remarkable results. Reinforcement learning (RL) technology formalizes the disease treatment process as a sequential decision-making process, learning the best strategy by interacting with the environment or using only previous experience. In recent years, some personalized treatment strategies based on reinforcement learning have been proposed, aiming to provide the best treatment guidelines by dynamically adjusting drug management during the treatment process. For example, some works construct the treatment process as a Markov decision process, and formulate the best treatment strategy based on the patient's physiological information at the current moment. There are also some works that use different encoding methods to encode the patient's historical physiological information, aiming to learn an effective physiological feature. However, these methods only formulate treatment strategies based on the patient's current state or entire medical history. Therefore, it may lead to the loss of data information during the learning process.

[0004] Therefore, it is necessary to focus on how to efficiently construct a rich patient information representation to capture the uncertainty in the patient treatment process. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a sepsis treatment system based on dual-channel reinforcement learning, which constructs a better representation containing rich information to capture the uncertainty of the patient's state by considering the current physiological state and historical decision information.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A sepsis treatment system based on dual-channel reinforcement learning, the system combines a fully connected (FC) neural network and a long short-term memory (LSTM) neural network, wherein the fully connected neural network is used to capture the patient's current physiological state, and the long short-term memory (LSTM) neural network is used to represent the patient's treatment history, including state, action and reward; the system specifically includes: a Markov decision module, a dual-channel data stream module, a deep reinforcement learning module, and an offline strategy evaluation module; the Markov decision module constructs the entire disease treatment process as a Markov decision process, selects the patient's historical treatment medical records to construct an input database of the reinforcement learning model, including state values, action values and reward function; the dual-channel data flow module selects multiple batches of data, and inputs the state value of the current moment into the fully connected (FC) neural network, and the state value, action value and reward value of the previous moment into the long short-term memory (LSTM) neural network to construct a dual-channel data flow mechanism; the deep reinforcement learning module performs decision learning, and better captures the patient's physiological changes at each moment by inputting the data of the dual-channel data flow module; the offline strategy evaluation module evaluates the strategy of the deep reinforcement learning module, reduces the impact of the large Q value variance, and calculates the value of the learning strategy and the unbiased estimate of the average result for the trajectory of each patient in the test set.

[0008] Furthermore, the Markov decision module in the system constructs the entire disease treatment process as a Markov decision process, selects the patient's historical treatment records to construct the input database of the reinforcement learning model, including state values, action values ​​and reward functions; the state value is the 48-dimensional physiological characteristics of the patient collected at each moment of treatment, the action value is the combined use of intravenous infusion IV and maximum vasopressor VP, and IV and VP are dispersed into 5 boxes within a given 4-hour window to obtain a 5×5 discrete action space, and the reward function uses the Sequential Organ Failure Assessment (SOFA) score and the patient's lactate level as the intermediate goals of reinforcement learning optimization to track the patient's physiological health changes over time, while adding a +15 reward to surviving patients and a -15 reward to dead patients at the end of the entire treatment.

[0009] Furthermore, the data used in this system is the patient's medical history, which extracts the patient's physiological characteristics information, treatment strategies and corresponding physiological index change values ​​such as continuous organ failure assessment (SOFA) and lactate level at each moment, and constructs the state value s, action value a and reward function r;

[0010] The reward function is defined as follows:

[0011]

[0012] in and represents the SOFA score and lactate level at time t, C0 = -0.025, C1 = -0.125, C2 = -2; for the final reward, a +15 positive reward is used for the patients who survived in the end, and a -15 negative reward is used for the patients who died.

[0013] Furthermore, the dual-channel data stream module is used to learn the best treatment strategy from the patient's current and historical treatment decision information sequence. Specifically, a fully connected (FC) neural network is used to learn the patient's current state, and a long short-term memory (LSTM) neural network is used to learn the physiological information at time t-1. Finally, the outputs of the two networks are used as the final Q value; assuming that at time t, the state is s t , action is a t , the reward function is r t , then the Q value of the dual-channel DRL is:

[0014] Q(s t ,a t ; θ,μ)=Q f (s t ,a t ;θ)+Q l (s t-1 ,a t-1 ,r t-1 ;μ) (2)

[0015] Where θ and μ represent the weight parameters of the fully connected (FC) neural network and the long short-term memory (LSTM) neural network, respectively.

[0016] Furthermore, in order to accelerate learning, this system uses the priority experience replay technology, which prioritizes the previously observed error samples. The loss function of the network is:

[0017] L(θ)=E[[(Q target -Q(s,a;θ,μ)) 2 ] (3)

[0018] in

[0019] Q target =r+γQ(s′,argmax a' Q(s′,a′;θ,μ);θ′,μ′) (4)

[0020] Where θ and θ′ are the weight parameters of FC, μ and μ′ are the weight parameters of LSTM; in addition, in order to encourage the network model to learn the optimal Q value in each given state, a regularization term λ is added to the loss function of the standard Q network to penalize the reward threshold R max The Q value of the final loss function is:

[0021] L(θ)=E[{(Q target -Q(s,a;θ) 2 ]+λ·|Q(s,a;θ)-R max | (5)

[0022] Where R max = ±15 is the absolute value of the reward or penalty at a terminal time step.

[0023] The beneficial effects of the present invention are:

[0024] Compared with existing technologies, this system can better capture the dynamic changes of patients' physiological characteristics by learning their historical and current physiological information, enrich the learning process of finding the optimal strategy, and its performance is significantly better than other existing technologies.

[0025] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0027] Figure 1 It is a schematic diagram of the framework of the present invention;

[0028] Figure 2 Schematic diagram of the relationship between expected return and mortality risk in the test data. The reward value range is -15 to 15. Higher values ​​indicate higher mortality, and lower values ​​indicate lower mortality.

[0029] Figure 3 This is a comparison of all possible actions taken by different models on the test set. The horizontal and vertical axes represent the drug doses of vasopressors and intravenous infusions, respectively, where 0 represents no administration and 4 represents the maximum dose administered. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0031] The framework of the present invention is as follows Figure 1 The system first preprocesses the data to extract the data information needed, then builds a reinforcement learning model based on a dual-channel data flow mechanism to learn strategies from the data information, and finally uses an offline strategy evaluator to evaluate the learned strategy.

[0032] In this embodiment, the data used are the patient's medical records. The patient's physiological characteristics information, treatment strategies and corresponding physiological indicator change values ​​such as continuous organ failure assessment (SOFA) and lactate levels at each moment are extracted to construct the state value s, action value a and reward function r.

[0033] The reward function is defined as follows:

[0034]

[0035] in and Represents the SOFA score and lactate level at time t, C0 = -0.025, C1 = -0.125, C2 = -2. For the final reward, a +15 positive reward is used for the patient who survives in the end, and a -15 negative reward is used for the patient who dies.

[0036] The dual-channel DRL module proposed in this invention is used to learn the best treatment strategy from the patient's current and historical treatment decision information sequence. Specifically, the FC network is used to learn the patient's current state, and the LSTM network is used to learn the physiological information at time t-1. Finally, the output of the FC and LSTM networks is used as the final Q value. Assume that at time t, the state is s t , action is a t , the reward function is r t , then the Q value of the dual-channel DRL is:

[0037] Q(s t ,a t ; θ,μ)=Q f (s t ,a t ;θ)+Q l (s t-1 ,a t-1 ,r t-1 ;μ) (2)

[0038] Where θ and μ represent the weight parameters of FC and LSTM, respectively.

[0039] To accelerate learning, this system uses a prioritized experience replay technique that prioritizes previously observed error samples. The loss function of the network is:

[0040] L(θ)=E[[(Q target -Q(s,a;θ,μ)) 2 ] (3)

[0041] in

[0042] Q target =r+γQ(s′,argmax a'Q(s′,a′;θ,μ);θ′,μ′) (4)

[0043] Among them, θ and θ′ are the weight parameters of FC, and μ and μ′ are the weight parameters of LSTM. In addition, in order to encourage the network model to learn the optimal Q value in each given state, this system adds a regularization term λ to the loss function of the standard Q network to penalize the reward threshold R max The Q value of . The final loss function is:

[0044] L(θ)=E[{(Q target -Q(s,a;θ) 2 ]+λ·|Q(s,a;θ)-R max | (5)

[0045] Where R max = ±15 is the absolute value of the reward or penalty at a terminal time step.

[0046] In this embodiment, the present invention uses data from the Medical Intensive Care Unit (MIMIC-III V1.4) dataset, which is a freely available data that includes treatment records of patients hospitalized in the ICU from 2001 to 2012. Following the guidelines of previous work, a data cohort that met the Sepsis-3 criteria was generated, and a total of 19,653 patients were obtained. The main extraction criteria included patients aged between 18 and 89 years old and hospital stays of more than 24 hours. In addition, 48-dimensional physiological characteristics of patients were collected, and variables with missing values ​​greater than 20% were removed to ensure the accuracy of data calculation. The medication and mortality of patients during treatment were also recorded. In this work, the focus is on hospital mortality because the mortality rate of patients after discharge may be affected by other unobserved factors.

[0047] The dataset was randomly shuffled and randomly divided into training, validation, and test sets in the ratio of 70%, 10%, and 20%. The architecture of the proposed model consists of recurrent LSTM and FC neural network layers. The LSTM has 1 layer and the number of hidden units is 128. The FC has two hidden layers, each with 128 neurons, and batch normalization is used after the activation function of the previous layer. In addition, the Adam optimizer is used to solve the optimization problem with a learning rate of 0.0001. The batch size and epochs are set to 32 and 60000, respectively.

[0048] In order to find the mapping relationship between different medication strategies and treatment outcomes for patients by clinicians, the present invention uses SARSA to estimate the relationship between the expected return of the doctor's policy and the risk of patient death. Specifically, it is divided into discrete buckets with multiple steps from -15 to +15. The Q values ​​obtained by SRASA on the test set are distributed into discrete buckets, and each Q value is marked as 1 if the patient dies, otherwise it is marked as 0. Finally, as Figure 2 As shown in Table 1, the average mortality rate in each box is calculated to obtain the relationship between the patient mortality rate and the expected outcome. The expected return in the test set is inversely proportional to the patient's risk of death. As expected, the larger the expected value, the lower the chance of death. As shown in Table 1, using Figure 2 The values ​​obtained are used to calculate the average expected returns and corresponding mortality rates based on the four policies.

[0049] Table 1 Comparison of expected returns and estimated mortality rates under SARSA, Dueling DDQN, AD-DDQN and the present invention.

[0050] method Expected Return mortality rate(%) SARSA 7.17 12.12±0.5 Dueling DDQN 10.71 12.28±0.5 AD-DDQN 9.98 12.28±0.5 Our method 14.04 5.45±0.5

[0051] The estimated value of the doctor's policy is 7.17, and the corresponding mortality rate is 12.12%. Dueling DDQN and AD-DDQN can obtain higher expected returns by retaining more patient physiological information in the continuous state space, but the corresponding mortality rate is lower. As can be seen from Table 1, the performance of the present invention is better than other schemes, with an expected return of 14.04 and a mortality rate of 5.45%. Compared with the doctor's policy, the present invention reduces the mortality rate by 6.67%. These results show that the historical treatment information of the patient learned by the present invention combined with the current physiological information can produce higher expected returns.

[0052] Figure 3 The comparison results of all possible actions taken by different models on the test set are shown. The action (0, 0) indicates that the dose of vasopressor and intravenous infusion is 0, and the larger the numerical index value, the higher the corresponding drug dose, where the numerical index value is represented by an integer value of 0-4. Figure 3As shown, the best treatment strategy inferred by the doctor is action (0, 0), that is, no vasopressors and intravenous infusion are given. This result is reliable because the proportion of mild patients is more significant than that of severe patients in the data used. In actual treatment, excessive use of intravenous infusion, especially vasopressors, can cause irreversible damage to patients with sepsis. Only when the patient's condition worsens will the doctor recommend a higher dose of vasopressors to the patient. The best treatment strategy obtained by Dueling DDQN is action (0, 3), while the best drug combination discovered by the present invention is action (0, 3). Both can learn a treatment strategy that does not require vasopressors. These results show that the present invention can make full use of the rich information provided by the state features to infer the optimal strategy close to that of the doctor.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A sepsis treatment system based on dual-channel reinforcement learning, characterized in that: The system combines a fully connected neural network and a long short-term memory neural network, where the fully connected neural network is used to capture the patient's current physiological state and the long short-term memory neural network is used to represent the patient's treatment history, including states, actions, and rewards; The system specifically includes: a Markov decision module, a dual-channel data flow module, a deep reinforcement learning module, and an offline strategy evaluation module; the Markov decision module constructs the entire disease treatment process as a Markov decision process, selects the patient's historical treatment medical records to construct an input database of the reinforcement learning model, including state values, action values, and reward functions; the dual-channel data flow module selects multiple batches of data, and inputs the current state value into a fully connected neural network, and the state value, action value, and reward value of the previous moment into a long short-term memory neural network to construct a dual-channel data flow mechanism; the deep reinforcement learning module performs decision learning, and by inputting the dual-channel data flow module data, better captures the changes in the patient's physiological state at each moment; the offline strategy evaluation module evaluates the strategy of the deep reinforcement learning module, reduces the impact of the large Q value variance, and calculates the value of the learning strategy and the unbiased estimate of the average result for the trajectory of each patient in the test set; The Markov decision module constructs the entire disease treatment process as a Markov decision process, selects the patient's historical treatment medical records to construct the input database of the reinforcement learning model, including state values, action values ​​and reward functions; wherein the state value is the collection of 48-dimensional physiological characteristics of the patient at each moment of treatment, the action value is the combined use of intravenous infusion IV and maximum vasopressor VP, and within a given 4-hour window, IV and VP are dispersed into 5 boxes to obtain a 5×5 discrete action space, and the reward function is to use the sequential organ failure assessment score and the patient's lactate level as the intermediate target of reinforcement learning optimization to track the patient's physiological health changes over time, and at the end of the entire treatment, add a +15 reward to the surviving patients and a -15 reward to the dead patients; The data used by the system is the patient's medical history, which extracts the patient's physiological characteristics information, treatment strategies and corresponding physiological index change values ​​such as continuous organ failure assessment values ​​and lactate levels at each moment, and constructs the state value s, action value a and reward function r; The reward function is defined as follows: in and represents the SOFA score and lactate level at time t, C0 = -0.025, C1 = -0.125, C2 = -2; for the final reward, a +15 positive reward is used for the patients who survived, and a -15 negative reward is used for the patients who died; The dual-channel data stream module is used to learn the best treatment strategy from the patient's current and historical treatment decision information sequence. Specifically, a fully connected neural network is used to learn the patient's current state, and a long short-term memory neural network is used to learn the physiological information at time t-1. Finally, the output of the two networks is used as the final Q value; assuming that at time t, the state is st, the action is at, and the reward function is rt, then the Q value of the dual-channel DRL is: Where θ and μ represent the weight parameters of the fully connected neural network and the long short-term memory neural network, respectively.

2. The sepsis treatment system based on dual-channel reinforcement learning according to claim 1, characterized in that: In order to accelerate learning, this system uses the priority experience replay technology, which prioritizes the previously observed error samples. The loss function of the network is: in Where θ and θ′ are the weight parameters of FC, μ and μ′ are the weight parameters of LSTM; in addition, in order to encourage the network model to learn the optimal Q value in each given state, a regularization term λ is added to the loss function of the standard Q network to penalize the Q value exceeding the reward threshold Rmax. The final loss function is: where Rmax = ±15 is the absolute value of the reward or penalty at a terminal time step.

Images