Personalized drug combination sequence generation method and system

Abstract

The invention discloses a personalized drug combination sequence generation method and system, and relates to the technical field of medical information, and the method specifically comprises the steps: obtaining any medical clinical data; inputting any medical clinical data into a pre-trained drug sequence decision model to obtain a drug combination sequence; the drug sequence decision model is constructed by using a Bayesian inverse reinforcement learning algorithm; and outputting the medicine combination sequence. According to the method, Bayesian reasoning and inverse reinforcement learning are combined under a Bayesian inverse reinforcement learning framework, uncertainty can be processed, posterior distribution of a reward function is deduced, new data are continuously updated and learned, a sepsis treatment guide is modeled into a decision tree to be embedded into the model, and the decision tree is more accurate. The relation between the patient state and the medicine is further identified, favorable support is provided for generation of personalized medicine combination sequences, the working intensity of doctors is reduced, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical information technology, and more particularly to a method and system for generating a personalized drug combination sequence. Background Art

[0002] Sepsis is a systemic inflammatory response syndrome caused by infection, which may lead to organ dysfunction, shock and even death. According to statistics, the incidence of sepsis is increasing year by year, with about 535 new cases per 100,000 people, and is accompanied by a high mortality rate, especially when the disease progresses to the septic shock stage, the patient's chance of survival decreases significantly.

[0003] Faced with sepsis, a serious public health problem, the medical community and researchers have been exploring more effective treatments to improve patient survival rates. In recent years, reinforcement learning (RL), as an artificial intelligence technology, has shown potential application value in the field of sepsis treatment. By learning the patient's status and recommending medication and managing drug dosage based on it, reinforcement learning is expected to achieve personalized medicine and thus improve treatment outcomes.

[0004] However, reinforcement learning faces a series of challenges when applied to sepsis treatment:

[0005] Delayed rewards: The effects of sepsis treatment often take some time to manifest, which means that the reinforcement learning model may have to wait a long time to receive feedback information, which makes the learning process difficult and is not conducive to the rapid optimization of treatment strategies.

[0006] Complex reward function design: The clinical manifestations of sepsis patients are diverse and dynamically changing, so it is very complicated to construct a reward function that can accurately reflect the effectiveness of treatment. An inappropriate reward function may mislead the learning direction of the model and even cause harm to the patient.

[0007] In summary, the treatment of sepsis has received a lot of attention. At present, the treatment strategy for sepsis mainly focuses on the dosage of medication, and there is little research on drug combinations and personalized drug recommendations. Therefore, how to assist in the generation of personalized drug combination sequences is an urgent problem that technicians in this field need to solve. Summary of the invention

[0008] In view of this, the present invention provides a method and system for generating a personalized drug combination sequence, which overcomes the above-mentioned defects.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A method for generating a personalized drug combination sequence, comprising the following specific steps:

[0011] Obtain any medical clinical data;

[0012] Inputting any of the medical clinical data into a pre-trained drug sequence decision model to obtain a drug combination sequence; the drug sequence decision model is constructed using a Bayesian inverse reinforcement learning algorithm;

[0013] The drug combination sequence is output.

[0014] In one embodiment, the training steps of the drug sequence decision model are:

[0015] Acquiring the medical clinical data, and constructing a training data set based on the medical clinical data;

[0016] Acquire a medical treatment strategy, and construct an auxiliary decision tree based on the medical treatment strategy;

[0017] Using the Bayesian inverse reinforcement learning algorithm to construct an initial drug sequence decision model;

[0018] The initial drug sequence decision model is trained using the training data set and the auxiliary decision tree to obtain the drug sequence decision model.

[0019] In one embodiment, the medical clinical data includes: patient demographic characteristics, vital signs, examination results, and medication sequence.

[0020] In one embodiment, the medical clinical data needs to be preprocessed when constructing the training data set, specifically: using machine learning methods to perform data cleaning, missing value processing, and numerical standardization on the medical clinical data.

[0021] In one embodiment, the steps of constructing the auxiliary decision tree are:

[0022] Preprocessing the medical treatment strategy to generate a preprocessing text;

[0023] Extracting drug treatment information based on the preprocessed text;

[0024] Decision points and decision results are determined based on the drug treatment information, and a logical path is constructed to obtain the auxiliary decision tree.

[0025] In one embodiment, the steps of constructing the initial drug sequence decision model are:

[0026] Defining various parameters, including state space, action space, Markov decision process, decision strategy and expert examples;

[0027] Introducing Bayes' theorem into the expert example to obtain likelihood probability;

[0028] Calculate the state value of the state and the action value of the state-action pair under the decision strategy based on the Markov decision process and the decision strategy;

[0029] Modeling the likelihood probability according to the state value and the action value to obtain a likelihood probability model;

[0030] The initial drug sequence decision model is constructed according to the likelihood probability model and the expert examples.

[0031] In one embodiment, the steps of obtaining the drug sequence decision model are:

[0032] Performing posterior distribution sampling on the training data set, and introducing the auxiliary decision tree to perform action selection, to obtain a state-action pair sequence;

[0033] Inputting the state-action pair sequence into the initial drug sequence decision model to calculate the expected reward function;

[0034] Solving the optimal decision-making strategy under the expected reward function by using a policy iteration method;

[0035] The action value function and the optimal reward function under the optimal decision strategy are calculated to generate a drug sequence decision model.

[0036] A personalized drug combination sequence generation system, comprising:

[0037] Data acquisition module, used to obtain any medical clinical data;

[0038] A drug decision module, used for inputting any of the medical clinical data into a pre-trained drug sequence decision model to obtain a drug combination sequence; the drug sequence decision model is constructed using a Bayesian inverse reinforcement learning algorithm;

[0039] An output module is used to output the drug combination sequence.

[0040] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a personalized drug combination sequence generation method and system, which not only combines Bayesian reasoning and inverse reinforcement learning under the Bayesian inverse reinforcement learning framework, can handle uncertainty and derive the posterior distribution of the reward function, continuously update and learn new data, but also models the sepsis treatment guidelines into a decision tree and embeds it into the model, further identifying the relationship between the patient status and the drug, providing favorable support for the generation of personalized drug combination sequences, reducing the workload of doctors and improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A schematic diagram of the method flow provided by the present invention;

[0043] Figure 2 This is a schematic diagram of the model training optimization process provided by the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] This embodiment includes a method for generating a personalized drug combination sequence, such as Figure 1 As shown, the specific steps are:

[0046] Step 1: Obtain any medical clinical data;

[0047] Step 2: Input any medical clinical data into a pre-trained drug sequence decision model to obtain a drug combination sequence; wherein the drug sequence decision model is constructed using a Bayesian inverse reinforcement learning algorithm;

[0048] Step 3: Output the drug combination sequence.

[0049] In one embodiment, the training steps of the drug sequence decision model are:

[0050] Step 21: Acquire medical clinical data, and construct a training data set based on the medical clinical data;

[0051] Step 22: obtaining a medical treatment strategy, and constructing an auxiliary decision tree based on the medical treatment strategy;

[0052] Step 23, constructing an initial drug sequence decision model using a Bayesian inverse reinforcement learning algorithm;

[0053] Step 24: Use the training data set and the auxiliary decision tree to train the initial drug sequence decision model to obtain a drug sequence decision model.

[0054] In one embodiment, the medical clinical data includes: patient demographic characteristics, vital signs, examination results, and medication sequence.

[0055] Furthermore, the medical clinical data come from the clinical data of sepsis patients collected in the international open source medical database, including the patients' demographic characteristics, vital signs, laboratory test results, and related drugs used.

[0056] Furthermore, the medical clinical data are clinical data of sepsis patients 3 days after entering the ICU collected from the international open source medical database, including the patients' demographic characteristics, vital signs, laboratory test results, related drugs used, and final survival status.

[0057] In one embodiment, when constructing a training data set, the medical clinical data also needs to be preprocessed, specifically: using machine learning methods to perform data cleaning, missing value processing, and numerical standardization on the medical clinical data.

[0058] Furthermore, machine learning methods were used to perform data preprocessing operations on medical clinical data, such as cleaning data, processing missing values, and standardizing numerical values, to obtain a complete sepsis patient dataset D. Finally, the sepsis patient dataset D was divided into a training dataset D1 and a test dataset D2 according to an 8:2 ratio.

[0059] Furthermore, we first use machine learning methods to clean the medical clinical data and remove outliers and missing values ​​in the data. Then we label the patient's final survival status: survival is 1 and death is 0. Then we filter out the drugs used by the patients to obtain the drug set: {'medicina_name_1', 'medicina_name_2', ... 'medicina_name_n'}, where medicina_name is the name of the drug, such as vasopressors or amoxicillin antibiotics. Finally, we obtain a complete dataset D of sepsis patients.

[0060] In one embodiment, the steps of constructing the auxiliary decision tree are:

[0061] Step 221, preprocessing the medical treatment strategy to generate a preprocessing text;

[0062] Step 222, extracting drug treatment information based on the preprocessed text;

[0063] Step 223: Determine the decision points and decision results based on the drug treatment information, and construct a logical path to obtain an auxiliary decision tree.

[0064] Furthermore, the medical treatment strategy comes from the sepsis treatment guidelines. The steps of obtaining the preprocessed text are: converting the electronic document of the sepsis treatment guidelines (from PDF to text format), removing irrelevant content (title, author-related information, header, footer, index, etc.), breaking the text, etc., to obtain the preprocessed text M;

[0065] The specific steps of drug treatment information extraction are as follows: split the preprocessed text M into paragraphs for analysis, determine the keywords and phrases used to identify drug treatment information, such as "antibiotics", "dosage", "treatment duration", "indications", "contraindications", etc., and then use natural language processing technology - named entity recognition (NER) method to identify relevant information about drug treatment from the text, including medication conditions, drug name, recommended dosage, route of administration, treatment duration, precautions, etc.

[0066] The specific steps of auxiliary decision tree construction are:

[0067] Based on the extracted information, the decision points (such as severity of the disease, patient age, etc.) and possible decision results (such as specific drug treatment plans) of the decision tree are defined. According to the recommendations in the treatment guidelines, a logical path from the decision point to the decision result is constructed. Each path represents a specific treatment process. Then the decision tree is visualized to ensure that each node of the tree clearly represents a decision point or decision result.

[0068] In another embodiment, the accuracy and completeness of the auxiliary decision tree is verified by medical experts to ensure that all treatment recommendations are consistent with the latest sepsis treatment guidelines, and the final auxiliary decision tree E is obtained.

[0069] In one embodiment, the steps for constructing the initial drug sequence decision model are:

[0070] Step 231, defining various parameters, including state space, action space, Markov decision process, decision strategy and expert examples;

[0071] Step 232, introducing Bayesian theorem into the expert example to obtain likelihood probability;

[0072] Step 233: Calculate the state value of the state and the action value of the state-action pair under the decision strategy based on the Markov decision process and the decision strategy;

[0073] Step 234: Model the likelihood probability according to the state value and the action value to obtain a likelihood probability model;

[0074] Step 235: construct an initial drug sequence decision model based on the likelihood probability model and expert examples.

[0075] Furthermore, first, we define the state space S and action space A. The state space includes the demographic characteristics, vital signs, and laboratory test results of sepsis patients. The action space is the patient's medication sequence. Based on the drug set, if a drug is used, it is marked as 1, and if it is not used, it is marked as 0. Then we define the Markov decision process M: {s, a, T, γ, R}, where s is the state, s∈S; a is the action, a∈A; T is the state transition probability, which indicates the probability that the current patient state s is converted to the next state s': T ss' =[S t+1 =s'S t =s], γ is the reward discount rate, which is used to measure the importance of future rewards to current rewards. Its range is between 0 and 1. The larger γ is, the higher the weight of future rewards is. R is the reward function, which represents the immediate reward obtained after taking an action in a certain state. The reward function of inverse reinforcement learning is obtained through learning. The decision strategy π is defined again, which represents the rule for the agent to choose actions in a specific state. The strategy can be divided into stochastic strategy and deterministic strategy. The stochastic strategy maps each state to the probability distribution of an action. In state s, the strategy π(a|s) is the probability of taking action a in this state: π(a|s)=P(A t =a|S t =s), where A t is the action at time t, S t is the state at time t; in the present invention, a random strategy is used.

[0076] Then, define the expert example:

[0077] Ox={(s1,a1)(s2,a2)...(s k ,a k )};

[0078] In the formula, x is the expert agent, (s k ,a k ) is the state-action pair of the expert; according to Bayes' theorem:

[0079]

[0080] Among them, Prx(Ox|R) is the likelihood probability, P R (R) is the prior distribution of rewards, Pr(Ox) is the marginal likelihood function of the expert example Ox;

[0081] According to the above formula, the expression of likelihood probability can be obtained as follows:

[0082] Prx(OxR)=Prx((s1,a1)R)Prx((s2,a2)R)...Prx((s k,a k )R);

[0083] According to the Bellman Equations, for the Markov decision process M = {s, a, T, γ, R} and the decision strategy π:

[0084]

[0085] Where V π (s) is the state value corresponding to state s in the decision strategy π, Q π (s, a) is the action value under state s and action a in the decision strategy π, s' is the next state of state s, and R(s) is the reward function value under state s. According to the above formula, the larger the cumulative reward value corresponding to (s, a), the larger the corresponding Q value, so the exponential distribution is used to model the likelihood probability and obtain the probability model:

[0086]

[0087] In the formula, α is the confidence parameter, that is, the more confident the expert is that the action is selected based on the maximum reward value, the larger the value, Q*(s i ,a i ,R) is the state s i The maximum Q value under ; Substituting into the likelihood probability expression, we get:

[0088]

[0089] Where, E(Ox,R)=∑ i Q * (s i ,a i ,R), the posterior probability is as follows:

[0090]

[0091] Where Z' is the normalization constant;

[0092] Then take the value extracted from the posterior distribution as the true value and the mean of the posterior distribution as the estimated value to obtain the loss function:

[0093]

[0094] Where V * (R) is the vector corresponding to the optimal value of each state in the optimal decision strategy under reward R, V π (R) is the vector corresponding to the optimal value of each state in the decision strategy π that minimizes the loss in the posterior distribution under reward R, and p is the norm;

[0095] After obtaining the initial drug sequence decision model, maximizing the average reward can find the optimal decision strategy, thereby minimizing the loss function to obtain the optimal drug sequence decision model.

[0096] In one embodiment, the steps for obtaining the drug sequence decision model are:

[0097] Step 241, sampling the posterior distribution of the training data set, and introducing an auxiliary decision tree for action selection to obtain a state-action pair sequence;

[0098] Step 242, inputting the state-action pair sequence into the initial drug sequence decision model to calculate the expected reward function;

[0099] Step 243, solving the optimal decision strategy under the expected reward function through a policy iteration method;

[0100] Step 244: Calculate the action value function and the optimal reward function under the optimal decision strategy to generate a drug sequence decision model.

[0101] Furthermore, first, the Monte Carlo method is used to sample the training set D1 and initialize the reward R, and then the Q value is calculated based on the state, action and auxiliary decision tree in the calculation sample;

[0102] Then, the strategy is iterated and updated until the Q value reaches the maximum, and the optimal reward function R and loss function are obtained.

[0103] Furthermore, in this step, the prior distribution and the likelihood function are combined to obtain the posterior distribution of the reward function, which is expressed as:

[0104] P(R|Ox)∝P(Ox|R)P(R);

[0105] like Figure 2 As shown, the Monte Carlo method is used to sample the posterior distribution of the training set D1 to obtain the state-action pair sequence Ox. When selecting the corresponding drug according to the patient's state, an auxiliary decision tree E is added to standardize the action selection; then N reward functions R are obtained. i , the expected value of the reward function is: Solving the expected reward function through policy iteration method The optimal policy strategy π* under , the solution process is as follows:

[0106] (1) Randomly initialize the decision strategy π0, state value function V(s), and action value function Q(s,a);

[0107] (2) For state s∈S, calculate the current decision strategy π k The state value function V πk (s);

[0108] (3) Use the Bellman equation to iteratively update until V πk (s) converges, and the expression is:

[0109]

[0110] (4) Calculate the action value function Q πk (s,a):

[0111]

[0112] (5) For each state s, select the action a that maximizes the Q value as the new strategy, and the strategy is updated as:

[0113]

[0114] (6) If Then convergence is achieved, the Q value reaches the maximum, and the optimal decision strategy π*=π is obtained k+1 , stop updating; if Then let k←k+1, return to step (2), and continue to update the strategy.

[0115] Under the optimal decision strategy π*, calculate the corresponding optimal action value function Q π* (s,a):

[0116]

[0117] Where P(s'|(s,a)) is the probability of the next state s' when the current state action is (s,a); is the expected reward value of the current state action (s, a) and the next action s'; V π* (s') is the optimal state value function, which satisfies the Bellman optimal equation, and its expression is:

[0118]

[0119] Finally, according to the optimal Q value and the optimal reward function R, the optimal object sequence decision model is obtained.

[0120] In one embodiment, the clinical data of the patient collected are as follows: {'age','weight','gender','mechvent','sirs','sofa_score','lactate','pco2','ph','po2','hemato crit','hemoglobin','platelet','albumin','bun','calcium','chloride','creatinine','glucose','potassium','sodium','inr','pt','ptt','gcs','outputevents','dbp','mbp','sbp','spo2','temperatu re','shock_index','heart_rate',medicine'}, patient clinical data includes patient demographic characteristics ('age','weight','gender'), clinical indicators ('pco2','ph','po2', etc.) and medication information ('medicine'); the patient training set is divided into state space and action space as the input of the drug decision module for model training, the state space includes the patient's demographic characteristics and clinical indicators, and the action space includes the patient's medication information; after the training, the drug decision module can generate a personalized drug combination sequence {'drug_1','drug_2'}.

[0121] On the other hand, this embodiment discloses a personalized drug combination sequence generation system, including:

[0122] Data acquisition module, used to obtain any medical clinical data;

[0123] The drug decision module is used to input any medical clinical data into a pre-trained drug sequence decision model to obtain a drug combination sequence; the drug sequence decision model is constructed using the Bayesian inverse reinforcement learning algorithm;

[0124] The output module is used to output the drug combination sequence.

[0125] The present invention discloses a personalized drug combination sequence generation method and system. Under the Bayesian inverse reinforcement learning framework, the Bayesian reasoning and inverse reinforcement learning are combined, the uncertainty can be processed, and the posterior distribution of the reward function can be derived. By continuously updating and learning new data, the drug combination sequence can be dynamically adjusted. Doctors can make dynamic treatment strategy recommendations for patients based on the generated drug combination sequence to improve the survival rate of patients. The present invention also models the sepsis treatment guideline into a decision tree and embeds it into the model, which helps to identify the relationship between the patient's status and the drug, so as to generate a personalized drug combination sequence, so that doctors can provide personalized treatment suggestions based on the patient's specific situation and optimize the treatment effect.

[0126] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0127] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a personalized drug combination sequence, characterized in that: The specific steps are: Obtain any medical clinical data; Inputting any of the medical clinical data into a pre-trained drug sequence decision model to obtain a drug combination sequence; the drug sequence decision model is constructed using a Bayesian inverse reinforcement learning algorithm; The drug combination sequence is output.

2. A method for generating a personalized drug combination sequence according to claim 1, characterized in that: The training steps of the drug sequence decision model are: Acquiring the medical clinical data, and constructing a training data set based on the medical clinical data; Acquire a medical treatment strategy, and construct an auxiliary decision tree based on the medical treatment strategy; Using the Bayesian inverse reinforcement learning algorithm to construct an initial drug sequence decision model; The initial drug sequence decision model is trained using the training data set and the auxiliary decision tree to obtain the drug sequence decision model.

3. A method for generating a personalized drug combination sequence according to claim 1 or 2, characterized in that: The medical clinical data include: patient demographic characteristics, vital signs, examination results and medication sequence.

4. A method for generating a personalized drug combination sequence according to claim 2, characterized in that: When constructing the training data set, the medical clinical data also needs to be preprocessed, specifically: using machine learning methods to perform data cleaning, missing value processing and numerical standardization on the medical clinical data.

5. The method for generating a personalized drug combination sequence according to claim 2, characterized in that: The construction steps of the auxiliary decision tree are: Preprocessing the medical treatment strategy to generate a preprocessing text; Extracting drug treatment information based on the preprocessed text; Decision points and decision results are determined based on the drug treatment information, and a logical path is constructed to obtain the auxiliary decision tree.

6. A method for generating a personalized drug combination sequence according to claim 2, characterized in that: The steps for constructing the initial drug sequence decision model are: Defining various parameters, including state space, action space, Markov decision process, decision strategy and expert examples; Introducing Bayes' theorem into the expert example to obtain likelihood probability; Calculate the state value of the state and the action value of the state-action pair under the decision strategy based on the Markov decision process and the decision strategy; Modeling the likelihood probability according to the state value and the action value to obtain a likelihood probability model; The initial drug sequence decision model is constructed according to the likelihood probability model and the expert examples.

7. A method for generating a personalized drug combination sequence according to claim 6, characterized in that: The steps for obtaining the drug sequence decision model are: Performing posterior distribution sampling on the training data set, and introducing the auxiliary decision tree to perform action selection, to obtain a state-action pair sequence; Inputting the state-action pair sequence into the initial drug sequence decision model to calculate the expected reward function; Solving the optimal decision-making strategy under the expected reward function by using a policy iteration method; The action value function and the optimal reward function under the optimal decision strategy are calculated to generate a drug sequence decision model.

8. A personalized drug combination sequence generation system, characterized in that: include: Data acquisition module, used to obtain any medical clinical data; A drug decision module, used for inputting any of the medical clinical data into a pre-trained drug sequence decision model to obtain a drug combination sequence; the drug sequence decision model is constructed using a Bayesian inverse reinforcement learning algorithm; An output module is used to output the drug combination sequence.

Images