A learning method and device for sepsis treatment strategy

Abstract

The present invention discloses a learning method and device for sepsis treatment strategy, the method comprising: establishing a Markov decision model for sepsis treatment strategy based on sample data; the sample data is pathological data of sepsis patients within a preset time; according to the Markov decision model, establishing a sepsis mortality prediction model, and obtaining the mortality weight of each characteristic of sepsis patients; labeling the sample data using the mortality weight to generate a target sample; predicting sepsis treatment strategy, and using a deep reinforcement learning method to learn the treatment strategy. The present invention labels the patient sample priority weights, so that the sample quality of deep intensity learning is higher, the training effect of the model is improved, and the problem of quickly falling into the local optimum in a large number of suboptimal samples is avoided; the treatment strategy is learned by adopting an improved deep intensity learning algorithm, and realistic restrictions are added to the action selection of the AI ​​treatment strategy, so that the recommended treatment strategy is better.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and medical technology, and in particular to a method and device for learning sepsis treatment strategies. Background Art

[0002] In recent years, the use of artificial intelligence (AI) algorithms to explore optimal treatment strategies for intravenous fluids (IV) and vasopressors (VP) in patients with sepsis has become increasingly widespread. When evaluating sepsis treatment strategies using AI algorithms, offline estimators, such as the Importance Sampling (IS) and Dual Robust Estimators (DR) combined with the SARSA algorithm, are primarily used to assess the expected return of physician treatment strategies. Exploring AI treatment strategies often involves several approaches, including discrete state-space models and policy iteration, deep neural networks and value iteration, hybrid models, and continuous state-space models.

[0003] However, these methods, which utilize deep reinforcement learning models to learn treatment strategies based on discrete data, fail to account for the uneven distribution of patient sample quality. Due to this imbalance in sample quality and distribution, the model can quickly become trapped in a local optimum by learning from a large number of suboptimal samples, without adequately learning from the smaller proportion of high-quality samples. Consequently, the treatment recommendations provided by the model after learning often deviate significantly from reality. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and device for learning sepsis treatment strategies to solve the problem that when existing deep reinforcement learning models learn sepsis data, the quality and distribution of samples are uneven, which affects the learning effect and leads to a large deviation between the learning results and the actual results.

[0005] To achieve the above objectives, the present invention provides a method for learning sepsis treatment strategies, comprising:

[0006] Establishing a Markov decision model for sepsis treatment strategy based on sample data; the sample data is pathological data of sepsis patients within a preset time;

[0007] Based on the Markov decision model, a sepsis mortality prediction model is established, and the mortality weights of various characteristics of sepsis patients are obtained;

[0008] Using the mortality weight to mark the sample data to generate a target sample;

[0009] Predicting sepsis treatment strategies and using deep reinforcement learning methods to learn treatment strategies.

[0010] Furthermore, after establishing the Markov decision model for sepsis treatment strategy based on the sample data, the method further includes:

[0011] The data in the Markov decision model were cleaned, including removing treatment trajectories whose data loss exceeded the preset value and filling the data using the KNN algorithm.

[0012] Furthermore, based on the Markov decision model, a sepsis mortality prediction model is established, and the mortality weights of various characteristics of sepsis patients are obtained, including:

[0013] Based on the Markov decision model, the extreme gradient boosting algorithm is used for binary classification training, with the patient's status in each time series as the feature value and the final survival status as the label to train and generate a mortality prediction model;

[0014] Based on the mortality prediction model, patients were divided into groups according to whether they ultimately survived or died. The KNN algorithm was used to calculate the mortality weight of each patient characteristic based on the Minkowski distance formula.

[0015] Furthermore, the deep reinforcement learning method is a Double Dueling DQN deep reinforcement learning method.

[0016] Furthermore, the use of deep reinforcement learning methods to learn treatment strategies includes modifying the loss function and adding realistic constraints to the action selection of the AI ​​treatment strategy.

[0017] The present invention also provides a method for learning a sepsis treatment strategy, comprising:

[0018] Establishing a Markov decision model for sepsis treatment strategy based on sample data; the sample data is pathological data of sepsis patients within a preset time;

[0019] Based on the Markov decision model, a sepsis mortality prediction model is established, and the mortality weights of various characteristics of sepsis patients are obtained;

[0020] The mortality weight is used to label the sample data to generate a target sample, and the target sample is used as a learning sample for the sepsis treatment strategy.

[0021] Furthermore, after establishing the Markov decision model for sepsis treatment strategy based on the sample data, the method further includes:

[0022] The data in the Markov decision model were cleaned, including removing treatment trajectories whose data loss exceeded the preset value and filling the data using the KNN algorithm.

[0023] Furthermore, based on the Markov decision model, a sepsis mortality prediction model is established, and the mortality weights of various characteristics of sepsis patients are obtained, including:

[0024] Based on the Markov decision model, the extreme gradient boosting algorithm is used for binary classification training, with the patient's status in each time series as the feature value and the final survival status as the label to train and generate a mortality prediction model;

[0025] Based on the mortality prediction model, patients were divided into groups according to whether they ultimately survived or died. The KNN algorithm was used to calculate the mortality weight of each patient characteristic based on the Minkowski distance formula.

[0026] The present invention also provides a device for learning sepsis treatment strategies, comprising:

[0027] A data processing unit, configured to establish a Markov decision model for sepsis treatment strategies based on sample data; the sample data being pathological data of sepsis patients within a preset time period;

[0028] a weight calculation unit, configured to establish a sepsis mortality prediction model based on the Markov decision model, and obtain mortality weights of various characteristics of sepsis patients;

[0029] a marking unit, configured to mark the sample data using the mortality weight to generate a target sample;

[0030] A learning unit is used to predict sepsis treatment strategies and learn treatment strategies using deep reinforcement learning methods.

[0031] The present invention also provides a device for learning sepsis treatment strategies, comprising:

[0032] A data processing module, configured to establish a Markov decision model for sepsis treatment strategies based on sample data; the sample data being pathological data of sepsis patients within a preset time period;

[0033] A weight calculation module is used to establish a sepsis mortality prediction model based on the Markov decision model and obtain the mortality weight of each characteristic of sepsis patients;

[0034] The sample generation module is used to mark the sample data using the mortality weight to generate a target sample, and use the target sample as a learning sample for the sepsis treatment strategy.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) Based on the sample weights, the sampling rate of the model is adjusted to prioritize high-quality samples, thereby improving the model's training effect and avoiding rapid regression into local optimality among a large number of suboptimal samples. This is especially important for providing more accurate treatment recommendations for patients with severe conditions who have effective treatment options.

[0037] 2) Based on realistic offline data, a distribution restriction framework for medical actions is proposed, allowing AI models to learn more effective treatment strategies under more realistic conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 is a flowchart of a method for learning a sepsis treatment strategy provided by an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of prediction results of a mortality prediction model provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of comparing the evaluation results of learning effects of different models provided by an embodiment of the present invention;

[0042] Figure 4 is a flowchart of a method for learning a sepsis treatment strategy provided by another embodiment of the present invention;

[0043] Figure 5 is a schematic structural diagram of a device for learning sepsis treatment strategies provided by an embodiment of the present invention;

[0044] Figure 6 FIG. 4 is a schematic structural diagram of a device for learning sepsis treatment strategies provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are to be executed.

[0047] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0049] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0050] See also Figure 1 , a certain embodiment of the present invention provides a method for learning sepsis treatment strategies. Figure 1 As shown, the method for learning a sepsis treatment strategy includes steps S10 to S40. The specific steps are as follows:

[0051] S10. Establishing a Markov decision model for sepsis treatment strategy based on sample data; the sample data is pathological data of sepsis patients within a preset time period;

[0052] S20. Establishing a sepsis mortality prediction model based on the Markov decision model, and obtaining mortality weights of various characteristics of sepsis patients;

[0053] S30, labeling the sample data using the mortality weight to generate a target sample;

[0054] S40. Predict sepsis treatment strategies and use deep reinforcement learning methods to learn treatment strategies.

[0055] To help understand the difference between this embodiment and the prior art, we first explain the approach used in the prior art: For learning sepsis treatment strategies, the current data processing method is primarily through MDP modeling. This involves obtaining demographic data, vital signs, medical indicator laboratory values, and intravenous fluids and vasopressor medications received from a public clinical database (MIMIC-III) for sepsis patients from the time of diagnosis and within 72 hours of their stay in the ward. Patient data is aggregated into 18 time-ordered sequences using a 4-hour time window. The evaluation of AI-based treatment strategies primarily utilizes offline estimators such as the Importance Sampling Estimator (IS) and the Doubly Robust Estimator (DR). The SARSA algorithm is used to calculate the expected return of the physician's strategy and to fit the relationship between the expected return and mortality rate, thereby indirectly confirming the correlation between high Q values ​​and low mortality rates.

[0056] Among them, the current technical solutions for exploring AI treatment strategies mainly include the following:

[0057] A) Method based on discrete state space models and policy iteration: This approach directly applies the on-policy SARSA algorithm and model-based policy iteration to the patient's discrete state space and behavior space. This discretizes the state space into clusters, reducing the model's interpretability and impacting the clinical applicability of personalized treatment for patients.

[0058] B) Using a deep neural network and value iteration: This approach uses a Dueling Double Deep Q network model. Patient states are encoded using an autoencoder and then fed into the neural network. This implicitly assumes the state space is continuous and does not directly model state-action transition probabilities. The optimal policy is obtained through value iteration. This can lead to overfitting the model to the underlying dataset and overestimating the Q values ​​of some state-action pairs.

[0059] C) Hybrid model approach. The AI ​​policy is trained using the same method as in step (B). The doctor policy is formed by using the k-nearest neighbor method to obtain the probability of the doctor's treatment action for the patient's state at a nearby distance. The AI ​​policy's action selection is constrained based on factors such as the probability of the doctor's action occurring. For patient treatment actions, it is recommended to switch between the doctor's policy and the AI ​​policy base. Constraining AI actions based on the probability of the doctor's action occurring in the patient's state does not prove that the doctor's action is more effective than the AI.

[0060] D) Establish a continuous state space model and use reinforcement learning strategies to explore the model space to obtain better treatment strategies, etc.

[0061] However, the aforementioned methods, which use deep reinforcement learning to learn treatment strategies from offline sepsis patient data, do not consider the actual uneven distribution of patient sample quality, which can lead to deviations between the model's treatment recommendations and reality. In most cases, patients with good conditions can be effectively cured with appropriate treatment. However, in some extreme cases, the actual treatment plan chosen is worthless in learning and may even have a negative impact. For example:

[0062] Case A: The patient's condition is extremely mild and can be cured even with poor treatment methods. In this case, the doctor's actions are given an inflated value that is inconsistent with the facts.

[0063] Case b: The patient's condition is extremely serious and cannot be cured even with the most ideal treatment. In this case, the doctor's actions are given a value that is inconsistent with the facts and is too low.

[0064] Furthermore, there are smaller but more valuable patient samples that are not being learned more effectively:

[0065] Case C: The patient is in critical condition but is eventually saved through effective treatment. In this case, the doctor’s action is assigned the correct value, but due to the scarcity of samples, it is not fully learned by the model.

[0066] Case d: The patient is in a curable condition, but dies due to incorrect treatment. In this case, the doctor's action is assigned a correct negative value, but again, due to the small sample size, it is not fully learned by the model.

[0067] Therefore, when using deep reinforcement learning models to learn optimal treatment strategies based on discrete data, two problems arise. The first is that learning from samples of situations a and b may actually lead to underestimating the value of medical actions. In reality, patient samples of these situations are common (13.5% of 17,331 patient trajectory samples ultimately died). On the other hand, the proportion of samples of situations c and d is relatively small, resulting in the model not fully learning these high-quality samples. Furthermore, due to the imbalance in sample quality and distribution, the model may quickly fall into a local optimum by learning from a large number of suboptimal samples, without fully learning from the smaller proportion of high-quality samples. The second is that treatment strategies based solely on AI may propose treatment actions that deviate from reality and may cause serious harm to patients (such as the excessive use of pressor drugs). A purely model doctor strategy is overly conservative, preventing AI from effectively exploring and proposing more effective treatment strategies.

[0068] Therefore, to address the above issues, this embodiment preferentially provides a method for learning sepsis treatment strategies. Before executing step S10, sample data must first be obtained. In this embodiment, sepsis patient data is obtained using the MIMIC-III intensive care unit database. According to the Sepsis-3 definition, all demographic data (height, age, gender, etc.), vital signs, laboratory data, intravenous infusion, vasopressor use data, etc., are obtained for the sepsis patient within 72 hours before diagnosis and 48 hours after diagnosis, and the patient's final survival status is obtained.

[0069] Furthermore, step S10 is performed to establish a Markov decision model for sepsis treatment strategies based on the sample data. In this embodiment, the preprocessing and encoding process is mainly based on the sample data. The sequential decision problem for the optimal sepsis treatment strategy is modeled as a Markov decision model, defined as a quintuple (S, A, T, Y, R), specifically:

[0070] Patient state space (S): The patient's demographic data, vital signs data, and laboratory test data are divided into 18 time series by 4 hours. These 18 time series constitute the patient's complete treatment trajectory, and the patient's data in each time series represents the patient's state.

[0071] Medical Action Space (A): Similarly, the actual treatment actions received by the patient are divided into 4-hour intervals. Intravenous infusion is the total amount of vasopressors used within the 4-hour period, and vasopressors are the maximum amount of vasopressors used within the 4-hour period. Intravenous infusion and vasopressors are divided into 4 actions based on the amount of medication used. Adding actions without medication, the total number of actions is 25.

[0072] State transfer matrix (T): transfer function P(s t+1 |s t ,a t ), giving the transition probability of the state according to the current state and action. By embedding the state or state into a distributed representation using a deep neural network, we implicitly assume that the state space is continuous and do not directly model the transition probability matrix.

[0073] Reward (R): Provides a sparse reward of +15 or -15 at the end of the treatment trajectory, depending on whether the patient survives or not.

[0074] In one embodiment, after executing step S10, the method further includes:

[0075] The data in the Markov decision model were cleaned, including removing treatment trajectories whose data loss exceeded the preset value and using the KNN algorithm for data filling.

[0076] It should be noted that in this example, after completing data modeling, patient trajectories with an average missing data value exceeding 20% ​​or a specific data loss exceeding 75% were removed. The missing data in the patient state space features were filled using the K-nearest neighbor (KNN) algorithm, with k = 3. The patient trajectories were then split into a training set and a test set with an 8:2 ratio. Both the training and test sets were then encoded using a sparse autoencoder.

[0077] Furthermore, step S20 is performed to obtain the mortality weight of each characteristic of the sepsis patient. In a specific embodiment, step S20 further includes:

[0078] 2.1) Based on the Markov decision model, a binary classification training was performed using the extreme gradient boosting algorithm, with the patient's status in each time series as the feature value and the final survival status as the label, to train and generate a mortality prediction model;

[0079] 2.2) Based on the mortality prediction model, patients were divided into groups according to their eventual survival or death. The KNN algorithm was used to calculate the mortality weight of each patient characteristic based on the Minkowski distance formula.

[0080] In this embodiment, the training set data of the patient trajectory (80% of the patient data) is first randomly divided into a training set and a test set in a ratio of 7:3. The training set is used to establish a death prediction model, and the test set is used to verify the accuracy of the algorithm. The extreme gradient boosting algorithm (XGBOOST) is used for binary classification training. The patient's state in each time series is used as the feature value and the final survival status is used as the label to train the mortality prediction model. The relative weight of the feature for predicting mortality is then obtained, which is recorded as Wd. The curve of the prediction result is shown in the figure below. Figure 2 shown.

[0081] Specifically, the optimal parameters for the Extreme Gradient Boosting (XGB) model were set to: maximum depth 1, learning rate 0.1, regularization term 5, and minimum loss function drop gamma required for splitting (gamma = 0.1). The final AUC was 0.841, with a mortality prediction accuracy of 0.883 and a recall of 0.187.

[0082] Furthermore, step S30 is performed to identify the sample quality and mark the weight. Specifically, in this embodiment, the patients in the training set are divided according to whether they survive or die, and are recorded as data_a and data_d. Using the KNN algorithm, based on Minkowski Distance:

[0083]

[0084] Let n = 100, P = 2, ki The mortality weights of each patient characteristic calculated in step 2 include:

[0085] (1) Calculate the average distance between the state of the surviving patient and the 100 states closest to the patient who eventually died;

[0086] (2) Calculate the average distance between the state of the deceased patient and the 100 states closest to the patient who survived.

[0087] Then, the distances obtained from (1) and (2) are summed up. The closer the distance, the closer the current state (the state of the surviving patient) is to death, but eventually survives after effective treatment; or the current state (the state of the deceased patient) is closer to survival, but eventually dies due to improper treatment. That is, the closer the distance, the higher the quality of the patient sample; conversely, the larger the distance, the lower the quality of the representative sample. According to the above content, set the sampling weight W of any transfer sample i for

[0088]

[0089] Among them, distance_mean i is the average value of the distance, k is the weight amplification parameter, and the sampling priority is proportional to the weight.

[0090] Further, step S40 is executed: predicting sepsis treatment strategy. In this step, the KNN algorithm is mainly used, and k=300. Then, in the training set and the test set, find the patient status s i The closest other 300 patient states are used to count the doctor's actions under these 300 states. Based on this, the doctor's treatment strategy π is summarized according to the action distribution. p It should be emphasized that 300 here is only a preferred way to select the value of k, which can be adjusted according to actual needs in application and is not limited here.

[0091] Finally, a deep reinforcement learning method is used to learn the treatment strategy. In this embodiment, the deep reinforcement learning method is preferably the Double Dueling DQN deep reinforcement learning method. When using the deep reinforcement learning method to learn the treatment strategy, it also includes modifying the loss function and adding realistic constraints to the action selection of the AI ​​treatment strategy.

[0092] Specifically, we first modify the loss function to remove the Q-value clipping of the target network. Compared with the classic implementation, the Q-value output of the target network is clipped during model training (limited to [-15, 15]). The loss function is:

[0093]

[0094] in:

[0095] Q double-target =r+γQ(s,argmax a′ Q(s,a′;θ);θ′);

[0096] It should be noted that if the model does not add Q-value clipping and L2 regularization, the expected return (expectreturn) cannot converge and monotonically increases; after adding it, it is easy to fall into the local optimum before sufficient learning. Because by clipping the target network Q value and adding the L2 regularization term to the loss function, the information previously accumulated to distinguish the next state is lost, which weakens the network's ability to predict better behavior. In fact, it causes the model's expected return to almost lose its exploration ability after reaching the critical value (+15). Therefore, this embodiment uses the Huber loss function to calculate the difference between Q_(double-target) and Q(s,a;θ), because Huber is more robust to extreme values ​​and outliers than the mean square error, and the gradient will gradually decrease as the loss value approaches its minimum value, making it more accurate. Therefore, the loss function is designed to be:

[0097]

[0098] Here k = 1.5, and the output of the target network is no longer clipped, which improves the network's ability to predict better behaviors.

[0099] Secondly, realistic constraints are added to the action selection of the AI ​​treatment strategy, using the π obtained in the above embodiment. p , during the training process of the model, the patient status s i The optimal action a proposed k Make a judgment, if π p (a k |s i ) ≥ 0, then the action is optional. Otherwise, the suboptimal action proposed by the model is selected, and the action selection is determined again, and so on. The actions selected by the AI ​​are controlled to explore under realistic and feasible conditions.

[0100] Finally, set the parameters and conduct formal training:

[0101] Hyperparameter settings: learning rate of 0.0001, 10 hidden layers, 1024 neurons per layer, batch normalization and Leaky-ReLU activation, training with a batch size of 256. The input has 46 nodes, each node corresponds to a state space feature, and the output layer has 25 nodes, each node corresponds to the Q value of each possible action in the action matrix.

[0102] Random model setting: Randomly obtain samples to train the model. As the comparison model of this experiment, train 250,000 epochs.

[0103] Dis_priority model: Prioritized experience replay is used. According to the weights in step 3, important samples are prioritized for training the model for 250,000 epochs. The sampling probability of transfer sample i is:

[0104]

[0105] Among them, p i is the sample weight calculated in step 3, Used to adjust the priority of the connection.

[0106] In one embodiment, after executing step S40, the model effect is further evaluated. Finally, the expected return of the model trained by sample importance sampling converges to around 16, and the expected return of the model trained by random sampling converges to around 14. Figure 3 As shown in the figure, the two models were compared in the test set. The average expected return for the dis_priority model was 16.75, while the average expected return for the random model was 14.61. The WIS and WDR scores for the doctor strategies in the test set were calculated, and the WIS and WDR scores for the two models in the strategy set were compared to generate the treatment strategy evaluation results.

[0107] In summary, this embodiment marks patient samples by priority weights, which makes the sample quality of deep intensity learning higher, improves the training effect of the model, and avoids the problem of quickly falling into local optimality in a large number of suboptimal samples; by adopting an improved deep intensity learning algorithm to learn treatment strategies and adding realistic restrictions to the action selection of AI treatment strategies, the recommended treatment strategies are better.

[0108] See also Figure 4 A certain embodiment of the present invention further provides a method for learning a sepsis treatment strategy, comprising:

[0109] S50, establishing a Markov decision model for sepsis treatment strategy based on sample data; the sample data is pathological data of sepsis patients within a preset time;

[0110] S60. Establishing a sepsis mortality prediction model based on the Markov decision model, and obtaining mortality weights of various characteristics of sepsis patients;

[0111] S70: Use the mortality weight to label the sample data to generate a target sample, and use the target sample as a learning sample for sepsis treatment strategy.

[0112] Furthermore, after executing step S50, the method further includes:

[0113] The data in the Markov decision model were cleaned, including removing treatment trajectories whose data loss exceeded the preset value and filling the data using the KNN algorithm.

[0114] Furthermore, step S60 includes:

[0115] Based on the Markov decision model, the extreme gradient boosting algorithm is used for binary classification training, with the patient's status in each time series as the feature value and the final survival status as the label to train and generate a mortality prediction model;

[0116] Based on the mortality prediction model, patients were divided into groups according to whether they ultimately survived or died. The KNN algorithm was used to calculate the mortality weight of each patient characteristic based on the Minkowski distance formula.

[0117] It should be noted that this embodiment primarily provides a method for optimizing samples when learning sepsis treatment strategies. By considering sample weights, this method can avoid rapidly falling into local optima among a large number of suboptimal samples, thereby enhancing learning effectiveness. The process for implementing each step is identical to steps S10-S30 in the aforementioned embodiment and will not be further elaborated here.

[0118] See also Figure 5 A certain embodiment of the present invention further provides a device for learning sepsis treatment strategies, comprising:

[0119] Data processing unit 01 is used to establish a Markov decision model for sepsis treatment strategy based on sample data; the sample data is pathological data of sepsis patients within a preset time;

[0120] The weight calculation unit 02 is used to establish a sepsis mortality prediction model based on the Markov decision model and obtain the mortality weight of each characteristic of sepsis patients;

[0121] The marking unit 03 is used to mark the sample data using the mortality weight to generate a target sample;

[0122] Learning unit 04 is used to predict sepsis treatment strategies and use deep reinforcement learning methods to learn treatment strategies.

[0123] It is understood that the apparatus provided in this embodiment is used to perform steps S10-S40 in the above embodiment. This embodiment labels patient samples by priority weight, thereby improving the quality of samples for deep intensity learning, enhancing the model's training effectiveness, and avoiding the problem of rapidly falling into local optimality among a large number of suboptimal samples. By adopting an improved deep intensity learning algorithm to learn treatment strategies and adding realistic constraints to the action selection of AI treatment strategies, the recommended treatment strategies are more optimal.

[0124] See also Figure 6 A certain embodiment of the present invention further provides a device for learning sepsis treatment strategies, comprising:

[0125] Data processing module 05, for establishing a Markov decision model for sepsis treatment strategy based on sample data; the sample data is pathological data of sepsis patients within a preset time;

[0126] The weight calculation module 06 is used to establish a sepsis mortality prediction model based on the Markov decision model and obtain the mortality weight of each characteristic of sepsis patients;

[0127] The sample generation module 07 is configured to mark the sample data using the mortality weight to generate a target sample, and use the target sample as a learning sample for the sepsis treatment strategy.

[0128] It is understood that the apparatus provided in this embodiment is used to perform steps S50-S70 in the above embodiment. By considering sample weights, this embodiment can avoid the problem of quickly falling into local optimum in a large number of suboptimal samples, and ultimately enhance the learning effect.

[0129] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for learning a sepsis treatment strategy, characterized in that: include: Establish a Markov decision model for sepsis treatment strategy based on sample data; The sample data is pathological data of sepsis patients within a preset time; Based on the Markov decision model, a sepsis mortality prediction model is established, and the mortality weights of various characteristics of sepsis patients are obtained; Using the mortality weight to mark the sample data to generate a target sample; Predicting sepsis treatment strategies and using deep reinforcement learning methods to learn treatment strategies; The step of labeling the sample data using the mortality rate weight to generate a target sample includes: Using the KNN algorithm, based on Minkowski distance: Let n=100, P=2, The mortality weights of each patient characteristic calculated in step 2 include: (1) Calculate the average distance between the state of the surviving patient and the 100 states closest to the patient who eventually died; (2) Calculate the average distance between the state of the deceased patient and the 100 states closest to the surviving patient; Summarize the distances obtained from (1) and (2). The closer the distance, the closer the current state is to death, but the patient eventually survives after effective treatment; or the closer the current state is to survival, but the patient eventually dies due to inappropriate treatment. That is, the closer the distance, the higher the quality of the patient sample. Conversely, the larger the distance, the lower the quality of the sample. Set the sampling weight of any transfer sample for: ； in, is the average value of the distance, and k is the weight amplification parameter; The method for predicting sepsis treatment strategies utilizes a deep reinforcement learning method to learn treatment strategies, including: Using the KNN algorithm, a preset number of patient states closest to the patient state are found in the training set and the test set, and the doctor's actions under the preset number of patient states are counted. The doctor's treatment strategy is then summarized based on the action distribution. The Huber loss function is used to modify the deep reinforcement learning loss function to obtain the modified loss function.

2. The method for learning sepsis treatment strategies according to claim 1, characterized in that: After establishing the Markov decision model for sepsis treatment strategy based on the sample data, the method further includes: The data in the Markov decision model were cleaned, including removing treatment trajectories whose data loss exceeded the preset value and filling the data using the KNN algorithm.

3. The method for learning sepsis treatment strategies according to claim 2, characterized in that: According to the Markov decision model, a sepsis mortality prediction model is established, and the mortality weights of various characteristics of sepsis patients are obtained, including: Based on the Markov decision model, the extreme gradient boosting algorithm is used for binary classification training, with the patient's status in each time series as the feature value and the final survival status as the label to train and generate a mortality prediction model; Based on the mortality prediction model, patients were divided into groups according to whether they ultimately survived or died. The KNN algorithm was used to calculate the mortality weight of each patient characteristic based on the Minkowski distance formula.

4. The method for learning sepsis treatment strategies according to claim 2, wherein: The deep reinforcement learning method is the Double Dueling DQN deep reinforcement learning method.

5. The method for learning sepsis treatment strategy according to claim 4, characterized in that: The method of using deep reinforcement learning to learn treatment strategies includes adding realistic constraints to the action selection of AI treatment strategies.

6. A device for learning sepsis treatment strategies, characterized in that: include: a data processing unit, configured to establish a Markov decision model for sepsis treatment strategies based on sample data; The sample data is pathological data of sepsis patients within a preset time; a weight calculation unit, configured to establish a sepsis mortality prediction model based on the Markov decision model, and obtain mortality weights of various characteristics of sepsis patients; a marking unit, configured to mark the sample data using the mortality weight to generate a target sample; A learning unit, used to predict sepsis treatment strategies and learn treatment strategies using deep reinforcement learning methods; The step of labeling the sample data using the mortality rate weight to generate a target sample includes: Using the KNN algorithm, based on Minkowski distance: Let n=100, P=2, The mortality weights of each patient characteristic calculated in step 2 include: (1) Calculate the average distance between the state of the surviving patient and the 100 states closest to the patient who eventually died; (2) Calculate the average distance between the state of the deceased patient and the 100 states closest to the surviving patient; Summarize the distances obtained from (1) and (2). The closer the distance, the closer the current state is to death, but the patient eventually survives after effective treatment; or the closer the current state is to survival, but the patient eventually dies due to inappropriate treatment. That is, the closer the distance, the higher the quality of the patient sample. Conversely, the larger the distance, the lower the quality of the sample. Set the sampling weight of any transfer sample for: ； in, is the average value of the distance, and k is the weight amplification parameter; The method for predicting sepsis treatment strategies utilizes a deep reinforcement learning method to learn treatment strategies, including: Using the KNN algorithm, a preset number of patient states closest to the patient state are found in the training set and the test set, and the doctor's actions under the preset number of patient states are counted. The doctor's treatment strategy is then summarized based on the action distribution. The Huber loss function is used to modify the deep reinforcement learning loss function to obtain the modified loss function.

Images