Strategy recommendation method for treating sepsis by heparin based on Q value model

Through the strategy recommendation method based on the Q-value model, the improved cxSOFA score and DDQN model, combined with the comprehensive comparison matrix evaluation, the accuracy and comprehensiveness of AI strategy evaluation were solved, and the strategy recommendation for heparin for the treatment of sepsis was realized. Improve the timeliness and accuracy of treatment strategies.

CN118983080APending Publication Date: 2024-11-19CHONGQING INST OF GREEN & INTELLIGENT TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411124035.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing AI strategy evaluation methods lack accuracy and comprehensiveness in the medical field, especially in the treatment of sepsis. Traditional methods rely on expert intuition and single outcome indicators, and cannot effectively distinguish the advantages and disadvantages of different treatment strategies, and ignore the different strategies. Dynamicity in environmental state.

Method used

Using a strategy recommendation method based on Q-value model, design reward function and optimal strategy termination conditions through improved cxSOFA scores and Double Deep Q Network (DDQN) model, combine the comprehensive comparison matrix to conduct multi-index evaluation to identify the best and worst AI strategy to implement strategies for heparin to treat sepsis.

Benefits of technology

It improves the timeliness and accuracy of treatment strategies, can evaluate AI strategies more comprehensively, and is suitable for the best and worst strategy evaluation between the same Q-value model and different Q-value models, and assists medical staff in taking effective measures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118983080A_ABST
    Figure CN118983080A_ABST
Patent Text Reader

Abstract

The invention discloses a strategy recommendation method for treating sepsis by heparin based on a Q value model, and belongs to the field of medical big data and artificial intelligence, and the method comprises the following steps: S1, preprocessing historical data of sign monitoring and heparin dosage, and obtaining time sequence data of a patient; s2, scoring by adopting cxSOFA and merging into time sequence data of the patient; s3, establishing a Q value model by taking the score difference of adjacent moments as a reward function, taking the heparin dosage as an action and taking the sign monitoring as a state; s4, training the model to obtain an optimal strategy and a worst strategy of the current round; s5, marking an over-rhoep treatment scheme and a secondary rhoep treatment scheme; s6, calculating a comprehensive comparison matrix, and screening out a global optimal strategy; s7, repeating the steps S4-S6 until a termination criterion is met; and S8, realizing strategy recommendation for treating sepsis by heparin of the patient by using the optimal Q value model. According to the invention, development and application of the AI strategy are realized, and medical personnel can be assisted to take effective treatment measures.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a strategy recommendation method for treating sepsis with heparin based on a Q-value model, belongs to the fields of medical big data and artificial intelligence, and is particularly suitable for strategy recommendation and evaluation of treating sepsis with heparin based on a Q-value model. Background Art

[0002] Sepsis is a systemic inflammatory response syndrome caused by infection, which can lead to multiple organ dysfunction and even death in severe cases. The complex pathophysiology and high patient heterogeneity pose a major challenge to optimizing individualized treatment. Timely intervention is crucial for the treatment of sepsis, and delayed treatment or suboptimal decisions may increase the risk of death. Heparin is the most widely used anticoagulant drug, which can inhibit the binding of HMGB1 to lipopolysaccharide and prevent heparanase from degrading the glycocalyx of macrophages, thereby producing a significant anti-inflammatory effect to improve the prognosis of patients with postoperative sepsis.

[0003] Accurate patient status assessment and optimized treatment strategies are crucial to improving the prognosis of sepsis treatment. As an important method of reinforcement learning in the field of artificial intelligence, the core goal of the Q-value model (Q-Learning, QL) is to learn the optimal strategy through interaction with the environment to maximize the reward of a specific task. In the field of healthcare, the development and application of AI strategies are increasingly valued, especially in complex decision-making processes such as treatment plan selection. However, the evaluation of AI strategies faces many challenges, especially how to accurately evaluate the pros and cons of AI strategies in the absence of clear standards or in the presence of inconsistent opinions among multiple experts.

[0004] Traditionally, the evaluation of AI strategies mainly relies on the subjective judgment of experts, especially in the medical field. This evaluation is often based on clinical experience and intuition, lacking quantitative and standardized methods. Although some studies have tried to introduce data-driven evaluation methods, such as evaluation based on value function (Q(s, a)), this method can provide certain quantitative evaluation indicators, but its definition is beyond ρ. ep , times ep Treatment options are often oversimplified and limited to a single outcome measure, such as survival or mortality. This approach makes it difficult to distinguish between different strategies when all patients in the data set survive, as all strategies achieve the same end result.

[0005] According to statistics, almost all evaluations of AI strategies only consider the value function Q(s, a) to determine whether it is the best strategy, while ignoring worse or worst AI strategies. This value function-based evaluation method is not comprehensive enough. The value function-based evaluation method ignores the dynamics and complexity of the strategy during execution and fails to fully consider the performance differences of the strategy under different environmental conditions. Summary of the invention

[0006] In view of this, the present invention provides a strategy recommendation method for treating sepsis with heparin based on a Q-value model, which aims to establish a network model through a Q-value model, train the network model using data in a database, and more accurately identify the "best" and "worst" AI strategies through a multi-index evaluation system, so as to achieve a strategy recommendation for treating sepsis with heparin for patients. In detail, the method of the present invention intends to smooth the discrete SOFA score to a continuous cxSOFA score to design a reward function, realize the reward update of the Q-value model, and evaluate the Q-value model from a comprehensive comparison matrix, and set a reasonable optimal strategy selection termination condition, screen the optimal parameters, and finally ensure the timeliness and accuracy of the recommended strategy and early warning output by the method of the present invention.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A strategy recommendation method for treating sepsis with heparin based on the Q-value model, combined with Figure 1 , characterized in that the method comprises the following steps:

[0009] S1: preprocessing the historical data of vital sign monitoring and heparin dosage according to the patient and time series data to obtain the patient time series data;

[0010] S2: The cxSOFA scoring method is used to score the patient's time series data at each moment, and the score at that moment is incorporated into the patient's time series data as a new feature;

[0011] S3: Taking the difference in cxSOFA scores at adjacent moments as the reward function, the heparin dose as the action, and the vital sign monitoring as the state, a Q-value model is established;

[0012] S4: Input the patient's time series data into the Q value model for training, obtain the best strategy and the worst strategy for the current round by comparing the Q values, and record the Q value model parameters;

[0013] S5: Calculate the positive incentive rate ρ for each patient in the current round ep , according to ρ ep The patient time series data are marked as superρ ep Treatment options and times ep Treatment options;

[0014] S6: Traverse the best strategy, the worst strategy and the corresponding super ρ selected in all rounds ep , times ep Label the treatment plans, calculate the comprehensive comparison matrix for each batch, and compare and select the global best strategy and the global worst strategy;

[0015] S7: Repeat steps S4 to S6 until the Q value model meets the η optimal termination criterion;

[0016] S8: Use the Q-value model parameters corresponding to the global best strategy and the global worst strategy to construct a Q-value model to recommend strategies for heparin treatment of sepsis in patients.

[0017] Furthermore, the preprocessing described in step S1 is: sorting all the data with the same unique identification ID of the patient according to the time series data and performing normalization processing.

[0018] The cxSOFA score described in step S2 ranges from 0 to 24 points and consists of 6 parts, each ranging from 0 to 4 points. The 6 parts are respiratory system, coagulation system, liver function, circulatory system, nervous system, and renal function. Each part requires several individual sign monitoring data to calculate a score, as shown in Table 1:

[0019] Table 1 Symbols for vital signs

[0020]

[0021] The cxSOFA scoring method described in step S2 is an improvement on the SOFA scoring method, which makes the SOFA scoring method continuous:

[0022] cxSOFA(x)=f1(x)+f2(x)+f3(x)+f4(x)+f5(x)+f6(x);

[0023] Going further, the specific calculation is as follows:

[0024] (1) f1(x) is calculated as:

[0025] (2)f2(x) is calculated as:

[0026] (3)f3(x) is calculated as:

[0027] (4) f4(x) is calculated as: f4(x) = max(x 41 ,x 42 , x 43 , x 44 , x 45 );

[0028] in, x 43 =bool(x6>0)*2,x 44 =min(bool(x7>0),10*x7+3),x 45 =min(bool(x8>0),10*x8+3);

[0029] (5)f5(x) is calculated as:

[0030] (6) f6(x) is calculated as: f6(x) = max(x 61 ,x 62 );

[0031] in,

[0032] Among them, the vital sign monitoring data x at a certain moment = (x1, ..., x 11 ), Corresponding to the vital signs in Table 1, 0≤i≤11; f i (x) is the corresponding step function obtained by fitting the polynomial function.

[0033] Further, the cxSOFA score difference of the states at adjacent moments in step S3 is: t =cxSOFA(s t )-cxSOFA(s t+1 ), the Q value model is a Double Deep Q Network (DDQN) model, which is composed of a Q network, a target network and an experience buffer pool; the Q network input is the current state s in the experience buffer pool t and the current action a t , the output is the predicted Q value Q(s t , a t ; θ), where θ is the Q network training parameter; get the next state s from the experience buffer pool t+1 , input the Q network to obtain the Q values ​​obtained by taking each action in this state, take the action a′ corresponding to the largest Q value among all Q values, and compare it with the next state s in the experience buffer pool t+1 Input the target network together to obtain the optimal Q value Q(st+1 , a′; θ′), where θ′ is the target network training parameter; the time series data in the data set is converted into (s t , a t ,r t ,s t+1 ) in the form of experience stored in the experience buffer pool; during training, a batch of indexes will be randomly given, and then the experience corresponding to the index will be extracted from the experience buffer pool; where the state s t Corresponding to the vital sign monitoring data at time t, action a t Corresponding to the heparin dose at time t, the reward r t Corresponding to the state s at time t t With status s t+1 The cxSOFA score is poor, status s t+1 Corresponding to the state s at time t t Execute action a t The next state after the transition.

[0034] Furthermore, the network update process of DDQN is divided into the following parts:

[0035] [1]. Network parameter synchronization: Update the parameters θ of the Q network to the parameters θ′ of the target network. A soft update method is adopted, that is, an update rate τ is introduced, and the weighted average value of the Q network parameters and the target network parameters is assigned to the target network, that is, θ′=τ*θ+(1-τ)θ′;

[0036] [2]. Gradient update: Update the hidden layer parameters of the Q network and use the MSE algorithm to calculate the loss function L = (r t +γ·Q(s t+1 ,a′;θ′)-Q(s t , a t ;θ)) 2 ; Among them, r t is the reward function value at time t, that is, the difference in cxSOFA scores between time t and time t-1; γ is the discount factor and is a parameter to be learned.

[0037] Furthermore, the Q value model described in step S3 is a Deep Q network (DQN) model, which is composed of a Q network, a target network and an experience buffer pool; the Q network is an updated network, which is trained to generate the best state and action value by predicting the Q value, and its input is the current state s in the experience buffer pool. t and the current action a t , the output is the predicted Q value Q(s t , a t; θ), where θ is the Q network training parameter; the target network obtains the next state s from the experience buffer pool t+1 , and calculate the best Q value maxQ(s) from all actions a′ executed in this state t+1 , a′; θ′), where θ′ is the Q network training parameter; the experience buffer pool will (s t , a t ,r t ,s t+1 ) is stored as experience. During training, a batch of indexes will be randomly given, and then the experience corresponding to the index will be extracted from the experience buffer pool. t Corresponding to the physical sign monitoring at time t as the state, action a t Corresponding to the heparin dose at time t.

[0038] Furthermore, the update process of the DQN network is divided into the following parts:

[0039] [1] Network parameter synchronization: The parameters θ of the Q network are updated to the parameters θ′ of the target network. The update method adopted is soft update, that is, an update rate τ is introduced, and the value of the weighted average of the Q network parameters and the target network parameters is assigned to the target network. The specific update formula is: θ′=τ*θ+(1-τ)θ′.

[0040] [2] Gradient update: Update the hidden layer parameters of the Q network and use the MSE algorithm to calculate the loss function L = (r t +γ·maxQ(s t+1 ,a′;θ′)-Q(s t , a t ;θ)) 2 ; Among them, r t is the reward function value at time t, that is, the score difference between time t and time t-1; γ is the discount factor, which is a parameter to be learned.

[0041] Furthermore, the Q-value model in step S3 is a Q-Learning (QL) model composed of a Q table; the Q table stores each state s t and action a t The corresponding Q value.

[0042] Furthermore, the Q-value update formula in the Q-Learning model is as follows: in, Indicates that the state s at the next moment t+1 The maximum Q value of all possible actions.

[0043] Further, the comparison of Q values ​​in step S4 obtains the best strategy and the worst strategy for the current round, specifically: for any state s, calculate the output Q value under state s and action a, that is, Q(s, a), according to and Select the strategy with the largest and smallest Q value, that is, the best strategy π * , the worst strategy π′; where A is the action space.

[0044] Further, the step S5 is specifically as follows:

[0045] S501: Count the number of positive excitations N of the patient's time series data in the current round g , that is, r t The number of states ≥ 0;

[0046] S502: Calculate the positive excitation rate: ρ ep =N g / n; where n is the total number of states of the patient time series data in the current round;

[0047] S503: Set a threshold τ∈[0.5, 1] ​​as the positive excitation rate threshold. When ρ ep ≥τ, add the patient time series data at the corresponding time to the super ρ ep The set of treatment options P G Otherwise, add to the sub-ρ ep The set of treatment options P B middle.

[0048] Further, in combination with Table 2 and Table 3, the step S6 is specifically as follows:

[0049] S601: Calculate vital signs monitoring data t In the optimal strategy π * Action of π * (s t ) and the super ρ at time t ep Treatment plana t The action similarity of in, idx(*) represents the subscript of *, n is the total number of states of the patient time series data in the current round, p∈P G , P G For the current round of Super League ρ ep The collection of time series data of treatment plans, P G The number of time series data included is denoted as |P G |;

[0050] S602: Calculate vital sign monitoring data t In the optimal strategy π * Action of π * (st ) and the times ρ at time t ep Treatment plana t The action similarity of p∈P B , P B is the number ρ in the current round ep The collection of time series data of treatment plans, P B The number of time series data included is denoted as |P B |;

[0051] S603: Calculate vital sign monitoring data t In the worst strategy π′, the action π′(s t ) and the super ρ at time t ep Treatment plana t The action similarity of

[0052] S604: Calculate vital sign monitoring data t In the worst strategy π′, the action π′(s t ) and the times ρ at time t ep Treatment plana t The action similarity of

[0053] S605: Calculate the difference between the best and the worst o =OG-OB, inferior extreme difference Ξ w =WB-WG, comprehensive confidence and the combined bias μ = (OG-WG)-(WB-OB);

[0054] S606: Traverse all rounds, for any two rounds where i≠j, if o (π i )≥Ξ o (πj) and Ξ w (π i )≥Ξ w (π j ), we assume that strategy π i Compared with strategy π j Otherwise, if σ(π i )≥σ(π j ), and also believe that strategy π i Compared with strategy π j To be better, the best global strategy is The global worst strategy is π′ i .

[0055] Preferably, the method for selecting the global best strategy in step S606 can introduce a temporary variable to store the global best strategy, and only compare the current round with the temporary variable to determine the global best strategy.

[0056]

[0057] Furthermore, the η-optimal termination criterion described in step S7 is specifically: set a hyperparameter η≥1, if the global best strategy and the global worst strategy selected by step S6 are obtained in the most recent η rounds, then continue the training of the next round of Q-value model; otherwise, terminate the training.

[0058] The step S8 is specifically as follows: according to the global best strategy and the global worst strategy corresponding to when the termination criterion is reached, the corresponding Q value model parameters are selected to construct a Q value model, thereby realizing the strategy recommendation for the heparin treatment of sepsis in patients.

[0059] Beneficial effects of the present invention: The present invention provides a strategy recommendation method for treating sepsis with heparin based on a Q-value model, updates the reward function of the Q-value model through an improved cxSOFA scoring method, designs the η optimal termination criterion of the Q-value model, and introduces a comprehensive comparison matrix to accurately evaluate the Q-value model, intelligently screen out the optimal network parameters, and ensure the timeliness and accuracy of the model output strategy. It is foreseeable that the method of the present invention has strong adaptability, and can be applied not only to the evaluation of different strategies of the same Q-value model, but also to the evaluation of the best and worst strategies between different Q-value models, realizing the development and application of AI strategies, and can assist medical personnel in taking effective treatment measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to illustrate the purpose and technical solution of the present invention, the present invention provides the following drawings for illustration:

[0061] Figure 1 It is a flowchart of the method of the present invention;

[0062] Figure 2 This is a DDQN model architecture diagram of Example 1 of the present invention;

[0063] Figure 3 This is a comparative experimental result diagram of Example 1 of the present invention;

[0064] Figure 4 This is a comparative experimental result diagram of Example 2 of the present invention. DETAILED DESCRIPTION

[0065] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0066] The accompanying drawings are only used for illustrative explanations and are only schematic diagrams, and should not be construed as limitations of the present invention.

[0067] Example 1: MIMIC-IV (a large public medical database) https: / / physionet.org / content / mimiciv / 2.2 / ) and eICU( https: / / physionet.org / content / eicu-crd-demo / 2.0.1 / ) to organize the patient data that is publicly available, including: the patient's unique ID, PaO2 / FiO2 oxygenation index (in mmHg), whether mechanical ventilation is required, platelet count (in 10 9 / L), bilirubin (in μmol / L), mean arterial pressure (in mmHg), dopamine dose (in μg / (kg*min)), dobutamine dose (in μg / (kg*min)), epinephrine dose (in μg / (kg*min)), norepinephrine dose (in μg / (kg*min)), GCS score, creatinine (in μmol / L), urine volume (in mL), dose of heparin injected by the patient, recording time, whether the patient died, a total of 16 features. In order to better assist medical personnel in using heparin to quickly and correctly treat sepsis during surgery, this embodiment proposes a strategy recommendation method for treating sepsis with heparin based on the DDQN model, which includes the following steps:

[0068] Step 1: Preprocess the historical data of vital sign monitoring and heparin dosage according to patient and time series data to obtain patient time series data.

[0069] The preprocessing is as follows: exclude patients whose surgical operation sites are shallow, whose abdominal cavity is not involved, and whose surgical records are incomplete or repeated; exclude patients with sepsis, suspected infection, and severe complications before surgery; exclude patients who did not use heparin during surgery; exclude patients with missing data; sort all data with the same unique identification ID of the patient according to time series data, and perform normalization processing.

[0070] Step 2: Use the cxSOFA scoring method to score the patient's time series data at each moment, and incorporate the score at that moment into the patient's time series data as a new feature.

[0071] The cxSOFA score ranges from 0 to 24 points and consists of 6 parts, each of which ranges from 0 to 4 points. The 6 parts are respiratory system, coagulation system, liver function, circulatory system, nervous system, and renal function. Each part requires a number of individual sign monitoring data to calculate a score. Here, the individual sign detection data required for each part are recorded as x = (x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30, x31, x32, x33, x34, x35, x36, x37, x38, x39, x40, x41, x42, x43, x44, x45, x46, x47, x48, x49, x50, x49, x51, x52, x53, x54, x55, x56, x57, x58, x59, x60, x61, x62, x63, x64, x65 10 , x 11 ), 0≤i≤11.

[0072] Table 4 Patient data at a certain time

[0073]

[0074] The calculation of the cxSOFA scoring method is specifically as follows:

[0075] f1(x) is calculated as:

[0076] f2(x) is calculated as:

[0077] f3(x) is calculated as:

[0078]

[0079] The calculation method of f4(x) is: f4(x) = max(x 41 , x 42 , x 43 , x 44 , x 45 );

[0080] in, x 43 =bool(x6>0)*2,x 44 =min(bool(x7>0),10*x7+3),x 45 =min(boolCx8>0),10*x8+3);

[0081] f5(x) is calculated as:

[0082] The calculation method of f6(x) is: f6(x) = max(x 61 , x 62 );

[0083] in,

[0084] cxSOFA score cxSOFA(x)=f1(x)+f2(x)+f3(x)+f4(x)+f5(x)+f6(x);

[0085] Specifically, for the data of patient ID=20008587 at a certain time, as shown in Table 4, its cxSOFA score is:

[0086] The respiratory system score was 0.7, the coagulation system score was 1.416, the liver system score was 3.744, the circulatory system score was 3.56, the central nervous system score was 1.982, the kidney score was 4, and the SOFA score = 0.7 + 1.416 + 3.744 + 3.56 + 1.982 + 4 = 15.402.

[0087] Step 3: Use the difference in cxSOFA scores at adjacent moments as the reward function, the heparin dose as the action, and the vital sign monitoring as the state to establish a Double Deep Q Network (DDQN) model.

[0088] The cxSOFA score difference of the states at adjacent moments in step 3 is: t =cxSOFA(s t )-cxSOFA(s t+1 ), the DDQN model, such as Figure 2 As shown, it is composed of a Q network, a target network and an experience buffer pool; the Q network input is the current state s in the experience buffer pool t and the current action a t , the output is the predicted Q value Q(s t , a t ; θ), where θ is the Q network training parameter; get the next state s from the experience buffer pool t+1 , input the Q network to obtain the Q values ​​obtained by taking each action in this state, take the action a′ corresponding to the largest Q value among all Q values, and compare it with the next state s in the experience buffer pool t+1 Input the target network together to obtain the optimal Q value Q(s t+1 , a′; θ′), where θ′ is the target network training parameter; the time series data in the data set is converted into (s t , a t ,rt ,s t+1 ) in the form of experience stored in the experience buffer pool; during training, a batch of indexes will be randomly given, and then the experience corresponding to the index will be extracted from the experience buffer pool; where the state s t Corresponding to the vital sign monitoring data at time t, action a t Corresponding to the heparin dose at time t, the reward r t Corresponding to the state s at time t t With status s t+1 The cxSOFA score is poor, status s t+1 Corresponding to the state s at time t t Execute action a t The next state after the transition.

[0089] Table 5 Some characteristics of patient time series data

[0090]

[0091] Table 6 Relationship between heparin dosage and action

[0092]

[0093] Preferably, the network update process of DDQN is divided into the following parts:

[0094] (1) Network parameter synchronization: Update the parameters θ of the Q network to the parameters θ′ of the target network using a soft update method, that is, introduce an update rate τ, and assign the weighted average of the Q network parameters and the target network parameters to the target network, that is, θ′=τ*θ+(1-τ)θ′;

[0095] (2) Gradient update: Update the hidden layer parameters of the Q network and use the MSE algorithm to calculate the loss function L = (r t +γ·Q(s t+1 ,a′;θ′)-Q(s t , a t ;θ)) 2 ; Among them, r t is the reward function value at time t, that is, the difference in cxSOFA scores between time t and time t-1; γ is the discount factor and is a parameter to be learned.

[0096] Specifically, some features of the time series data of patient ID=20008587 are shown in Table 5, and the relationship between the dose and action of heparin is set as shown in Table 6.

[0097] In time step 5, 1.38<the dose of heparin injected into the patient<1.88, so the action is a2, the cxSOFA score is 12, and the cxSOFA score of the next time step is 10, so the reward at this time is 12-10=2; in time step 6, the dose of heparin injected into the patient is>3.5, so the action is a4, the cxSOFA score is 10, and the cxSOFA score of the next time step is 15, so the reward at this time is 10-15=-5.

[0098] Step 4: Input the patient time series data into the DDQN model for training, obtain the best strategy and the worst strategy for the current round by comparing the Q values, and record the DDQN model parameters;

[0099] For any state s, calculate the output Q value under state s and action a, that is, Q(s, a), according to and Select the strategy with the largest and smallest Q value, that is, the best strategy π * , the worst strategy π′; where A is the action space.

[0100] Step 5: Calculate the positive incentive rate ρ of each patient in the current round ep , according to ρ ep The patient time series data are marked as superρ ep Treatment options and times ep Treatment options;

[0101] The positive incentive rate calculation method is as follows:

[0102] S501: Count the number of positive excitations N of the patient's time series data in the current round g , that is, r t The number of states ≥ 0;

[0103] S502: Calculate the positive excitation rate: ρ ep =N g / n; where n is the total number of states of the patient time series data in the current round;

[0104] S503: Set a threshold τ∈[0.5,1] as the positive excitation rate threshold. When ρ ep ≥τ, add the patient time series data at the corresponding time to the super ρ ep The set of treatment options P G Otherwise, add to the sub-ρ ep The set of treatment options P B middle.

[0105] Step 6: Traverse the best strategy, worst strategy and corresponding super ρ selected in all rounds ep , times epThe treatment plans are labeled, and the comprehensive comparison matrix of each batch is calculated to compare and screen out the global best strategy and the global worst strategy.

[0106] The process of combining the comprehensive comparison matrix to select the global best strategy and the worst strategy is specifically as follows:

[0107] S601: Calculate vital signs monitoring data t In the optimal strategy π * Action of π * (s t ) and the super ρ at time t ep Treatment plana t The action similarity of in, idx(*) represents the subscript of *, n is the total number of states of the patient time series data in the current round, p∈P G , P G is the current round of super ρ ep The collection of time series data of treatment plans, P G The number of time series data included is denoted as |P G |;

[0108] S602: Calculate vital sign monitoring data t In the optimal strategy π * Action of π * (s t ) and the times ρ at time t ep Treatment plana t The action similarity of p∈P B , P B is the current round ρ ep The collection of time series data of treatment plans, P B The number of time series data included is denoted as |P B |;

[0109] S603: Calculate vital sign monitoring data t In the worst strategy π′, the action π′(s t ) and the super ρ at time t ep Treatment plana t The action similarity of

[0110] S604: Calculate vital sign monitoring data t In the worst strategy π′, the action π′(s t ) and the times ρ at time t ep Treatment plana t The action similarity of

[0111] S605: Calculate the difference between the best and the worst o =OG-OB, inferior extreme difference Ξ w =WB-WG, comprehensive confidence and the combined bias μ = (OG-WG)-(WB-OB);

[0112] S606: Compare the best and worst strategies selected in the current round with those selected in the previous round. For strategies i≠j∈[1,2] in different rounds, if o (π i )≥Ξ o (π j ) and w (π i )≥Ξ w (π j ), we assume that strategy π i Compared with strategy π j Well, the global optimal strategy is The global worst strategy is π′ i ; Otherwise, if σ(π i )≥σ(π j ), we assume that strategy π i Compared with strategy π j Well, the global optimal strategy is The global worst strategy is π′ i .

[0113] Step 7: Repeat steps S4 to S6 until the DDQN model meets the η optimal termination criterion.

[0114] The η-optimal termination criterion is: set a hyperparameter η≥1. If the global best strategy and the global worst strategy selected in step 6 are obtained in the most recent η rounds, continue the training of the next round of DDQN model; otherwise, terminate the training. The number η is set to 50, 100, 150, 200, and 250 respectively, and the comprehensive comparison matrix results are shown in Figure 3 .

[0115] DDQN model global optimal strategy and super ρ ep The consistency rate of treatment options is 69.7%, and the worst strategy of the DDQN model is ρ ep The consistency rate of treatment plans was 73.7%, and the comprehensive confidence and comprehensive bias were 1.808 and -0.006, respectively.

[0116] Step 8. Use the DDQN model parameters corresponding to the global best strategy and the global worst strategy to build a DDQN model to recommend strategies for heparin treatment of sepsis in patients.

[0117] The strategy recommendation for heparin treatment of sepsis in patients is the predicted action of the DDQN model with the optimal parameters.

[0118] Example 2: For the scenario of Example 1, in order to better demonstrate the advantages of the method of the present invention, this example provides a comparative experiment of "strategy recommendation method for heparin treatment of sepsis based on Q-value model". Similar to the steps in Example 1, they will not be repeated here. Taking the cxSOFA score difference or SOFA score difference at adjacent moments as the reward function, the heparin dose as the action, and the vital sign monitoring as the state, SOFA-QL, cxSOFA-QL, SOFA-DQN, cxSOFA-DQN, SOFA-DDQN and cxSOFA-DDQN models are established respectively, where SOFA-QL represents a Q-Learning model that uses the SOFA score difference at adjacent moments as the reward function, cxSOFA-QL represents a Q-Learning model that uses the cxSOFA score difference at adjacent moments as the reward function, and the symbols of the other models are also similar, where the SOFA score is the scoring method of the invention patent "Strategy recommendation and evaluation method for heparin treatment of sepsis based on reinforcement learning" (application number: 202410362596). According to steps 1 to 7 in Example 1, SOFA-QL, cxSOFA-QL, SOFA-DQN, cxSOFA-DQN, SOFA-DDQN and cxSOFA-DDQN models are trained respectively to obtain their global best strategy and global worst strategy and the corresponding super ρ ep Treatment options and times ep Treatment options, respectively, calculate the comprehensive comparison matrix, the results are as follows Figure 4 shown.

[0119] When the threshold τ = 0.7, the global optimal strategy of the cxSOFA-QL model is consistent with the super ρ ep The agreement rate of treatment options was 64.4%, and the worst strategy of the cxSOFA-QL model was ρ ep The consistency rate of the treatment plan was 59.8%, and the comprehensive confidence and comprehensive bias were 1.471 and 0.084, respectively. The global optimal strategy of the cxSOFA-DQN model and the super ρ ep The consistency rate of treatment options is 69.1%, and the worst strategy of cxSOFA-DQN model is ρ ep The consistency rate of treatment plans was 63.4%, and the comprehensive confidence and comprehensive bias were 1.625 and 0.082, respectively. ep The consistency rate of treatment options was 69.7%, and the worst strategy of the cxSOFA-DDQN model was ρ epThe consistency rate of treatment plan was 73.7%, and the comprehensive confidence and comprehensive bias were 1.808 and -0.006, respectively. At the same time, the Q value model based on SOFA scoring method was overall worse than the Q value model based on cxSOFA scoring method in performance.

[0120] By comparison, the method of the present invention recommends the AI ​​strategy generated by the cxSOFA-DDQN model to achieve strategy recommendation for heparin treatment of sepsis in patients.

[0121] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A strategy recommendation method for treating sepsis with heparin based on a Q-value model, characterized in that: The method comprises the following steps: S1: preprocessing the historical data of vital sign monitoring and heparin dosage according to the patient and time series data to obtain the patient time series data; S2: The cxSOFA scoring method is used to score the patient's time series data at each moment, and the score at that moment is incorporated into the patient's time series data as a new feature; S3: Taking the difference in cxSOFA scores at adjacent moments as the reward function, the heparin dose as the action, and the vital sign monitoring as the state, a Q-value model is established; S4: Input the patient's time series data into the Q value model for training, obtain the best strategy and the worst strategy for the current round by comparing the Q values, and record the Q value model parameters; S5: Calculate the positive incentive rate ρ for each patient in the current round ep , according to ρ ep The patient time series data are marked as superρ ep Treatment options and times ep Treatment options; S6: Traverse the best strategy, the worst strategy and the corresponding super ρ selected in all rounds ep , times ep Label the treatment plans, calculate the comprehensive comparison matrix for each batch, and compare and select the global best strategy and the global worst strategy; S7: Repeat steps S4 to S6 until the Q value model meets the η optimal termination criterion; S8: Use the Q-value model parameters corresponding to the global best strategy and the global worst strategy to build a Q-value model to recommend strategies for heparin treatment of sepsis in patients; The preprocessing described in step S1 is: sorting all the data with the same unique identification ID of the patient according to the time series data and performing normalization processing; The cxSOFA score ranges from 0 to 24 points and consists of 6 parts, each of which ranges from 0 to 4 points. The 6 parts are respiratory system, coagulation system, liver function, circulatory system, nervous system, and renal function. Each part requires several individual sign monitoring data to calculate a score, which is an improvement on the SOFA scoring method and makes the SOFA scoring method continuous. The Q value model is one of a DDQN model, a DQN model or a QL model.

2. The strategy recommendation method for treating sepsis with heparin based on the Q-value model according to claim 1, characterized in that: The cxSOFA scoring method described in step S2 is specifically: cxSOFA(x)=f1(x)+f2(x)+f3(x)+f4(x)+f5(x)+f6(x); in: f1(x) is calculated as: f2(x) is calculated as: f3(x) is calculated as: The calculation method of f4(x) is: f4(x) = max(x 41 , x 42 , x 43 , x 44 , x 45 ); f5(x) is calculated as: The calculation method of f6(x) is: f6(x) = max(x 61 , x 62 ); in, x 43 =bool(x6>0)*2,x 44 =min(bool(x7>0),10*x7+3),x 45 =min(bool(x8>0),10*x8+3); Vital sign monitoring data at a certain moment Corresponding to the vital signs in Table 1, 0≤i≤11; f i (x) is the corresponding step function obtained by fitting the polynomial function.

3. The method for recommending a strategy for treating sepsis with heparin based on a Q-value model according to claim 1, characterized in that: The cxSOFA score difference of the states at adjacent moments in step S3 is: t =cxSOFA(s t )-cxSOFA(s t+1 ).

4. The method for recommending a strategy for treating sepsis with heparin based on a Q-value model according to claim 1, characterized in that: The comparison of Q values ​​in step S4 obtains the best strategy and the worst strategy for the current round, specifically: for any state s, calculate the output Q value under state s and action a, i.e., Q(s,a), according to and Select the strategy with the largest and smallest Q value, that is, the best strategy π * , the worst strategy π′; where A is the action space.

5. The method for recommending a strategy for treating sepsis with heparin based on a Q-value model according to claim 1, characterized in that: Step S5 is specifically as follows: S501: Count the number of positive excitations N of the patient's time series data in the current round g , that is, r t The number of states ≥ 0; S502: Calculate the positive excitation rate: ρ ep =N g / n; where n is the total number of states of the patient time series data in the current round; S503: Set a threshold τ∈[0.5, 1] ​​as the positive excitation rate threshold. When ρ ep ≥τ, add the patient time series data at the corresponding time to the super ρ ep The set of treatment options P G Otherwise, add to the sub-ρ ep The set of treatment options P B middle.

6. The method for recommending a strategy for treating sepsis with heparin based on a Q-value model according to claim 1, characterized in that: The step S6 is specifically as follows: S601: Calculate vital signs monitoring data t In the optimal strategy π * Action of π * (s t ) and the super ρ at time t ep Treatment plana t The action similarity of in, idx(*) represents the subscript of *, n is the total number of states of the patient time series data in the current round, p∈P G , P G The current round of Super League ρ ep The collection of time series data of treatment plans, P G The number of time series data included is denoted as |P G |; S602: Calculate vital sign monitoring data t In the optimal strategy π * Action of π * (s t ) and the times ρ at time t ep Treatment plana t The action similarity of p∈P B , P B is the number ρ in the current round ep The collection of time series data of treatment plans, P B The number of time series data included is denoted as |P B |; S603: Calculate vital sign monitoring data t In the worst strategy π′, the action π′(s t ) and the super ρ at time t ep Treatment plana t The action similarity of S604: Calculate vital sign monitoring data t In the worst strategy π′, the action π′(s t ) and the times ρ at time t ep Treatment plana t The action similarity of S605: Calculate the difference between the best and the worst o =OG-OB, inferior extreme difference Ξ w =WB-WG, comprehensive confidence and the combined bias μ = (OG-WG)-(WB-OB); S606: Traverse all rounds, for any two rounds where i≠j, if o (π i )≥Ξ o (π j ) and w (π i )≥Ξ w (π j ), we assume that strategy π i Compared with strategy π j Otherwise, if σ(π i )≥σ(π j ), and also believe that strategy π i Compared with strategy π j To be better, the best global strategy is The global worst strategy is π′ i .

7. The strategy recommendation method for treating sepsis with heparin based on the Q-value model according to claim 1, characterized in that: The η-optimal termination criterion described in step S7 is specifically: set a hyperparameter η≥1. If the global best strategy and the global worst strategy selected by step S6 are obtained in the most recent η rounds, continue the training of the next round of Q value model; otherwise, terminate the training.

8. The method for recommending a strategy for treating sepsis with heparin based on a Q-value model according to claim 6, characterized in that: The method for selecting the global best strategy described in step S606 can be achieved by introducing a temporary variable to store the global best strategy, and only comparing the current round with the temporary variable to determine the global best strategy.