A decision intervention method and system based on reinforcement learning and digital twinning

Abstract

The application relates to the technical field of medical informatization, and discloses a decision intervention method and system based on reinforcement learning and digital twinning. The method comprises the following steps: S1, continuously collecting patient data for processing to form a standardized data set; S2, constructing a hybrid individual heart digital twinning model; S3, constructing a reinforcement learning intelligent agent, simulating the effects of different intervention schemes, and generating multiple groups of candidate intervention strategies; S4, checking the compliance and safety of the intervention strategies; and S5, issuing the intervention strategies and explanation documents to execute intervention. The application combines individual heart digital twinning and reinforcement learning to establish an individualized cardiovascular treatment optimization method and system for intensive care environments, accurately simulates the cardiac dynamics state of a patient by using an individual twinning body, dynamically corrects the model by combining multi-modal real-time monitoring data, ensures that the intervention strategies can be individually adapted to the differences of patients, and can produce significant technical and application value in clinical practice.

Description

Technical Field

[0001] This invention relates to the field of medical information technology, specifically to a decision intervention method and system based on reinforcement learning and digital twins. Background Technology

[0002] Currently, cardiovascular disease is one of the most common critical illnesses in the ICU. Clinical treatment often relies on physicians' experience and guideline recommendations, but existing cardiac digital twin technology is mostly used for preoperative simulation and teaching, failing to achieve dynamic individualized treatment in the ICU setting. This is because different patients vary significantly in hemodynamics, myocardial contractility, and drug sensitivity, making it difficult to guarantee efficacy using a uniform treatment pathway. Current treatments mainly rely on periodic manual assessments and empirical adjustments, lacking real-time response and intelligent control to dynamic changes in patients. Summary of the Invention

[0003] In view of this, the present invention provides a decision intervention method and system based on reinforcement learning and digital twins to solve the above problems.

[0004] To address the above technical problems, this invention provides a decision intervention method based on reinforcement learning and digital twins, comprising: S1. Continuously collect physiological status, clinical equipment operating parameters, test results and basic information of ICU cardiovascular patients, and perform noise reduction, time alignment, outlier removal and standardization on the collected data to form a standardized dataset. S2. Construct a hybrid individual cardiac digital twin model. The twin model is used to simulate the patient's cardiovascular dynamics and predict short-term physiological changes, and is updated according to a preset period and / or changes in the patient's acute condition. S3. Construct a reinforcement learning agent to simulate the effects of different intervention schemes and generate multiple sets of candidate intervention strategies. S4. Call upon the medical knowledge graph, which includes patient-associated disease types, drug information, clinical guidelines, and physiological indicator standards, to verify the compliance and safety of the intervention strategy; if it passes, generate an explanation document; if it fails, trigger the reinforcement learning agent secondary generation strategy. S5. Distribute the validated intervention strategy and interpretation documents to clinical devices for intervention; extract actual efficacy data and process abnormal data after intervention, and calibrate the twin model parameters.

[0005] As an alternative approach, a hybrid individual heart digital twin model includes a kinetic model and a parametric model; the kinetic model is used to calculate cardiac structural and functional parameters, while the parametric model is configured with a feedforward neural network for real-time prediction. The kinetic model is used to output myocardial contractility coefficient, stroke volume and peripheral vascular resistance. The parametric model receives real-time collected physiological indicators and synchronous output data of the kinetic model in a preset cycle. It outputs the patient's physiological prediction value and prediction uncertainty through a feedforward neural network.

[0006] As an optional approach, the kinetic model includes a ventricular function calculation module and a vascular resistance calculation module; The ventricular function calculation module calculates stroke volume based on the patient's end-diastolic volume, end-systolic volume, and myocardial contractility parameters. The vascular resistance calculation module combines mean arterial pressure, central venous pressure, and cardiac output to calculate peripheral vascular resistance, and introduces a vascular elasticity coefficient related to the patient's age to correct for peripheral vascular resistance. Among them, the calculation of end-diastolic volume is related to the patient's body surface area and central venous pressure, while the calculation of end-systolic volume is related to end-diastolic volume and stroke volume.

[0007] As an optional approach, the parameterized model includes an input layer, a hidden layer, and an output layer; The input layer has a preset number of nodes, and the input parameters accepted include directly collected patient parameters and mechanistic parameters output by the dynamic model according to a preset period. The hidden layer is configured with a fully connected structure to extract dynamic correlation features between input parameters; The output layer has a set of nodes for outputting the predicted values ​​of the patient's heart rate, systolic blood pressure, diastolic blood pressure, cardiac output, and central venous pressure within a preset time period, and also outputs the prediction uncertainty corresponding to each predicted value.

[0008] As an alternative approach, after the dynamic model and the parameterized model have completed initialization training, dynamic updates, and validity verification, they can be merged to form a twin model. Initialization training includes: using the output of the dynamic model as the training input of the parameterized model, and using the root mean square error between the model's predicted values ​​and the actual monitored values ​​as the objective function to complete the initialization training; Dynamic updates include: triggering immediate updates at preset intervals and / or when patients experience acute changes in their condition; comparing clinical efficacy data with the current prediction results of the model; adjusting the weight parameters of the parameterized model using the gradient descent method; and verifying the rationality of the physiological parameters of the dynamic model. The validity verification includes: after each model initialization or update, calculating the root mean square error between the model's predicted value and the patient's actual monitored value. If the error meets the preset validity threshold, the model is deemed valid. At the same time, the model's predicted trend is submitted to the clinic for review to confirm the match with the patient's known condition.

[0009] As an optional approach, merging to form a twin model also includes eliminating the fusion error between the dynamic model and the parametric model, including: The prediction error and prediction uncertainty of the kinetic model are calculated. The prediction error of the kinetic model is the deviation between the predicted value of the physiological index and the actual monitored value of the patient. The prediction uncertainty is determined based on the fluctuation range of the corresponding physiological parameter. The prediction error and prediction uncertainty of the parameterized model are calculated. The prediction error of the parameterized model is the deviation between the predicted value of the physiological index output by the feedforward neural network and the actual monitored value of the patient. The prediction uncertainty is the confidence quantification result of the corresponding predicted value output by the parameterized model. The error weight of the dynamic model is determined based on the prediction uncertainties of both models. The error weight is the ratio of the prediction uncertainty of the parameterized model to the sum of the prediction uncertainties of the dynamic model and the parameterized model. The prediction errors of the dynamic model and the parameterized model are weighted and fused according to the error weights to obtain the final prediction error of the hybrid individual heart digital twin model; when the final prediction error meets the preset threshold, it serves as the prerequisite for validating the twin model.

[0010] As an alternative approach, a reinforcement learning agent is constructed to simulate the effects of different intervention schemes, generating multiple sets of candidate intervention strategies, including: Using a hybrid individual cardiac digital twin model as a virtual decision-making environment for reinforcement learning, the state space is defined as the predicted physiological indicators output by the twin model and the deviation between these indicators and the doctor's set goals, and the action space is defined as the adjustment of vasoactive drug infusion rate, fluid volume, and clinical equipment parameters. A reinforcement learning agent is constructed. The agent is trained through offline pre-training and online adjustment. After the agent outputs an intervention action, the twin model returns the corresponding physiological indicator changes and reward value. After the agent is trained, multiple sets of candidate intervention strategies are generated. Different disease scenarios are simulated in the twin model and the reward value of each strategy is calculated. Candidate intervention strategies that meet the preset reward criteria are then selected.

[0011] As an alternative approach, reinforcement learning agents perform multi-objective proximal policy optimization based on proximal policy optimization algorithms; Candidate intervention strategies include infusion rate adjustment protocols for vasoactive drugs, fluid resuscitation volume adjustment protocols, and ventilator parameter adjustment protocols. Candidate intervention strategies that meet the preset reward criteria are selected by simulating three scenarios—baseline state, disease deterioration, and disease improvement—in an individual cardiac digital twin model, and calculating the average reward value of each candidate strategy in the three scenarios; strategies whose average reward value meets the preset high reward criteria and whose reward value in a single scenario is not lower than the preset basic criteria are retained.

[0012] As an optional approach, compliance and security verification of intervention strategies includes: Candidate intervention strategies are decomposed into triplets that include intervention measures, patient status, and expected effects, and then converted into a structured format that can be recognized by medical knowledge graphs. The system utilizes a medical knowledge graph containing information on ICU cardiovascular treatment-related disease types, drug information, physiological indicators, and clinical guidelines, and employs a rule engine to verify the compliance and safety of candidate intervention strategies. The compliance verification includes verifying whether the drug dosage is within the range recommended by clinical guidelines, whether there is a contraindication between the drug and the patient's underlying disease, and whether there is a risk of interaction when multiple drugs are used in combination. Safety verification includes assessing whether there is a risk of physiological indicators exceeding the tolerance range after the intervention strategy is implemented, based on the collected real-time physiological indicators of the patient.

[0013] On the other hand, the present invention also provides a decision intervention system based on reinforcement learning and digital twins, comprising: The data acquisition and preprocessing module is used to continuously collect the physiological status, clinical equipment operating parameters, test results and basic information of ICU cardiovascular patients, and to perform noise reduction, time alignment, outlier removal and standardization on the collected data to form a standardized dataset. The twin model module is used to construct a hybrid individual cardiac digital twin model, which simulates the patient's cardiovascular dynamics and predicts short-term physiological changes. The twin model is updated according to a preset period and / or changes in the patient's acute condition. The reinforcement learning strategy generation module is used to build reinforcement learning agents, simulate the effects of different intervention schemes, and generate multiple sets of candidate intervention strategies. The medical knowledge graph verification and document generation module is used to call upon a medical knowledge graph that includes patient-associated disease types, drug information, clinical guidelines, and physiological indicator standards to verify the compliance and safety of intervention strategies. If it passes, an explanatory document is generated; if it fails, a secondary generation strategy of a reinforcement learning agent is triggered. The intervention execution and model calibration module is used to distribute the validated intervention strategies and interpretation documents to clinical devices for intervention execution; extract actual efficacy data and process abnormal data after intervention, and calibrate the twin model parameters.

[0014] The beneficial effects of this invention are as follows: This invention establishes a personalized cardiovascular treatment optimization method and system for intensive care environments by combining individual cardiac digital twins with reinforcement learning. It uses individual twins to accurately simulate the patient's cardiac dynamic state and combines multimodal real-time monitoring data to dynamically correct the model, thereby ensuring that intervention strategies can be personalized to patient differences and generating significant technical and application value in clinical practice. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the decision intervention method based on reinforcement learning and digital twins of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand the present invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0017] Please see Figure 1 This embodiment addresses the clinical monitoring challenges of cardiovascular patients in the ICU by providing a decision-making intervention method based on reinforcement learning and digital twins, including: S1 continuously collects multi-dimensional patient data and preprocesses it to form a standardized dataset. This includes physiological data such as heart rate, systolic blood pressure, diastolic blood pressure, mean arterial pressure, cardiac output, central venous pressure, stroke volume, and real-time monitoring data like electrocardiogram and arterial blood pressure waveforms; clinical equipment operating parameters covering ventilator mode, inhaled oxygen concentration, positive end-expiratory pressure, tidal volume, respiratory rate, and infusion pump drug infusion rate, start-stop time, and cumulative infusion volume; laboratory test results including blood gas analysis results, electrolyte levels, hemoglobin levels, renal function indicators, inflammatory marker levels, and left ventricular ejection fraction from echocardiography; and basic information including the patient's age, gender, weight, and medical history such as cardiovascular disease, hypertension, and diabetes. In the data preprocessing process, wavelet filtering algorithm is first used to remove high-frequency noise in physiological state data. Then, time sequence alignment of all collected data is achieved by using a unified timestamp method to ensure that data from different sources are consistent in the time dimension. Subsequently, outliers that exceed the normal fluctuation range are removed to avoid interference from abnormal data to subsequent analysis. Finally, the Min-Max normalization method is used to standardize all data to a unified numerical range of [0,1], ultimately forming a standardized data set with a regular structure and reliable data.

[0018] Next, in S2, a hybrid individual cardiac digital twin model is constructed and dynamically updated. This hybrid individual cardiac digital twin model consists of a kinetic model and a parametric model. The kinetic model is based on cardiovascular physiological mechanisms and simulates the patient's cardiovascular dynamic state, such as cardiac pumping function and changes in vascular resistance, by calculating core parameters such as myocardial contractility coefficient, corrected peripheral vascular resistance, and ventricular basal volume. The parametric model adopts a feedforward neural network structure, using the core parameters output by the kinetic model and real-time data from the standardized dataset in S1 as inputs, to achieve rapid prediction of the patient's short-term physiological indicators. Together with the kinetic model, it simulates the patient's current cardiovascular dynamic state and predicts the future trends of short-term physiological indicators such as heart rate, systolic blood pressure, and cardiac output within a preset time, such as 10 minutes. In this embodiment, the model update triggering conditions are divided into two categories. One category is preset periodic triggering, which can be set to trigger a regular update every 30 minutes to ensure that the model can adapt to the slow changes in the patient's condition. The other category is triggering due to acute changes in the patient's condition. In the ICU scenario described in this embodiment, when acute fluctuations in the patient's condition are detected, such as a sudden drop in systolic blood pressure exceeding 30 mmHg or a sudden increase in heart rate exceeding 20 beats / minute, an immediate update is triggered. By adjusting the physiological parameters of the dynamic model and the network weights of the parameterized model, it is ensured that the model always remains consistent with the patient's actual condition.

[0019] Subsequently, S3 is executed, using the hybrid individual cardiac digital twin model constructed in S2 as the virtual decision-making environment for reinforcement learning. The predicted short-term physiological indicators output by the twin model, the deviation between the indicators and the doctor's treatment goals, together constitute the agent's state space. Clinically executable operations such as adjusting the infusion rate of vasoactive drugs (e.g., norepinephrine, dobutamine), adjusting fluid volume (e.g., changes in fluid volume from 50-200ml per administration), and adjusting ventilator parameters (e.g., inhaled oxygen concentration, positive end-expiratory pressure) are set as the agent's action space. During agent training, a combination of offline pre-training and online adjustment is used. Offline pre-training is first completed using standardized data from historical patients and a virtual case database generated by the twin model, and then online fine-tuning is performed using real-time data from the current patients. For each intervention action output by the agent, the twin model simulates the changes in physiological indicators after the action is executed and provides corresponding reward values ​​based on whether the indicators approach the target value or are within a safe range. Through multiple rounds of training and simulation, the agent can generate multiple sets of different candidate intervention strategies. For example, strategy one: first administer 100ml of fluid, then increase the norepinephrine infusion rate by 0.1μg / (kg). Alternatively, strategy 2 could be a combination of strategies such as adjusting the positive end-expiratory pressure of the ventilator to 8 cmH2O while maintaining the current dobutamine dose.

[0020] Next, S4 is executed, which involves performing safety verification on the candidate intervention strategy using a medical knowledge graph and generating an explanatory document. This involves calling a pre-built medical knowledge graph containing information on ICU cardiovascular treatment-related disease types such as heart failure, septic shock, and acute coronary syndrome; recommended dosage ranges, contraindications, and drug interactions for various vasoactive drugs; medical guidelines; and physiological indicator standards. During the verification process, it first determines whether the drug dosage in the candidate intervention strategy conforms to the recommended range in the knowledge graph. Then, it checks whether there are contraindications between the drug and the patient's underlying disease, whether there are interaction risks when multiple drugs are used in combination, and, in conjunction with the patient's current physiological indicators, whether the intervention might exceed the safe range. If all verification items pass, a document containing explanatory content is generated, clearly indicating the clinical guideline clauses on which the strategy is based, the drug's mechanism of action, and the expected treatment effect. If the verification fails, the reason for failure is fed back to S3, triggering the reinforcement learning agent to adjust the action generation logic based on the verification results and regenerate a candidate intervention strategy that meets safety requirements.

[0021] Finally, step S5 is executed, which involves implementing the intervention and calibrating the twin model based on feedback data. The validated intervention strategy and corresponding explanatory documents from step S4 are pushed to the ICU physician's workstation. After the physician confirms or fine-tunes the strategy based on the patient's real-time condition, the final strategy is distributed to the corresponding clinical equipment. The infusion pump automatically administers the medication, and the ventilator automatically adjusts its parameters to complete the intervention. During the intervention, data on changes in the patient's physiological indicators and abnormal equipment operation are continuously collected. After the intervention, the actual efficacy data and process abnormality data are integrated. Based on this data, the parameters of the hybrid individual cardiac digital twin model are calibrated to ensure that subsequent simulations and predictions can more accurately match the patient's individual response characteristics.

[0022] In this embodiment, the construction and operation of the hybrid individual cardiac digital twin model can be carried out using the following optional implementation methods to further improve the accuracy and real-time performance of the simulation of the cardiovascular dynamic state of ICU cardiovascular patients.

[0023] During data processing, the parametric model continuously extracts real-time physiological indicators from the preprocessed data, including heart rate, systolic blood pressure, diastolic blood pressure, mean arterial pressure, central venous pressure, and cardiac output. Simultaneously, it receives synchronous output data from the kinetic model at preset intervals, ensuring that the parametric model can promptly acquire the latest calculation results from the kinetic model. The parametric model then uses a feedforward neural network to fuse the received real-time indicators and kinetic parameters, outputting short-term physiological predictions for the patient. It also outputs the prediction uncertainty for each prediction value, which quantifies the model's confidence in the prediction results and provides a basis for subsequent model result fusion.

[0024] Furthermore, as an optional implementation of the aforementioned kinetic model, the kinetic model internally includes a ventricular function calculation module and a vascular resistance calculation module. The ventricular function calculation module takes the patient's end-diastolic volume, end-systolic volume, and myocardial contractility parameters as inputs, and calculates the stroke volume through the physiological correlation between these three parameters.

[0025] As an alternative approach, the Frank-Starling model is used, with the myocardial contractility correction coefficient K = 0.8 + 0.02 × LVEF (ejection fraction); stroke volume SV = K × (EDV) ESV)×(MAP / 80).

[0026] In calculating end-diastolic volume (EDV), the patient's body surface area and central venous pressure (CVP) are considered. Typically, body surface area is calculated first based on the patient's height and weight, and then the CVP value is used to adjust the base calculation for EDV. Higher CVP generally results in a larger calculated EDV. End-systolic volume (ESV), on the other hand, is directly related to EDV and stroke volume. It is obtained by subtracting stroke volume from EDV, reflecting the relationship between the amount of blood remaining in the ventricles after cardiac contraction and the amount of blood pumped. Specifically, EDV is estimated as follows: EDV = Body surface area × 10 + 5 × CVP (CVP is central venous pressure; body surface area = 0.0061 × height + 0.0128 × weight - 0.1529); ESV = EDV. SV.

[0027] The vascular resistance calculation module combines mean arterial pressure, central venous pressure, and cardiac output to derive peripheral vascular resistance using physiological formulas. Since the vascular elasticity coefficient decreases with age, an age-related vascular elasticity coefficient is introduced to correct the calculation results. This corrected peripheral vascular resistance calculation better reflects the physiological characteristics of increased resistance due to decreased vascular elasticity in elderly patients, thus improving the individual suitability of the calculation. The calculation is: Peripheral vascular resistance SVR = 80 × (MAP) CVP) / CO, where MAP is mean arterial pressure and CO is cardiac output. The vascular elasticity coefficient E = 1.2 - 0.01 × age is introduced and corrected to SVRcorr = SVR × E.

[0028] Furthermore, as an optional implementation of the parametric model, a three-layer network structure consisting of an input layer, a hidden layer, and an output layer is adopted. The input layer has a preset number of nodes, corresponding to two types of input parameters: one type is directly collected patient parameters, including real-time monitoring indicators such as heart rate, systolic blood pressure, diastolic blood pressure, mean arterial pressure, central venous pressure, and cardiac output; the other type is mechanistic parameters output by the kinetic model according to a preset period, namely myocardial contractility coefficient, stroke volume, and peripheral vascular resistance. This preset period is consistent with the period during which the parametric model receives kinetic data, ensuring the timeliness of the input data. The hidden layer uses a fully connected structure, typically containing two fully connected network layers, each with a certain number of nodes. This structure extracts dynamic correlation features between input parameters, capturing the synchronous trend of heart rate and cardiac output changes, the dependence of peripheral vascular resistance on systolic blood pressure, etc., while appropriate regularization methods reduce the risk of overfitting and ensure the stability of feature extraction. The output layer also has a preset number of nodes, which correspond to the predicted values ​​of five physiological indicators of the patient within a preset time period. The preset time can be set to 5-10 minutes to meet the short-term prediction needs of ICU clinical decision-making. At the same time, the output layer will also output the prediction uncertainty corresponding to each prediction value. This uncertainty is calculated based on the confidence information output by the feedforward neural network.

[0029] In the process of constructing a hybrid individual heart digital twin model, initialization training, dynamic updates and validity verification need to be completed separately. After the performance of the two models meets clinical needs, they are then merged to form a complete and reliable hybrid individual heart digital twin model.

[0030] During training, the calculation results of the kinetic model are first used as the training input for the parametric model, including parameters reflecting cardiovascular physiological state such as myocardial contractility coefficient, stroke volume, and corrected peripheral vascular resistance. Simultaneously, the root mean square error (RMSE) between the predicted physiological indicators output by the parametric model and the actual monitored values ​​is used as the training objective function. Through continuous iterative optimization, the deviation between the predicted and actual values ​​is constantly reduced. If the deviation between the systolic blood pressure predicted by the parametric model and the actual monitored systolic blood pressure is large, the objective function guides the model to adjust its internal calculation logic until the RMS error stabilizes at a low level. This completes the initial training of the parametric model, enabling it to initially possess short-term predictive capabilities based on physiological mechanisms.

[0031] The update trigger conditions in the dynamic update process are divided into two categories, which have already been explained in S2 above and will not be repeated here. During the update process, the actual changes in the patient's heart rate, blood pressure, cardiac output, and other indicators after intervention are first compared with the predicted results of the current output of the model to identify the differences between the two. Then, the weight parameters of the parameterized model are adjusted using the gradient descent method, including the connection weights between the input layer and the hidden layer, and between the hidden layer and the output layer, so that the prediction of the parameterized model is closer to the actual therapeutic effect. At the same time, the rationality of the physiological parameters of the kinetic model is verified, checking whether the myocardial contractility coefficient is within the normal physiological range for this type of patient and whether the correction results of peripheral vascular resistance conform to the age-related vascular elasticity law. If unreasonable deviations in the parameters are found, the calculation parameters of the kinetic model are adjusted synchronously to ensure that the two models match the patient's current condition.

[0032] After each model initialization training or dynamic update, a validity check is required to determine whether the model possesses reliable simulation and prediction capabilities. The root mean square error (RMSE) between the model's predicted physiological indicators and the patient's actual monitored values ​​is calculated, and this error is compared to a preset validity threshold. Optionally, a preset validity threshold is preferred; if the error is less than or equal to the threshold, the model is considered valid; if the error exceeds the threshold, the process returns to the previous training or update stage to re-optimize the model parameters. Simultaneously, the predicted trends of the physiological indicators output by the model are submitted to clinicians for manual review. Doctors, considering the patient's known medical information, assess the degree to which the predicted trends match the patient's actual condition. If the predicted trends align with the disease's progression, the model's validity is further confirmed; if the predicted trends do not match the condition, the model parameters or training process must be re-examined until the model passes the validity check.

[0033] After the architecture design of the dynamic model and the parameterized model is completed, their prediction errors are fused to eliminate fusion errors during the merging and use of the two models, ensuring that the prediction results output by the merged twin model are more accurate and reliable. Therefore, it also includes: The prediction error and corresponding prediction uncertainty of the kinetic model are calculated. The prediction error of the kinetic model is the deviation between the predicted values ​​of the output physiological indicators and the actual monitored values ​​by the patient; the difference between the two is the prediction error of the kinetic model. The prediction uncertainty of the kinetic model is determined based on the fluctuation range of its physiological parameters, including the myocardial contractility coefficient output by the ventricular function calculation module and the peripheral vascular resistance corrected by the vascular resistance calculation module. The fluctuation range of these parameters needs to be obtained through parameter rationality analysis during the model validity verification phase; the quantified result of this fluctuation range is the prediction uncertainty of the kinetic model.

[0034] Next, the prediction error and corresponding prediction uncertainty of the parametric model are calculated. The prediction error of the parametric model is the deviation between the predicted values ​​of physiological indicators output by the feedforward neural network and the actual monitored values ​​of the patients; the difference is the prediction error of the parametric model. The prediction uncertainty of the parametric model is the confidence quantification result generated simultaneously when outputting the predicted values ​​of physiological indicators. This result directly reflects the reliability of the model for the current predicted values ​​and can be obtained directly without additional calculation.

[0035] Subsequently, the error weight of the dynamic model is determined based on the prediction uncertainties of both models, specifically the ratio of the prediction uncertainty of the parametric model to the sum of the prediction uncertainties of the dynamic model and the parametric model. In this embodiment, the prediction error fusion formula for the hybrid individual cardiac digital twin model is as follows:

[0036]

[0037] in, This represents the final prediction error of the hybrid twin model at time t; The error weights of the dynamic model at time t are represented. This represents the prediction error of the dynamic model at time t; This represents the prediction error of the parameterized model at time t; This represents the prediction uncertainty of the dynamic model at time t; This represents the prediction uncertainty of the parameterized model at time t.

[0038] Using the above formula, when the uncertainty of the parameterized model is high, sudden changes in the disease lead to... When it increases, Automatically increases; when the parametric model is stable, Hour, This reduces the real-time performance of the parametric model. For example, if the prediction uncertainty of the parametric model is 10% and the prediction uncertainty of the dynamic model is 5%, then the error weight of the dynamic model is 10% ÷ (5% + 10%) ≈ 66.7%. This weight means that subsequent fusion needs to rely more on the error correction logic of the dynamic model to balance the impact of the current high uncertainty of the parametric model.

[0039] Finally, the prediction errors of the kinetic model and the parameterized model are weighted and fused according to the aforementioned error weights to obtain the final prediction error of the hybrid individual heart digital twin model. In one optional scenario, if the error weight of the kinetic model is 66.7% and the prediction error is 2 mmHg, and the error weight of the parameterized model is 33.3% and the prediction error is 5 mmHg, then the final prediction error is 66.7% × 2 mmHg + 33.3% × 5 mmHg ≈ 3 mmHg. Only when this final prediction error meets the preset threshold can it be used as a prerequisite for validating the twin model. If the final prediction error exceeds the threshold, the parameters or error weight calculation logic of the two models is readjusted until the final prediction error meets the standard, ensuring that the merged twin model has sufficient prediction accuracy and provides a reliable virtual decision-making environment for subsequent reinforcement learning agents.

[0040] In this embodiment, a hybrid individual cardiac digital twin model is used as the virtual decision-making environment for reinforcement learning during the preparation of the reinforcement learning steps. This virtual decision-making environment is used to simulate or reproduce the patient's current cardiovascular dynamics and predict future short-term physiological changes. When defining the state space, not only are the predicted physiological indicators output by the twin model included, such as heart rate, systolic blood pressure, diastolic blood pressure, mean arterial pressure, cardiac output, and central venous pressure, but the deviations between these predicted indicators and the doctor's set targets are also calculated, and these deviation values ​​are included in the state space as input parameters for the agent.

[0041] The definition of action space includes adjustments to the infusion rate of vasoactive drugs, the amount of fluid replacement, and the parameters of clinical equipment. Vasoactive drugs include commonly used drugs such as norepinephrine, dobutamine, and nitroglycerin. The parameters of clinical equipment include settings such as positive end-expiratory pressure, inhaled oxygen concentration, and tidal volume of the ventilator, ensuring that the action space is fully matched with the actual clinical intervention.

[0042] Next, a reinforcement learning agent was constructed and trained using a combination of offline pre-training and online adjustment. During offline pre-training, the agent was trained using standardized historical patient data and a large virtual case database generated by the twin model. Through iterative optimization of the policy network parameters, a basic intervention logic adapted to the ICU cardiovascular treatment scenario was initially formed. During online adjustment, the agent's strategy was fine-tuned by combining real-time monitoring data of the current patient with the dynamic prediction results of the twin model. This enabled the agent to quickly learn individual differences among patients and respond correctly to variations in drug sensitivity and the impact of underlying diseases on cardiovascular function. During training, whenever the agent outputs an intervention action, the hybrid individual cardiac digital twin model immediately simulates the physiological indicator changes after the action, generating corresponding physiological indicator changes. Simultaneously, it calculates and returns a reward value according to preset reward rules. The agent continuously adjusts its decision-making logic based on this feedback, gradually improving its ability to select the optimal intervention action.

[0043] Optionally, the reinforcement learning agent performs multi-objective proximal policy optimization based on the Proximal Policy Optimization (PPO) algorithm. This algorithm can effectively balance blood pressure control, heart rate stabilization, and cardiac function improvement while ensuring the stability of policy updates. By incorporating multi-objective reward calculation logic into the algorithm framework, the agent will not neglect other key indicators during the learning process due to the optimization of a single objective. For example, it will not pursue a rapid increase in blood pressure to the point that the heart rate exceeds the safe range, nor will it sacrifice the improvement in cardiac output to stabilize the heart rate.

[0044] After the agent is trained, multiple candidate intervention strategies are generated. Each strategy includes specific intervention details, such as the adjustment range of the infusion rate of a certain vasoactive drug, the size of a single fluid resuscitation, and the adjustment values ​​of specific ventilator parameters. To ensure that the candidate strategies can achieve stable effects under different disease conditions, they are input into a hybrid individual cardiac digital twin model. The twin model simulates different disease scenarios, including the patient's current baseline state, a state of slight deterioration, and a state of slight improvement. In each scenario, the twin model executes the simulated strategy, records changes in physiological indicators, and calculates the reward value corresponding to each strategy in that scenario. Finally, all candidate strategies are screened according to preset reward criteria, retaining those with an average reward value that meets the higher standard and whose reward value is not lower than the baseline standard in a single disease scenario.

[0045] To select candidate intervention strategies that meet preset reward criteria, it is necessary to simulate various disease scenarios and conduct quantitative evaluations using an individual cardiac digital twin model. This embodiment, based on work scenarios and experience, constructs three typical disease scenarios in the twin model: the patient's current baseline state, the state of disease deterioration, and the state of disease improvement.

[0046] The baseline state is constructed based on real-time patient monitoring data, recreating the current stable cardiovascular dynamics. The worsening state simulates scenarios such as exacerbated infection and decreased drug sensitivity, which may lead to a sudden increase in heart rate and a sudden drop in blood pressure. The improving state simulates scenarios where cardiac output increases and vascular resistance tends to normalize after the intervention takes effect. Then, each group of candidate intervention strategies is input into these three scenarios for simulation execution. The twin model outputs the changes in physiological indicators after the strategy execution in each scenario in real time, and calculates the reward value of each strategy in the corresponding scenario according to the preset reward function.

[0047] Finally, strategies are selected based on reward values: Calculate the average reward value of each candidate strategy group across the three scenarios, then compare the average reward value with a preset high reward standard. Simultaneously, check whether the reward value of each strategy group in a single scenario is lower than the preset baseline standard. Finally, only strategies whose average reward value meets the preset high reward standard and whose single-scenario reward value in the baseline state, worsening condition state, and improving condition state is not lower than the preset baseline standard are retained.

[0048] In this embodiment, compliance and security verification of the intervention strategy includes: Candidate intervention strategies are decomposed into triplets containing the patient's condition and expected outcome of the intervention, and then converted into a structured format recognizable by a medical knowledge graph. The medical knowledge graph, containing information on ICU cardiovascular treatment-related disease types, drug information, physiological indicators, standards, and clinical guidelines, is then used by a rule engine to perform compliance and safety checks on the candidate intervention strategies. The rule engine operates through specific query statements and rule matching logic. On one hand, it works forward from the patient's condition to match whether the intervention is suitable for the individual patient; on the other hand, it works backward from the intervention strategy to check for rules that conflict with the patient's condition, ensuring a comprehensive and accurate verification process.

[0049] The compliance verification process involves three main steps: First, verifying whether the drug dosage conforms to the clinical guidelines' recommended range. This is done by comparing the initial dosage with the guidelines stored in the knowledge graph to determine if the initial dosage adjustment and maximum daily dosage within the recommended range. Second, verifying whether there are contraindications between the drugs and the patient's underlying diseases. This involves querying the knowledge graph to find the association between the patient's current underlying diseases and the drugs in the strategy. If the patient has severe tachycardia (heart rate exceeding 140 beats per minute), the strategy is checked to see if drugs like dobutamine, which may worsen tachycardia, are included. Third, verifying the potential for drug interactions when multiple drugs are used in combination. If a candidate strategy includes two or more drugs, the knowledge graph is used to confirm whether there are any interactions between these drugs that could affect efficacy or increase adverse reactions. The safety verification is based on real-time physiological indicators collected from the patient to determine if there is a risk of physiological indicators exceeding the tolerable range after the intervention strategy is implemented.

[0050] At this point, if the candidate intervention strategy passes the compliance and safety verification, the system will automatically generate a corresponding explanatory document. This document records the medical basis and implementation details of the strategy, ensuring clinicians can clearly understand its rationale. The document contains both structured and natural language components. The structured portion clearly indicates the rule identifier upon which the verification was based, the corresponding clinical guideline name and specific clauses, and the correction basis for drug dosages or equipment parameters. The generated explanatory document is named according to the patient identifier, strategy version number, and verification time, and stored in an encrypted database. Its retention period is consistent with the patient's ICU stay, facilitating subsequent clinical traceability and auditing.

[0051] If a candidate intervention strategy fails the validation, the specific reason for failure is output. The reason for failure and the corresponding validation rules are then fed back to the reinforcement learning agent, triggering a second process for generating candidate intervention strategies. After receiving the feedback, the reinforcement learning agent adjusts its strategy generation logic based on the reason for failure. If the test fails due to dosage exceeding the range, the action space boundary for drug adjustment will be narrowed to avoid generating overdose strategies again. If the drug fails to pass due to contraindications, the contraindicated drug will be removed from the action space, and the alternative drug recommended by the guideline will be selected first. If the drug interaction test fails, the drug combination or dosage ratio will be adjusted.

[0052] Simultaneously, the AI ​​will access data from the historical safety intervention sample pool, prioritizing the generation of new strategies based on verified safe action patterns to improve the pass rate of secondary generated strategies. The secondary generated candidate intervention strategies re-enter the compliance and safety verification process of the medical knowledge graph until the strategy passes verification; if multiple secondary generation attempts still fail, the system will trigger a manual review prompt, allowing clinicians to intervene and evaluate, thus avoiding delays in treatment due to automatic system processing.

[0053] On the other hand, this embodiment also provides a decision intervention system based on reinforcement learning and digital twins, including: The data acquisition and preprocessing module is used to continuously collect the physiological status, clinical equipment operating parameters, test results and basic information of ICU cardiovascular patients, and to perform noise reduction, time alignment, outlier removal and standardization on the collected data to form a standardized dataset. The twin model module is used to construct a hybrid individual cardiac digital twin model, which simulates the patient's cardiovascular dynamics and predicts short-term physiological changes. The twin model is updated according to a preset period and / or changes in the patient's acute condition. The reinforcement learning strategy generation module is used to build reinforcement learning agents, simulate the effects of different intervention schemes, and generate multiple sets of candidate intervention strategies. The medical knowledge graph verification and document generation module is used to call a medical knowledge graph that includes patient-associated disease types, drug information, clinical guidelines, and physiological indicator standards to verify the compliance and safety of intervention strategies. If it passes, an explanatory document is generated; if it fails, a secondary generation strategy of reinforcement learning agent is triggered. The intervention execution and model calibration module is used to distribute the validated intervention strategies and interpretation documents to clinical devices for intervention execution; extract actual efficacy data and process abnormal data after intervention, and calibrate the twin model parameters.

[0054] The embodiments of the present invention have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A decision intervention method based on reinforcement learning and digital twins, characterized in that, include: S1. Continuously collect patients' physiological status, clinical equipment operating parameters, test results and basic information, and perform noise reduction, time alignment, outlier removal and standardization on the collected data to form a standardized data set; S2. Construct a hybrid individual cardiac digital twin model, which is used to simulate the patient's cardiovascular dynamics and predict short-term physiological index changes, and is updated according to a preset period and / or changes in the patient's acute condition. S3. Construct a reinforcement learning agent to simulate the effects of different intervention schemes and generate multiple sets of candidate intervention strategies. S4. Call upon a medical knowledge graph that includes patient-associated disease types, drug information, clinical guidelines, and physiological indicator standards to verify the compliance and safety of the intervention strategy; If it passes, an explanatory document is generated; if it fails, a secondary generation strategy for the reinforcement learning agent is triggered. S5. Distribute the validated intervention strategy and interpretation documents to clinical devices for intervention; extract actual efficacy data and process abnormal data after intervention, and calibrate the twin model parameters.

2. The decision intervention method based on reinforcement learning and digital twins according to claim 1, characterized in that, The hybrid individual heart digital twin model includes a dynamic model and a parameterized model; the dynamic model is used to calculate cardiac structural and functional parameters, and the parameterized model is configured with a feedforward neural network for real-time prediction. The kinetic model is used to output myocardial contractility coefficient, stroke volume and peripheral vascular resistance. The parameterized model receives real-time collected physiological indicators and synchronous output data of the kinetic model in a preset cycle. It outputs the patient's physiological prediction value and prediction uncertainty through a feedforward neural network.

3. The decision intervention method based on reinforcement learning and digital twins according to claim 2, characterized in that, The dynamic model includes a ventricular function calculation module and a vascular resistance calculation module; The ventricular function calculation module calculates stroke volume based on the patient's end-diastolic volume, end-systolic volume, and myocardial contractility parameters. The vascular resistance calculation module combines mean arterial pressure, central venous pressure, and cardiac output to calculate peripheral vascular resistance, and introduces a vascular elasticity coefficient related to the patient's age to correct the peripheral vascular resistance. Among them, the calculation of end-diastolic volume is related to the patient's body surface area and central venous pressure, while the calculation of end-systolic volume is related to end-diastolic volume and stroke volume.

4. The decision intervention method based on reinforcement learning and digital twins according to claim 2, characterized in that, The parameterized model includes an input layer, a hidden layer, and an output layer; The input layer has a preset number of nodes, and the input parameters it accepts include directly collected patient parameters and mechanistic parameters output by the dynamic model according to a preset period. The hidden layer is configured with a fully connected structure to extract dynamic correlation features between input parameters; The output layer has preset nodes for outputting the predicted values ​​of the patient's heart rate, systolic blood pressure, diastolic blood pressure, cardiac output, and central venous pressure within a preset time period, and outputting the prediction uncertainty corresponding to each predicted value.

5. A decision intervention method based on reinforcement learning and digital twins according to claim 2, characterized in that, After the dynamic model and the parameterized model complete initialization training, dynamic updates, and validity verification, they are merged to form a twin model: The initialization training includes: using the output of the dynamic model as the training input of the parameterized model, and using the root mean square error between the model's predicted value and the actual monitored value as the objective function to complete the initialization training; The dynamic update includes: triggering an immediate update at preset intervals and / or when the patient experiences an acute change in condition, comparing clinical efficacy data with the current prediction results of the model, adjusting the weight parameters of the parameterized model through gradient descent, and verifying the rationality of the physiological parameters of the dynamic model. The validity verification includes: after each model initialization or update, calculating the root mean square error between the model's predicted value and the patient's actual monitored value; if the error meets the preset validity threshold, the model is deemed valid; and the model's predicted trend is submitted to the clinic for review to confirm the match with the patient's known condition.

6. A decision intervention method based on reinforcement learning and digital twins according to claim 5, characterized in that, The process of merging the models into a twin model also includes eliminating the fusion error between the dynamic model and the parametric model, including: The prediction error and prediction uncertainty of the kinetic model are calculated. The prediction error of the kinetic model is the deviation between the predicted value of the physiological index and the actual monitored value of the patient. The prediction uncertainty is determined based on the fluctuation range of the corresponding physiological parameter. The prediction error and prediction uncertainty of the parameterized model are calculated. The prediction error of the parameterized model is the deviation between the predicted value of the physiological index output by the feedforward neural network and the actual monitored value of the patient. The prediction uncertainty is the confidence quantification result of the corresponding predicted value output by the model. The error weight of the dynamic model is determined based on the prediction uncertainties of both models. The error weight is the ratio of the prediction uncertainty of the parameterized model to the sum of the prediction uncertainties of the dynamic model and the parameterized model. The prediction errors of the dynamic model and the parameterized model are weighted and fused according to the error weights to obtain the final prediction error of the hybrid individual heart digital twin model; when the final prediction error meets the preset threshold, it serves as the prerequisite for validating the twin model.

7. A decision intervention method based on reinforcement learning and digital twins according to claim 1, characterized in that, The construction of the reinforcement learning agent, simulating the effects of different intervention schemes, and generating multiple sets of candidate intervention strategies include: Using a hybrid individual cardiac digital twin model as a virtual decision-making environment for reinforcement learning, the state space is defined as the predicted physiological indicators output by the twin model and the deviation between these indicators and the doctor's set goals, and the action space is defined as the adjustment of vasoactive drug infusion rate, fluid volume, and clinical equipment parameters. A reinforcement learning agent is constructed. The agent is trained through offline pre-training and online adjustment. After the agent outputs an intervention action, the twin model returns the corresponding physiological indicator changes and reward value. After the agent is trained, multiple sets of candidate intervention strategies are generated. Different disease scenarios are simulated in the twin model and the reward value of each strategy is calculated. Candidate intervention strategies that meet the preset reward criteria are then selected.

8. A decision intervention method based on reinforcement learning and digital twins according to claim 7, characterized in that, The reinforcement learning agent performs multi-objective proximal policy optimization based on the proximal policy optimization algorithm. The candidate intervention strategies include infusion rate adjustment schemes for vasoactive drugs, fluid resuscitation volume adjustment schemes, and ventilator parameter adjustment schemes. The selection of candidate intervention strategies that meet the preset reward criteria includes: simulating three scenarios—baseline state, disease deterioration, and disease improvement—in an individual cardiac digital twin model, and calculating the average reward value of each candidate strategy in the three scenarios; retaining strategies whose average reward value meets the preset high reward criteria and whose reward value in a single scenario is not lower than the preset basic criteria.

9. A decision intervention method based on reinforcement learning and digital twins according to claim 1, characterized in that, The compliance and security verification of the intervention strategy includes: The candidate intervention strategy is decomposed into a triplet consisting of intervention measures, patient status, and expected effect, and then converted into a structured format recognizable by a medical knowledge graph. The system utilizes a medical knowledge graph containing information on ICU cardiovascular treatment-related disease types, drug information, physiological indicators, and clinical guidelines, and employs a rule engine to verify the compliance and safety of candidate intervention strategies. The compliance verification includes verifying whether the drug dosage is within the range recommended by clinical guidelines, whether there is a contraindication between the drug and the patient's underlying disease, and whether there is a risk of interaction when multiple drugs are used in combination. Safety verification includes assessing whether there is a risk of physiological indicators exceeding the tolerance range after the intervention strategy is implemented, based on the collected real-time physiological indicators of the patient.

10. A decision intervention system based on reinforcement learning and digital twins, characterized in that, include: The data acquisition and preprocessing module is used to continuously collect patients' physiological status, clinical equipment operating parameters, test results and basic information, and to perform noise reduction, time alignment, outlier removal and standardization on the collected data to form a standardized dataset. The twin model module is used to construct a hybrid individual cardiac digital twin model, which simulates the patient's cardiovascular dynamics and predicts short-term physiological changes. The twin model is updated according to a preset period and / or changes in the patient's acute condition. The reinforcement learning strategy generation module is used to build reinforcement learning agents, simulate the effects of different intervention schemes, and generate multiple sets of candidate intervention strategies. The medical knowledge graph verification and document generation module is used to call a medical knowledge graph, including patient-associated disease types, drug information, clinical guidelines, and physiological indicator standards, to verify the compliance and safety of the intervention strategy. If it passes, an explanatory document is generated; if it fails, a secondary generation strategy for the reinforcement learning agent is triggered. The intervention execution and model calibration module is used to distribute the validated intervention strategies and interpretation documents to clinical devices for intervention execution; extract actual efficacy data and process abnormal data after intervention, and calibrate the twin model parameters.

Images