Disease risk stratification method and system based on reinforcement learning

Abstract

The application discloses a disease risk stratification method and system based on reinforcement learning, relates to the technical field of reinforcement learning, and comprises the following steps: acquiring multi-modal time series data of a target patient in a monitoring process; dynamically fusing the multi-modal time series data to extract a patient state representation vector; inputting the patient state representation vector into a pre-trained risk stratification model to output a risk stratification action according to a current strategy; generating and pushing a clinical monitoring prompt according to the risk stratification action; after a preset time window, acquiring response data of the patient to clinical intervention, calculating a reward signal, and updating the risk stratification model by using the reward signal. The technical problems that the existing disease risk stratification model is static and fixed, cannot dynamically optimize a decision strategy according to the effect of clinical intervention, and results in insufficient accuracy of risk stratification results are solved.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning technology, specifically to a disease risk stratification method and system based on reinforcement learning. Background Technology

[0002] With the deepening of medical informatization construction, the multi-source heterogeneous data generated by intensive care units and various clinical monitoring scenarios are growing exponentially. Disease risk stratification is a core link in clinical early warning and resource allocation, and its accuracy is related to the improvement of patient prognosis and the control of medical costs.

[0003] However, traditional disease risk stratification methods rely on artificial feature engineering with fixed weights, which makes it difficult to capture the dynamic evolution of patient status. Furthermore, the scoring update frequency is limited, making it unable to adapt to rapid changes in the condition and lacking the ability to self-optimize based on clinical feedback. Summary of the Invention

[0004] This application provides a disease risk stratification method and system based on reinforcement learning, which solves the technical problem that existing disease risk stratification models are static and fixed, and cannot dynamically optimize decision-making strategies based on clinical intervention effects, resulting in insufficient accuracy of risk stratification results.

[0005] The technical solution to the above-mentioned technical problems in this application is as follows: Firstly, this application provides a disease risk stratification method based on reinforcement learning, the method comprising: Acquire multimodal time-series data of the target patient during monitoring; The multimodal time-series data are dynamically fused to extract patient state representation vectors; The patient state representation vector is input into a pre-trained risk stratification model, which outputs a risk stratification action according to the current strategy, wherein the risk stratification action corresponds to classifying the patient into one of a set of multiple risk levels. Based on the aforementioned risk stratification actions, generate and push clinical monitoring prompts; After a preset time window, the patient's response data to the clinical intervention is acquired, a reward signal is calculated based on the response data, and the risk stratification model is updated using the reward signal.

[0006] Secondly, this application provides a disease risk stratification system based on reinforcement learning, including: The data acquisition module is used to acquire multimodal time-series data of the target patient during the monitoring process; The data fusion module is used to dynamically fuse the multimodal time-series data and extract patient state representation vectors; The model training module is used to input the patient state representation vector into a pre-trained risk stratification model, which outputs a risk stratification action according to the current strategy, wherein the risk stratification action corresponds to classifying the patient into one of a set of multiple risk levels. The prompt generation module is used to generate and push clinical monitoring prompts based on the risk stratification actions; The model update module is used to acquire patient response data to the clinical intervention after a preset time window, calculate a reward signal based on the response data, and update the risk stratification model using the reward signal.

[0007] This application provides one or more technical solutions, which have at least the following technical effects or advantages: This application provides a disease risk stratification method and system based on reinforcement learning. First, it acquires multimodal time-series data of the target patient during monitoring, achieving a comprehensive understanding of the patient's physiological state. Second, it extracts patient state representation vectors through a dynamic fusion mechanism, capturing the temporal dependencies and complementary information between different modalities, thus improving the accuracy and robustness of state representation. Third, it constructs a risk stratification model using a reinforcement learning framework, modeling risk stratification decisions as a sequential decision-making process, enabling the model to adaptively adjust the stratification strategy based on dynamic changes in the patient's state. Finally, it constructs reward signals using clinical intervention response data and feeds them back to optimize the model, forming a closed-loop learning mechanism of "decision-execution-evaluation-optimization," allowing the risk stratification strategy to continuously iterate and improve, enhancing the timeliness and accuracy of risk stratification.

[0008] Through the above technical solution, this application can integrate multi-source heterogeneous data in real time, dynamically assess the patient's risk status, and continuously optimize the decision-making strategy based on the clinical intervention effect. It effectively overcomes the shortcomings of traditional methods such as static solidification, delayed response, and lack of self-optimization ability, and improves the accuracy and clinical applicability of risk stratification for critically ill patients. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the disease risk stratification method based on reinforcement learning provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of a disease risk stratification system based on reinforcement learning provided in an embodiment of this application.

[0011] The components represented by each number in the attached diagram are explained below: Data acquisition module 11, data fusion module 12, model training module 13, prompt generation module 14, and model update module 15. Detailed Implementation

[0012] This application provides a disease risk stratification method and system based on reinforcement learning, which addresses the technical problem that existing disease risk stratification models are static and cannot dynamically optimize decision-making strategies based on clinical intervention effects, resulting in insufficient accuracy of risk stratification results.

[0013] Example 1, as Figure 1 As shown in the embodiments of this application, a disease risk stratification method based on reinforcement learning is provided, including: S10: Acquire multimodal time-series data of the target patient during monitoring; The multimodal time-series data includes at least vital sign monitoring data, laboratory test results, clinical intervention records, and electronic medical record text data, as well as at least one of continuous physiological signals, medical imaging features, or genomic data from wearable devices.

[0014] In this embodiment, vital sign monitoring data includes continuous acquisition of parameters such as heart rate, blood pressure, respiratory rate, and blood oxygen saturation. The sampling frequency can be configured according to the clinical scenario; for example, it is usually set to once per minute in intensive care unit scenarios and once every fifteen minutes in general ward scenarios. Laboratory test results include time-series records of test items such as complete blood count, biochemical indicators, blood gas analysis, and coagulation function. For data with non-fixed test intervals, a method combining nearest neighbor interpolation and physiological constraints is used for time alignment.

[0015] The electronic medical record text data extracts key clinical information through natural language processing technology, including chief complaint, present medical history, diagnosis record, and description of changes in the condition in the course of the disease. The medical named entity recognition model is used to identify entities such as disease name, symptoms, and drugs, and encodes them into structured feature vectors.

[0016] Furthermore, continuous physiological signals from wearable devices, including gait, sleep, and heart rate variability data collected by devices such as smart bracelets and ECG patches, are transmitted in real time to a data center via Bluetooth or IoT protocols. Medical imaging features are extracted from chest X-rays, CT images, and ultrasound images. Genomic data includes information on drug metabolism-related gene polymorphisms and disease susceptibility loci.

[0017] S20: Dynamically fuse the multimodal time-series data to extract the patient state representation vector; In this embodiment of the application, the dynamic fusion process first preprocesses the multimodal time series data, including time alignment, missing value imputation and outlier detection, and extracts the patient state representation vector, which is used to describe the patient's pathophysiological state at the current moment.

[0018] In this process, time alignment employs a physiological event-based time anchoring method, establishing a unified time coordinate system based on key clinical events to ensure that data from different sampling frequencies can be compared and analyzed on the same time scale. Missing value imputation uses a differentiated strategy based on data type. For continuous vital sign monitoring data, linear interpolation is performed using preceding and following observations, and the reasonableness is verified by combining this with the patient's historical data distribution. After data preprocessing, the structured time-series data is input into a bidirectional long short-term memory network based on an attention mechanism to obtain the patient state representation vector.

[0019] Specifically, step S20 in the method includes: The multimodal time series data is time-aligned and missing values ​​are filled to obtain structured time series data; The structured time-series data is input into a bidirectional long short-term memory network based on an attention mechanism to extract the time-dependent features within each modality; The patient state representation vector is generated by interactively fusing the temporal dependent features of each modality through a cross-modal attention fusion layer.

[0020] In this embodiment, firstly, when performing time alignment on multimodal time series data, a differentiated alignment strategy is designed based on the sampling characteristics of different data sources.

[0021] Specifically, for high-frequency vital sign monitoring data, a sliding window aggregation method is used to calculate statistical characteristics, including mean, standard deviation, maximum, minimum, and coefficient of variation, with a five-minute time granularity. This method preserves the dynamic changes in the data while reducing computational complexity. For low-frequency, discrete laboratory test data, a nearest neighbor matching mechanism based on test timestamps is established. When multiple tests exist within the same time period, they are sorted and integrated according to test priority, which is determined based on the weights of key indicators for assessing acute and critical illnesses in clinical guidelines.

[0022] Missing value imputation introduces a collaborative imputation method based on patient similarity. First, a patient similarity network is constructed. By calculating the demographic characteristics, underlying disease spectrum, and Jaccard similarity coefficient of admission diagnoses of patients in historical cases, a reference group similar to the current patient's condition is identified. For missing data of a specific modality, conditional probability imputation is performed using the observed distribution of the reference group at that moment. Simultaneously, physiologically reasonable range constraints are set to avoid imputation values ​​exceeding medical common sense. For example, blood oxygen saturation imputation values ​​are forcibly limited to the range of 70% to 100%. Imputation results exceeding this range trigger an anomaly alarm and switch to a conservative estimation strategy.

[0023] Furthermore, the preprocessed structured temporal data is input into a bidirectional long short-term memory network based on an attention mechanism. This network contains two long short-term memory network branches: a forward branch and a backward branch. The forward branch processes the data in chronological order to capture the evolution trend of the patient's state from the past to the present, while the backward branch processes the data in chronological order to explore the reverse cue effect of the future state on the current assessment. The hidden state vectors of the two branches are spliced ​​and fused to form an intramodal representation containing bidirectional temporal dependent information.

[0024] Specifically, the attention mechanism is applied to the hidden layer output of the Long Short-Term Memory (LSTM) network, calculating the contribution weight of the hidden state to the representation of the current state at different times. The attention score is obtained by performing a dot product operation between the learnable query vector and the hidden state, followed by softmax normalization and weighted summation to obtain the context-aware feature vectors for each modality. For vital sign monitoring data, the attention mechanism can automatically identify subtle signs before the condition worsens, such as abnormal temperature fluctuations or decreased heart rate variability in sepsis patients, assigning higher feature weights to key time windows.

[0025] The cross-modal attention fusion layer adopts a hierarchical fusion architecture. First, primary fusion is performed within homogeneous modal groups, integrating data from different monitoring devices that reflect similar physiological systems. For example, central venous pressure, pulmonary artery wedge pressure, and invasive blood pressure signals are fused into a comprehensive representation of circulatory function. Subsequently, advanced fusion is performed between heterogeneous modalities, establishing cross-attention mappings between modalities. This enhances the sensitivity of interpreting changes in corresponding vital signs by improving the interpretation of abnormal biochemical indicators from laboratory tests. A modality reliability assessment mechanism is introduced during the fusion process. When the data quality of a modality deteriorates due to equipment failure or transmission interruption, the weight of that modality in the fusion is automatically reduced, ensuring the stability of the state representation.

[0026] S30: Input the patient state representation vector into a pre-trained risk stratification model, and the risk stratification model outputs a risk stratification action according to the current strategy, wherein the risk stratification action corresponds to classifying the patient into one of a set of multiple risk levels; In this embodiment, the risk stratification model is constructed using a deep reinforcement learning framework. The patient's state representation vector serves as the environmental state input, and the risk level classification decision serves as the agent's action output. A policy network is used to map the state space to the action space. The preset risk levels typically include four tiers: low risk, low-to-medium risk, medium-to-high risk, and high risk. Each tier corresponds to a differentiated clinical monitoring frequency and intervention intensity. For example, a low-risk tier suggests a higher frequency of routine nursing rounds, while a high-risk tier triggers a multidisciplinary consultation alert and prepares for transfer to the intensive care unit.

[0027] Furthermore, the policy network adopts an Actor-Critic architecture. The Actor network is responsible for generating the probability distribution of risk-layered actions, while the Critic network evaluates the value function of the current state-action pair. Both share a low-level feature extraction layer to reduce parameter size and accelerate convergence. The output layer of the Actor network uses the Gumbel-Softmax reparameterization technique to transform discrete risk level selection into a differentiable operation, supporting end-to-end gradient backpropagation optimization. The Critic network employs a dual-network structure, mitigating the Q-value overestimation problem by minimizing the mean squared error of the two independent value estimates, thereby improving the stability of policy evaluation.

[0028] For example, the policy network is trained using a proximal policy optimization algorithm, which avoids policy collapse during training by limiting the policy update magnitude. During training, an experience replay buffer of 10,000 transition samples is set, and 64 samples are randomly selected from the buffer for mini-batch updates each time. The learning rate of the Actor network is set to 3e-4, the learning rate of the Critic network is set to 1e-3, the discount factor γ is set to 0.99, and the GAE parameter λ is set to 0.95. The training cycle is set to 500 training epochs, each containing 2000 steps of environment interaction. Training stops when the hierarchical accuracy of the policy network on the validation set converges to 92%.

[0029] Specifically, before inputting the patient state representation vector into the reinforcement learning risk stratification model, the following steps are also included: The patient state representation vector is input into the causal inference module, which, based on a pre-constructed disease causal graph, identifies key features that have a causal effect on risk stratification in the current state and generates a causal enhancement representation vector. The causal enhanced representation vector is used as the input to the reinforcement learning risk hierarchical model; The causal inference module employs counterfactual reasoning to quantify the causal effects of different clinical interventions on patient prognosis and integrates these causal effects into the generation process of the causal enhancement representation vector.

[0030] The risk stratification model adopts a policy gradient network architecture, whose state space is composed of the patient state representation vectors and whose action space consists of the preset multiple risk levels.

[0031] In this embodiment, the causal inference module is first constructed based on a disease development causal knowledge graph collected from large-scale electronic medical record data. This graph includes the etiological chains, complication associations, and treatment response paths of common acute and critical illnesses. After receiving the patient state representation vector, the causal inference module performs message passing on the causal graph through a graph neural network to identify disease progression subgraphs that highly match the current patient state, thereby locating key node features with intervention sensitivity.

[0032] For example, for patients suspected of having sepsis, the causal inference module can distinguish the applicable conditions of two intervention paths: infection source control and fluid resuscitation, assess the differences in expected prognosis under different paths, and encode the strength of the causal effect as an additional feature dimension, which is then concatenated with the original state representation to form a causal enhanced representation vector.

[0033] Secondly, the counterfactual reasoning process employs a dual robust estimation method based on propensity score matching. First, a control group with similar baseline characteristics to the current patient is selected from the historical case database. Clinical intervention plans under different risk stratification decisions are simulated to predict the corresponding potential prognostic outcomes. By comparing the difference between the factual and counterfactual outcomes, the causal effect size of the specific intervention is calculated. This effect size serves as a dynamic weight to adjust the sensitivity of the risk stratification model to the corresponding clinical characteristics, enabling the model to prioritize changes in indicators with clear causal intervention value when making decisions.

[0034] Furthermore, the state space dimension of the policy gradient network architecture is determined by the length of the causal reinforcement representation vector, the action space is defined as a discrete set of risk levels, and the policy function is represented by a parameterized probability distribution. During network training, an entropy regularization term is introduced to encourage policy exploration and avoid premature convergence to local optima; at the same time, an importance sampling truncation technique is used to limit the magnitude of policy updates and ensure the stability of the learning process.

[0035] For example, when a patient’s condition characterization exhibits early features of acute respiratory distress syndrome, the risk level probability distribution output by the strategy network may be low risk 15%, low-to-medium risk 25%, medium-to-high risk 35%, and high risk 25%, which reflects the uncertainty of the current condition.

[0036] S40: Generate and push clinical monitoring prompts based on the aforementioned risk stratification actions; In this embodiment, clinical monitoring prompts are generated and pushed based on risk stratification actions, that is, the risk stratification decisions of the model are transformed into actionable clinical guidance information, realizing a closed-loop connection from intelligent analysis to clinical action. The generation of clinical monitoring prompts follows the principle of hierarchical progression, and differentiated prompt content and push strategies are designed for different risk levels to ensure the clinical relevance and operability of the prompt information.

[0037] S50: After a preset time window, acquire the patient's response data to the clinical intervention, calculate a reward signal based on the response data, and update the risk stratification model using the reward signal.

[0038] In this embodiment, the preset time window is dynamically adjusted based on the clinical scenario and disease type. For acutely ill patients, it is typically set to four to eight hours, while for patients with chronic diseases, it can be extended to twenty-four to seventy-two hours. The starting point of the time window is based on the output time of the risk stratification action, ensuring a clear and traceable causal relationship between the intervention and the assessment response. Patient response data to clinical intervention includes comprehensive changes in multiple prognostic indicators, mainly covering dimensions such as the degree of improvement in physiological indicators, organ function recovery, complication rate, and final clinical outcome.

[0039] Specifically, the improvement in physiological indicators is obtained by calculating the normalized rate of change of key vital signs within a time window. For example, for patients with heart failure, the focus is on monitoring the decline in brain natriuretic peptide levels and the recovery of urine output; for patients with sepsis, the focus is on the changing trends in lactate clearance rate and the required dosage of vasoactive drugs. The recovery of organ function is quantified using changes in the sequential organ failure score. A decrease in the score indicates effective intervention, while an increase or a persistently high score suggests the need to adjust the treatment strategy. The incidence of complications, such as new infections, bleeding events, and arrhythmias, within the statistical time window is recorded as an important component of the negative reward. The final clinical outcome is determined at the time of patient discharge or transfer to another department, including different endpoints such as survival upon discharge, in-hospital death, and voluntary discharge.

[0040] Furthermore, the reward signal calculation employs a multi-objective weighted fusion mechanism, transforming the response data from each dimension into scalar reward values. Positive rewards stem from improvements in physiological indicators, recovery of organ function, and favorable clinical outcomes, while negative rewards correspond to the occurrence of complications, disease deterioration, or poor prognosis. The reward function design incorporates prior knowledge from clinical experts to calibrate the weights of different prognostic indicators, ensuring consistency between the reward signal and clinical value judgments. For example, for intensive care patients, the weight of avoiding in-hospital mortality is significantly higher than that of simple improvements in physiological indicators, while for patients in the recovery phase, the quality of functional recovery and self-care ability receive higher weights.

[0041] Furthermore, an off-policy learning framework is employed to update the risk stratification model using reward signals. This framework allows the model to learn efficiently from historical experience. The experience replay buffer stores state transition tuples, including patient state representation vectors, risk stratification actions, reward signals, and the state representation for the next time step. During model updates, high-value transition samples are preferentially sampled from the buffer. The sampling priority is determined based on the absolute value of the temporal difference error, causing the model to focus more on state-action pairs with larger prediction biases, thus accelerating the convergence process.

[0042] Specifically, the policy network's parameter updates employ an importance sampling ratio with pruning to limit the difference between the old and new policies and avoid performance oscillations during the policy update process. The value network's update objective is to estimate the expected value of the discounted cumulative reward, which is optimized by minimizing the Bellman residual squared loss. The Bellman residual squared loss is a measure of the deviation between the current predicted value and the target value, which consists of the discounted sum of the immediate reward and the next state value estimate.

[0043] Furthermore, the method also includes: When the risk stratification action output by the reinforcement learning risk stratification model changes, a risk level change event is triggered, and the patient state representation vector, intervention measures and subsequent outcomes before and after the change are recorded to form an experience playback sample. The experience replay samples are stored in the experience pool for periodic offline training and updating of the reinforcement learning risk stratification model.

[0044] In this embodiment, the triggering mechanism for risk level change events is designed to be state-sensitive. When the outputs of two consecutive risk stratification decisions are inconsistent, it is determined to be a valid change event, rather than a random fluctuation of a single decision. The recorded content of the change event adopts a structured format, including the risk level before the change, the risk level after the change, the patient's complete state representation vector at the time of the change, the main characteristic change identifiers that triggered the change, the sequence of intervention measures actually taken clinically, and the actual prognostic outcome after a preset time window. This recording format supports subsequent counterfactual analysis, facilitating the assessment of possible alternative outcomes if the original risk level decision were maintained.

[0045] Secondly, the storage management of the experience pool adopts a hierarchical sampling architecture, which divides the pool into multiple sub-pools according to the risk level and change type. These include upgrade change sub-pools, such as upgrading from low-to-medium risk to high risk; downgrade change sub-pools, such as downgrading from high risk to low-to-medium risk; and same-level adjustment sub-pools, such as fine-tuning within the medium-to-high risk pool.

[0046] Furthermore, each sub-pool has an independent capacity limit and retention strategy. Upgraded sub-pools, due to their higher clinical decision-making value, are allocated a larger capacity proportion and employ a time-decay strategy that prioritizes retaining recent samples. Same-level adjustment sub-pools focus on retaining boundary cases with significant characteristic changes to avoid excessive accumulation of homogeneous samples. The total capacity of the experience pool is set at 500,000 samples. When the storage limit is reached, samples are eliminated based on a comprehensive score combining temporal difference error and clinical outcome severity, prioritizing the retention of samples with high learning value and clinical warning significance.

[0047] The periodic offline training update mechanism employs an asynchronous execution strategy, decoupling it from the online inference service to ensure that model updates do not impact the response performance of the real-time risk stratification service. Offline training tasks are triggered at preset intervals, typically weekly for a full training run, or earlier when the experience pool reaches 20% of its total capacity. During training, the policy network parameters used by the online service are frozen, and a replica is loaded onto independent computing resources for gradient updates. After the update, A / B testing is used to verify the new model's performance. The new model parameters are only synchronized to the online service when the stratification accuracy on the validation set improves by more than 2% and the policy stability metric meets the requirements.

[0048] Specifically, offline training employs an improved proximal policy optimization algorithm, combining historical samples from the experience pool with simulated synthetic samples for joint training. The synthetic samples are generated through a conditional generative adversarial network (GAN) to augment data for rare critical illness scenarios where samples in the experience pool are sparsely distributed. For example, for low-incidence but high-fatality disease states such as fulminant myocarditis and amniotic fluid embolism, synthetic patient state representations with reasonable physiological parameter correlations are generated, expanding the coverage and diversity of the training data.

[0049] Furthermore, the method is applied to risk stratification of various acute and critical illnesses, including at least one of sepsis, acute kidney injury, acute respiratory distress syndrome, aspiration pneumonia, intracranial hypertension, deep vein thrombosis and pulmonary embolism, gastrointestinal bleeding, or malignant arrhythmia. The reinforcement learning risk stratification model shares the underlying feature extraction network through a multi-task learning architecture and outputs corresponding risk stratification actions for different diseases.

[0050] In this embodiment, the core design of the multi-task learning architecture lies in identifying the common pathophysiological mechanisms and specific clinical manifestations among different acute and critical illnesses. Knowledge transfer is achieved through a shared underlying feature extraction network, while preserving the disease-specific hierarchical decision-making paths. The underlying feature extraction network employs a hybrid structure of stacked temporal convolutional layers and bidirectional long short-term memory networks. The temporal convolutional layers capture local fluctuation patterns of the patient's vital signs, while the bidirectional long short-term memory network models long-term dependencies and extracts dynamic trajectory features of disease progression. The dimension of the general representation vector output by the shared network is set to 512, which is then connected to a disease-specific adapter module. This module consists of lightweight fully connected layers and learns the mapping from the general representation to the specific risk stratification space for each disease.

[0051] Specifically, the disease-specific adapter module employs a residual connection design, allowing the model to fully utilize shared features while enhancing its sensitivity to disease-specific signals. For the sepsis task, the adapter strengthens the weighting of dynamic changes in infection biomarkers and organ perfusion indicators; for the acute respiratory distress syndrome task, it prioritizes the adjustment response of oxygenation index, lung compliance, and mechanical ventilation parameters. The output layer of each disease task is independently parameterized, and the risk level definition and clinical thresholds are set differently based on the corresponding disease's treatment guidelines. For example, sepsis uses a risk stratification standard related to the Rapid Sequential Organ Failure Score, while acute kidney injury is mapped to a level based on the staging standards of the Global Organization for Improved Kidney Disease Prognosis.

[0052] During training, multi-task learning employs a dynamic task weight scheduling strategy. Initially, all disease-specific tasks have equal weights. As training progresses, the weight coefficients are automatically adjusted based on the convergence speed and validation performance of each task. Tasks for rare diseases that are difficult to converge are assigned higher sampling probabilities and gradient weights to prevent dominant diseases from controlling the gradient update direction. The loss function is designed as a weighted sum of the policy loss and value loss for each disease, and a task-to-task representation alignment regularization term is introduced to encourage the unique representations output by different disease adapters to maintain a reasonable distribution in the shared semantic space, avoiding representation fragmentation caused by excessive expansion of task specificity.

[0053] For example, when a patient has clinical features of both sepsis and acute kidney injury, the multi-task architecture can output risk level assessments for both diseases in parallel and integrate them through a preset clinical rule engine to generate comprehensive monitoring priority recommendations. For example, when either disease is assessed as high risk, the highest level of clinical response process is triggered.

[0054] In summary, compared with existing technologies, this application effectively solves the problems of association confusion and static decision-making in traditional risk stratification methods by introducing a deep integration of causal inference mechanism and reinforcement learning framework.

[0055] In summary, the embodiments of this application have at least the following technical effects: This application provides a disease risk stratification method based on reinforcement learning. First, it acquires multimodal time-series data of the target patient during monitoring, achieving a comprehensive understanding of the patient's physiological state. Second, it extracts patient state representation vectors through a dynamic fusion mechanism, capturing the temporal dependencies and complementary information between different modalities, thus improving the accuracy and robustness of state representation. Third, it constructs a risk stratification model using a reinforcement learning framework, modeling risk stratification decisions as a sequential decision-making process, enabling the model to adaptively adjust the stratification strategy based on dynamic changes in the patient's state. Finally, it constructs reward signals using clinical intervention response data and feeds them back to optimize the model, forming a closed-loop learning mechanism of "decision-execution-evaluation-optimization," allowing the risk stratification strategy to continuously iterate and improve, enhancing the timeliness and accuracy of risk stratification.

[0056] Through the above technical solution, this application can integrate multi-source heterogeneous data in real time, dynamically assess the patient's risk status, and continuously optimize the decision-making strategy based on the clinical intervention effect. It effectively overcomes the shortcomings of traditional methods such as static solidification, delayed response, and lack of self-optimization ability, and improves the accuracy and clinical applicability of risk stratification for critically ill patients.

[0057] Example 2, as Figure 2 As shown, based on the same inventive concept as the reinforcement learning-based disease risk stratification method provided in Embodiment 1, this application also provides a reinforcement learning-based disease risk stratification system, including: Data acquisition module 11 is used to acquire multimodal time-series data of the target patient during the monitoring process; Data fusion module 12 is used to dynamically fuse the multimodal time series data and extract patient state representation vectors; The model training module 13 is used to input the patient state representation vector into a pre-trained risk stratification model, wherein the risk stratification model outputs a risk stratification action according to the current strategy, wherein the risk stratification action corresponds to classifying the patient into one of a set of multiple risk levels. The prompt generation module 14 is used to generate and push clinical monitoring prompts based on the risk stratification actions; The model update module 15 is used to acquire patient response data to the clinical intervention after a preset time window, calculate a reward signal based on the response data, and update the risk stratification model using the reward signal.

[0058] In one embodiment, the data fusion module 12 is specifically used for: The multimodal time series data is time-aligned and missing values ​​are filled to obtain structured time series data; The structured time-series data is input into a bidirectional long short-term memory network based on an attention mechanism to extract the time-dependent features within each modality; The patient state representation vector is generated by interactively fusing the temporal dependent features of each modality through a cross-modal attention fusion layer.

[0059] Furthermore, in one embodiment of the application, before inputting the patient state representation vector into the reinforcement learning risk stratification model, the method further includes: The patient state representation vector is input into the causal inference module, which, based on a pre-constructed disease causal graph, identifies key features that have a causal effect on risk stratification in the current state and generates a causal enhancement representation vector. The causal enhanced representation vector is used as the input to the reinforcement learning risk hierarchical model; The causal inference module employs counterfactual reasoning to quantify the causal effects of different clinical interventions on patient prognosis and integrates these causal effects into the generation process of the causal enhancement representation vector.

[0060] Furthermore, in one embodiment of the application, the risk stratification model adopts a policy gradient network architecture, whose state space is composed of the patient state representation vector, and whose action space is the preset multiple risk levels.

[0061] Furthermore, the method also includes: When the risk stratification action output by the reinforcement learning risk stratification model changes, a risk level change event is triggered, and the patient state representation vector, intervention measures and subsequent outcomes before and after the change are recorded to form an experience playback sample. The experience replay samples are stored in the experience pool for periodic offline training and updating of the reinforcement learning risk stratification model.

[0062] Furthermore, the multimodal time-series data includes at least vital sign monitoring data, laboratory test results, clinical intervention records, and electronic medical record text data, as well as at least one of continuous physiological signals, medical imaging features, or genomic data from wearable devices.

[0063] Furthermore, the method is applied to risk stratification of various acute and critical illnesses, including at least one of sepsis, acute kidney injury, acute respiratory distress syndrome, aspiration pneumonia, intracranial hypertension, deep vein thrombosis and pulmonary embolism, gastrointestinal bleeding, or malignant arrhythmia. The reinforcement learning risk stratification model shares the underlying feature extraction network through a multi-task learning architecture and outputs corresponding risk stratification actions for different diseases.

Claims

1. A disease risk stratification method based on reinforcement learning, characterized in that, The method includes: Acquire multimodal time-series data of the target patient during monitoring; The multimodal time-series data are dynamically fused to extract patient state representation vectors; The patient state representation vector is input into a pre-trained risk stratification model, which outputs a risk stratification action according to the current strategy, wherein the risk stratification action corresponds to classifying the patient into one of a set of multiple risk levels. Based on the aforementioned risk stratification actions, generate and push clinical monitoring prompts; After a preset time window, the patient's response data to the clinical intervention is acquired, a reward signal is calculated based on the response data, and the risk stratification model is updated using the reward signal.

2. The disease risk stratification method based on reinforcement learning according to claim 1, characterized in that, The multimodal time-series data is dynamically fused to extract a patient state representation vector, including: The multimodal time series data is time-aligned and missing values ​​are filled to obtain structured time series data; The structured time-series data is input into a bidirectional long short-term memory network based on an attention mechanism to extract the time-dependent features within each modality; The patient state representation vector is generated by interactively fusing the temporal dependent features of each modality through a cross-modal attention fusion layer.

3. The disease risk stratification method based on reinforcement learning according to claim 1, characterized in that, Before inputting the patient state representation vector into the reinforcement learning risk stratification model, the following steps are also included: The patient state representation vector is input into the causal inference module, which, based on a pre-constructed disease causal graph, identifies key features that have a causal effect on risk stratification in the current state and generates a causal enhancement representation vector. The causal enhanced representation vector is used as the input to the reinforcement learning risk hierarchical model; The causal inference module employs counterfactual reasoning to quantify the causal effects of different clinical interventions on patient prognosis and integrates these causal effects into the generation process of the causal enhancement representation vector.

4. The disease risk stratification method based on reinforcement learning according to claim 1, characterized in that, The risk stratification model adopts a policy gradient network architecture, whose state space is composed of the patient state representation vectors, and whose action space consists of the preset multiple risk levels.

5. The disease risk stratification method based on reinforcement learning according to claim 1, characterized in that, The method further includes: When the risk stratification action output by the reinforcement learning risk stratification model changes, a risk level change event is triggered, and the patient state representation vector, intervention measures and subsequent outcomes before and after the change are recorded to form an experience playback sample. The experience replay samples are stored in the experience pool for periodic offline training and updating of the reinforcement learning risk stratification model.

6. The disease risk stratification method based on reinforcement learning according to claim 1, characterized in that, The multimodal time-series data includes at least vital sign monitoring data, laboratory test results, clinical intervention records, and electronic medical record text data, as well as at least one of continuous physiological signals, medical imaging features, or genomic data from wearable devices.

7. The disease risk stratification method based on reinforcement learning according to claim 1, characterized in that, The method is applied to risk stratification of various acute and critical illnesses, including at least one of sepsis, acute kidney injury, acute respiratory distress syndrome, aspiration pneumonia, intracranial hypertension, deep vein thrombosis and pulmonary embolism, gastrointestinal bleeding, or malignant arrhythmia. The reinforcement learning risk stratification model shares the underlying feature extraction network through a multi-task learning architecture and outputs corresponding risk stratification actions for different diseases.

8. A disease risk stratification system based on reinforcement learning, characterized in that, For performing the reinforcement learning-based disease risk stratification method according to any one of claims 1-7, comprising: The data acquisition module is used to acquire multimodal time-series data of the target patient during the monitoring process; The data fusion module is used to dynamically fuse the multimodal time-series data and extract patient state representation vectors; The model training module is used to input the patient state representation vector into a pre-trained risk stratification model, which outputs a risk stratification action according to the current strategy, wherein the risk stratification action corresponds to classifying the patient into one of a set of multiple risk levels. The prompt generation module is used to generate and push clinical monitoring prompts based on the risk stratification actions; The model update module is used to acquire patient response data to the clinical intervention after a preset time window, calculate a reward signal based on the response data, and update the risk stratification model using the reward signal.

Images