A dynamic decision support system for reducing false alarms in non-equilibrium data

Abstract

The embodiment of the specification provides a dynamic decision support system for reducing false alarms of unbalanced data, and relates to the field of intelligent medical treatment. The system comprises a data acquisition module configured to acquire medical record data sets of a to-be-tested sample; an initial risk prediction module configured to generate an initial risk estimation value according to the medical record data sets; a stratified alarm system module configured to obtain an alarm prediction result that the to-be-tested sample belongs to one of a low-risk group, a medium-risk group and a high-risk group according to a comparison result of the initial risk estimation value and at least two first stratification thresholds; the first stratification thresholds are calculated based on precision-recall trade-off calibration and an optimization target of maximizing standardized net benefit; and a clinical decision module configured to output a clinical decision suitable for the to-be-tested sample according to the alarm prediction result.

Description

Technical Field

[0001] This invention relates to the field of intelligent healthcare, and more specifically, to a dynamic decision support system for reducing false alarms from unbalanced data. Background Technology

[0002] In clinical practice, accurately predicting rare but serious clinical outcomes (such as in-hospital death, sepsis, and readmission) is crucial for early intervention, optimized resource allocation, and improved patient prognosis. These positive events typically occur in extremely low rates (often below 5%), leading to severe class imbalance in clinical data. In such an extremely imbalanced scenario, even predictive models with high discriminative performance are prone to generating numerous false positive alerts, severely limiting their clinical usability. Therefore, accurately identifying high-risk individuals while effectively controlling the false positive rate has long been a core challenge in the field of clinical predictive models.

[0003] In recent years, with the widespread adoption of Electronic Health Records (EHRs) and the development of Artificial Intelligence (AI), numerous studies have attempted to utilize machine learning (ML) methods to model large-scale clinical data to predict in-hospital mortality risk. Studies have shown that these models exhibit excellent predictive performance, with receiver operating characteristic (AUROC) curves often exceeding 0.8, theoretically outperforming traditional manual scoring systems. However, in practical applications, the clinical value of these models is severely limited, primarily due to the following reasons: (1) Low incidence of events (data imbalance): The incidence of in-hospital mortality (IHM) is usually less than 5%. In this "extreme class imbalance" scenario, even if the model has strong discriminative ability, a large number of false positives will still be generated. This is because traditional machine learning algorithms tend to favor the majority class when dealing with imbalanced datasets, thus ignoring rare but clinically significant cases. To address this problem, previous studies have often used resampling (e.g., increasing minority class samples or decreasing majority class samples) or cost-sensitive learning. However, these strategies often fail to fully meet the fine requirements of risk prioritization in clinical decision-making and lack transparency. For example, Synthetic Minority Over-sampling Technique (SMOTE) improves class balance by interpolating between minority class samples to generate new synthetic samples, but it may overestimate the probability of minority class events, leading to distorted risk predictions after calibration, thus affecting clinical confidence.

[0004] (2) High false alarm rate (low PPV): Even if the model performs well in prediction, the positive predictive value (PPV) may still be low due to the extremely low event incidence, resulting in a large number of false alarms. In other words, data imbalance not only affects the model's identification of minority classes, but also directly reduces the reliability of clinical alerts. For example, the 7-day mortality prediction model constructed by Hsieh et al. (BMJ Open, 2021) has an area under the receiver operating characteristic curve (AUROC) as high as 0.81, but the PPV is only 10.55%, which means that nearly 90% of the alerts may be false alarms. This low PPV leads to frequent triggering of alerts by low-risk patients, exacerbating clinical vigilance fatigue. Even if the model has strong predictive performance, it is difficult to fully meet the operational needs of clinical decision-making, thereby reducing the clinical trust and adherence to the use of early warning tools.

[0005] (3) Lack of refined stratification and clinically actionable insights: Most existing models typically use a single threshold to simply divide patients into "high-risk / non-high-risk," ignoring the continuity and complexity of risk stratification. This "coarse-grained" stratification limits the model's clinical application value, as emergency triage and treatment decisions often require multi-level risk assessments to prioritize the most critically ill patients and allocate resources rationally. Furthermore, the lack of clear clinically actionable insights makes it difficult for medical staff to understand the specific logic behind risk scoring and corresponding intervention strategies, thereby reducing decision-making confidence.

[0006] Therefore, how to significantly improve PPV, alleviate alarm fatigue, and achieve cross-hospital applicability while ensuring sensitivity has become a key technical challenge that urgently needs to be solved. Summary of the Invention

[0007] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention provides a dynamic decision support system for reducing false alarms caused by imbalanced data. This system addresses core issues in clinical prediction models, such as prediction distortion due to class imbalance, high false alarm rates, and a lack of refined decision support. It proposes a system solution with the optimization objective of maximizing the Standardized Net Benefit (SNB). Unlike traditional simple stratification methods based on a single threshold, this solution achieves precise alignment between risk stratification and clinical utility through multi-threshold optimization and deep integration with decision theory.

[0008] To address the issues of prediction bias and rampant false positives caused by data imbalance, existing techniques such as resampling or cost-sensitive learning are insufficient to meet the clinical need for fine-grained risk stratification. This invention constructs a multi-threshold stratification mechanism, dividing continuous risk levels based on real clinical treatment preferences, significantly improving the identification accuracy of minority classes (high-risk events) and reducing misjudgments caused by extreme imbalance. Simultaneously, by incorporating a knowledge-enhanced filtering strategy, it reduces reliance on synthetic data, enhancing model calibration capabilities and the interpretability and reliability of prediction results.

[0009] This application discloses a dynamic decision support system for reducing false alarms from imbalanced data, comprising: The data acquisition module is configured to acquire the medical record dataset of the sample to be tested; The initial risk prediction module is configured to generate an initial risk estimate based on the medical record dataset. The stratified alert system module is configured to obtain an alert prediction result that the test sample belongs to one of the low-risk group, medium-risk group, and high-risk group based on the comparison results of the initial risk estimate and at least two first stratification thresholds; the first stratification thresholds are calculated based on precision-recall tradeoff calibration with the optimization objective of maximizing standardized net benefit; The clinical decision module is configured to output a clinical decision suitable for the test sample based on the alarm prediction results.

[0010] This application has the following beneficial effects: The core of this application lies in constructing a multi-layered, sequential intelligent decision support framework—AI-powered Tiered Early Warning (AI-TEW). This framework is not a single predictive model, but an integrated system adaptable to various clinical scenarios. It employs a sequential processing pipeline that integrates statistical prediction, hierarchical decision-making, interpretability analysis, and knowledge-enhanced filtering. Its design goal is to fundamentally solve the common problem of low positive predictive value (PPV) and high false alarm rate caused by highly imbalanced clinical data categories, thereby comprehensively improving the clinical utility and operability of predictive models in the real world; specifically, First, existing technologies such as resampling or cost-sensitive learning are insufficient to meet the clinical need for fine-grained risk stratification, addressing the prediction bias and false positives caused by data imbalance. This invention constructs a multi-threshold stratification mechanism, dividing continuous risk levels based on real clinical treatment preferences, significantly improving the identification accuracy of minority classes (high-risk events) and reducing misjudgments caused by extreme imbalance. Simultaneously, it incorporates a knowledge-enhanced filtering strategy to reduce reliance on synthetic data, enhancing model calibration capabilities and the interpretability and reliability of prediction results.

[0011] Secondly, to address the persistent issues of low positive predictive value (PPV) and false alarms even under high AUROC conditions, this invention introduces a threshold optimization framework based on decision curve analysis (DCA) and standardized net benefit (SNB). This framework comprehensively balances sensitivity, specificity, PPV, and NPV, dynamically determining the optimal threshold range with the goal of maximizing SNB. This significantly reduces the number of false alarms while maintaining high sensitivity, thereby effectively alleviating alarm fatigue among healthcare workers and improving system reliability and clinical usability.

[0012] Furthermore, addressing the shortcomings of existing models in lacking hierarchical decision support and actionable insights, this invention proposes a multi-level Tiered Early Warning (TEW) structure. This structure divides risk probabilities into multiple continuous intervals and matches each interval with differentiated treatment recommendations (such as routine observation, enhanced monitoring, and emergency intervention). In addition, by integrating SHAP interpretability technology and a large language model (LLM), it generates individualized interpretations and treatment guidelines that fit the clinical context, thereby enhancing physicians' trust in and compliance with the system's output and promoting the effective integration of early warning information with the treatment process.

[0013] The system disclosed in this application can extract data from electronic medical records (EHRs) in real time, automatically execute the entire process from prediction, stratification, interpretation to filtering, and finally push tiered alerts and interpretive results to clinical terminals, forming an end-to-end clinical decision support solution. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the dynamic decision support system provided in an embodiment of the present invention; Figure 2 This is the research design and AI-TEW (AI-based Early Warning) framework provided in the embodiments of the present invention. Figure 2 A shows the patient selection process in three multicenter cohorts, generating internal (H1 / H2 / H3) and external (H2Ext) validation sets, where the internal validation set uses a five-fold cross-validation method for model training and testing. Figure 2B outlines the two-stage design of the AI-TEW system. In the first stage, an artificial intelligence model is developed to estimate mortality risk, illustrating how class imbalance in rare event scenarios leads to high false alert rates and alert fatigue in current systems. The second stage combines machine learning-based risk stratification (multi-tier decision-making) with large language model-driven interpretability and alert filtering to reduce false positives and enhance clinical decision support in the ED workflow; Figure 3 shows the internal and external validation of the machine learning-based in-hospital mortality prediction model provided in the embodiments of the present invention. Figure 3 A uses a 5-fold cross-validation method to compare the performance of seven different machine learning models in three internal validation cohorts (H1, H2, and H3). Figure 3 B represents the performance of the LightGBM model with the best predictive power in the internal cross-validation and external temporal validation cohorts. Evaluation metrics include the Area Under the Receiver Operating Characteristic Curve (AUROC), the Area Under the Precision-Recall Curve (AUPRC), and Decision Curve Analysis (DCA); Figure 4 illustrates the exploration of the impact of the downsampling rate (sample size of positive predictions and negative predictions) on the training and testing phases provided in the embodiments of the present invention. Subplots a1 (H1), b1 (H2), and c1 (H3) represent validation during training using datasets with different downsampling rates, revealing that low PPV is associated with data imbalance (e.g., rare events such as low IHM). Subplots a2 (H1), b2 (H2), and c2 (H3) represent testing on full-scale (non-downsampled) data, demonstrating the low PPV challenge of the model in real-world applications; Figure 5 shows the optimization and clinical benefit analysis of risk stratification thresholds (k1 and k2) provided in the embodiments of the present invention. Figure 5 A shows the standardized net benefit derived from the three-tier risk stratification strategy (low (Prisk ≤ k1), medium (k1 < Prisk ≤ k2), and high risk (Prisk > k2)), compared with the treat-all and treat-none strategies, using the principles of decision curve analysis. The yellow region (a1) represents the threshold combination that yields the maximum net benefit, while the red region (a2) represents the clinically acceptable range where the net benefit remains positive, supporting robust model deployment under uncertainty. Figure 5 B shows the corresponding Negative Predictive Value (NPV) of the low-risk group (b1) and the Positive Predictive Value (PPV) of the high-risk group (b2) under different k1 and k2 values. This trade-off visualization enables the calibration of the AI-driven Tiered Early Warning (AI-TEW) system to simultaneously improve patient safety (via high NPV) and reduce alert burden (via high PPV), addressing the key limitations of traditional single-threshold models in emergency care; Figure 6This invention provides a threshold selection for risk stratification and an evaluation of the effectiveness of stratification. Figure 6 A shows the threshold divisions (dashed lines) for traditional binary classification (a1), risk stratification (a2), and multi-level decision-making methods (a3). Figure 6 B (H2 internal validation) and Figure 6 C (H2Ext external validation) shows the PPV changes (the proportion of IHM in each risk category) for traditional binary classification (b1 and c1), risk stratification (b2 and c2), and multilevel decision-making approaches (b3 and c3). Figure 7 This invention relates to risk alert filtering based on a large language model (LLMA) and its impact on reducing false alarms and clinical performance. Figure 7 A shows the proportional distribution of responses from the large language model to AI-generated high-risk alerts: "Yes" (continue alert), "Uncertain" (suggest clinical review), and "No" (suppress alert, as it may be a false positive). This demonstrates the role of the large language model in contextual interpretation and alert classification. Figure 7 B compared the positive predictive value (PPV) and sensitivity of different large language models. The red dashed line represents the baseline PPV after risk stratification but before large language model filtering; the red bars show the PPV after large language model integration, highlighting the significant improvement in alert accuracy. The purple line represents the sensitivity in the high-risk group; all models maintained ≥90% sensitivity, ensuring no critical cases were missed. These results illustrate how large language models can enhance the clinical utility of AI-based early warning systems by reducing false alarms while protecting patient safety. Figure 8 This is the distribution of reasons for the "uncertain" response in the large language model alarm filter layer of the external verification queue (H2Ext) provided in this embodiment of the invention. Figure 8 A shows the reasons for uncertainty, the most common being (a) missing key clinical features (66.7%); (b) conflicting clinical features (26.2%); and (c) insufficient clinical context (7.1%). Figure 8 B illustrates the different classification results (yes / no / uncertain) of different large language models for representative cases, highlighting model-specific reasoning patterns. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0017] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Definitions: 1. In-hospital Mortality (IHM): Deaths occurring during the same hospital stay, used as the primary outcome measure.

[0020] 2. Artificial Intelligence-Powered Tiered Early Warning (AI-TEW): A risk warning strategy that classifies patients into different risk levels (such as high, medium, and low) by setting multiple risk probability thresholds, and configures differentiated clinical monitoring and intervention strategies for each level, aiming to optimize resource allocation and reduce false positive alarms.

[0021] 3. Large Language Model (LLM) Filtering and Summary: LLM is used to perform knowledge-enhanced filtering on model-generated alerts, eliminating possible false alarms, and providing clinical interpretations and treatment suggestions through natural language generation, thereby improving the consistency between alerts and actual actions.

[0022] 4. Positive Predictive Value (PPV): Among all cases predicted as positive (high-risk) by the model, the proportion of actual positive events (such as death) occurs. The formula is: PPV = TP / (TP + FP), where TP is the number of true positives and FP is the number of false positives. This indicator directly reflects the reliability of the alert.

[0023] 5. Negative Predictive Value (NPV): The proportion of cases predicted as negative (low-risk) by the model that did not actually have a positive event (such as death). The formula is: NPV = TN / (TN + FN), where TN is the number of true negatives and FN is the number of false negatives. This indicator measures the model's ability to reliably exclude non-critical patients. A high NPV means that the low-risk prediction results are highly reliable and can support safe triage or reduce unnecessary interventions in clinical decision-making.

[0024] 6. Sensitivity / Recall: The proportion of cases that correctly predict a positive outcome among all actual positive events. It is calculated as: Sensitivity = TP / (TP + FN), where FN is the number of false negatives. This metric measures the model's ability to identify truly critically ill patients.

[0025] 7. Specificity: Among all cases where no positive event (such as death) actually occurred, the proportion correctly predicted as negative (low risk) by the model. The formula is: Specificity = TN / (TN + FP), where TN is the number of true negatives and FP is the number of false positives. This indicator measures the model's ability to accurately identify non-critical patients.

[0026] 8. Area Under the Receiver Operating Characteristic Curve (ROC-AUC): Used to evaluate the model's overall ability to distinguish between positive and negative samples at different decision thresholds. The closer the value is to 1, the better the discrimination performance.

[0027] 9. Area Under the Precision-Recall Curve (PR-AUC): More suitable for evaluating imbalanced datasets, it considers both precision and recall, and better reflects the model's performance when positive samples are scarce.

[0028] 10. Decision Curve Analysis (DCA): Using net benefit (NB) as a metric, this method evaluates the clinical value of the model at different risk thresholds.

[0029] 11. Standardized Net Benefit (SNB): The net benefit (NB) of the model at different probability thresholds is calculated and compared with two extreme strategies: "intervening in all patients" and "not intervening in any patients". This quantifies the clinical benefit of the model compared to the extreme strategies.

[0030] 12. Explanation (SHapley Additive exPlanations, SHAP): Shapley values ​​are used to quantitatively assess the contribution of each feature to a single prediction result or the overall model output, thereby providing an intuitive and interpretable basis for individual case prediction and global model behavior.

[0031] Figure 1 This is a schematic diagram of a dynamic decision support system for reducing false alarms from imbalanced data, provided by an embodiment of the present invention. Specifically, the system includes: The data acquisition module is configured to acquire the medical record dataset of the sample to be tested; The initial risk prediction module is configured to generate an initial risk estimate based on the medical record dataset. The stratified alert system module is configured to obtain an alert prediction result that the test sample belongs to one of the low-risk group, medium-risk group, and high-risk group based on the comparison results of the initial risk estimate and at least two first stratification thresholds; the first stratification thresholds are calculated based on precision-recall tradeoff calibration with the optimization objective of maximizing standardized net benefit; The clinical decision module is configured to output a clinical decision suitable for the test sample based on the alarm prediction results.

[0032] In some more specific embodiments, the stratified alarm system module is further configured to obtain an alarm prediction result for the test sample belonging to one of the medium-risk (M-), medium-risk (M), and medium-high-risk (M+) subgroups based on the comparison results of the initial risk estimate and at least two second stratification thresholds, wherein the value range of the second stratification threshold is between the two first stratification thresholds.

[0033] In some embodiments, the system further includes: a personalized risk interpretation module, configured to obtain medical record set data representing the risk group to which the test sample belongs based on the alarm prediction result, calculate the contribution of each feature in the medical record set, and filter key risk factors for each test sample based on the contribution.

[0034] In some embodiments, the system further includes: an intelligent filtering module configured to evaluate the accuracy of the alarm prediction results using a large language model and output a classification decision including any one of yes, no, and uncertain, the classification decision also including natural language reasoning; the large language model processes the structured input of the medical record dataset through a cueing framework to identify and filter false alarms, thereby significantly improving the PPV of the final alarm stream.

[0035] In some more specific embodiments, the clinical decision suitable for the sample to be tested includes: if it is a low-risk group, outputting a discharge or minimal monitoring plan recommendation; if it is a medium-risk group, outputting a continuous observation plan recommendation; and if it is a high-risk group, outputting an immediate intervention plan recommendation.

[0036] In some more specific embodiments, the clinical decision suitable for the sample to be tested includes: if it is a medium- to high-risk subgroup, outputting a recommended plan for timely assessment; if it is a medium- to low-risk subgroup, outputting a recommended plan for routine monitoring.

[0037] In some embodiments, the standardized net benefit is obtained by calculating the difference between the total net benefit of the alarm prediction results and the maximum of the utility values ​​of the two extreme strategies, wherein the total net benefit of the alarm prediction results is calculated by a utility function, and the two extreme strategies are all treatment and no treatment.

[0038] In some embodiments, the utility function is:

[0039] Where yi represents the true outcome of the sample, pi represents the predicted in-hospital mortality rate, and k1 and k2 are two risk thresholds. This indicates the correct identification and timely intervention of high-risk patients. and The maximum benefit brought about by ) This indicates the cost of false positives in the high-risk group, reflecting the resource and risk burden caused by unnecessary intervention; The cost of false negatives in the low-risk group reflects the serious harm caused by premature discharge followed by patient deterioration; for cases in the medium-risk group, delayed identification yields only partial benefits, denoted as ,in This is used to characterize the negative impact of delayed treatment on the condition; while the cost of over-observation in the medium-risk group is denoted as... ,in This is used to reflect that the resources required for observation are lower than those required for full intervention.

[0040] In some more specific embodiments, the initial risk estimate is obtained based on clinical experience. Alternatively, it can be predicted based on an in-hospital mortality prediction model. The method for constructing an in-hospital mortality prediction model includes: obtaining a medical record dataset of training samples; iteratively training a machine learning model based on the training samples of the medical record dataset; and using the trained model as the in-hospital mortality prediction model when the training stops.

[0041] In some embodiments, the data acquisition module interfaces with the hospital's electronic medical record system.

[0042] In some embodiments, the predicted results and recommended solutions include, but are not limited to, paper or electronic reports. These results are derived by the intelligent machine from the analysis of the subject's relevant data and are intended only as a reference for medical personnel, not as the subject's final diagnostic result.

[0043] In some embodiments, the stratification threshold is obtained through training on training set samples. It can be a specific threshold or an interval range. The specific form is not specifically limited in this embodiment.

[0044] In this embodiment, to more clearly describe the development process and results of the AI-driven hierarchical early warning framework aimed at solving the high false alarm rate problem in in-hospital mortality prediction, an AI-driven hierarchical early warning system (AI-TEW) is specifically constructed as an example for illustrative description. However, this description is only one use case and is not intended to limit the scope of protection of this data processing method. The specific illustrative description is as follows: Specific methods Research population and design This multicenter retrospective cohort study included data from 1,388,586 emergency department visits across three institutions, denoted as H1, H2, and H3. The study population was limited to adult patients (≥18 years of age) transferred from the emergency department to inpatient wards. Exclusion criteria included: age <18 years; lack of necessary emergency department records (e.g., diagnostic information); lack of transfer data; and illogical chronological order (e.g., discharge date earlier than admission date). A total of 174,292 emergency department visits were ultimately included after applying these criteria. The primary objective was to develop and externally validate a machine learning-based multilevel risk stratification model for predicting in-hospital mortality (IHM) in emergency department patients. Model development and internal validation used data from H1, H2 (before November 2022), and H3. Internal validation was supplemented with five-fold cross-validation to assess model stability. Temporal external validation was performed on a cohort from Hospital H2 (H2, n=1723 visits (November 2022 to April 2023), which was different from the model training period in time, to assess generalizability in real clinical practice (see Figure 6A).

[0045] Data collection and preprocessing Patient data were obtained from the electronic medical record system and included demographic information (age, sex, mode of visit: unknown, self-visit, ambulance, or other), triage grade score (Level 1: resuscitation, Level 2: emergency, Level 3: urgent, Level 4: non-urgent), vital signs (systolic and diastolic blood pressure, respiratory rate, heart rate, body temperature, and peripheral oxygen saturation), laboratory test results (e.g., creatinine, glucose, hematocrit, hemoglobin, alkaline phosphatase, anion gap, and bicarbonate), and timestamps of key clinical events for the diagnosis of clinical disease as defined in the International Classification of Diseases, Ninth and Tenth Revisions (ICD-9 and ICD-10) (e.g., triage time, admission time, and discharge time), as well as the severity of the patient's condition and outcome (inpatient ward: ICU or general ward; patient discharge outcome: survival or in-hospital death).

[0046] To ensure data quality and analytical consistency, standardized preprocessing procedures were implemented. For categorical data, variables with missing values ​​were assigned a separate "unknown" category to preserve data integrity. For continuous variables, missing data was preserved during modeling using tree-based ensemble methods, which themselves handle missing values ​​through mechanisms such as surrogate segmentation or gradient-aware optimization. For models requiring complete data (e.g., logistic regression), missing values ​​were imputed using the median (for skewed distributions) or the mean (for approximately normal distributions). Outliers in continuous variables (e.g., vital signs and laboratory data) were identified using interquartile range-based methods and further evaluated against clinically reasonable thresholds derived from prior knowledge, replacing extreme and biologically unreasonable values ​​with the closest clinically acceptable boundary values. Categorical variables (e.g., mode of arrival) were coded once for ease of numerical processing. Temporal variables were standardized to calculate clinically relevant time intervals (e.g., time from triage to admission). ICD-9 codes were systematically mapped to ICD-10 to ensure diagnostic consistency across datasets. Before analysis, records containing critical missing data (e.g., missing triage timestamps) or logical inconsistencies (e.g., discharge before admission) were excluded.

[0047] AI-TEW Framework Development The Artificial Intelligence-Driven Hierarchical Early Warning (AI-TEW) system is a structured decision support framework designed to enhance the clinical applicability of IHM prediction in emergency care. AI-TEW does not rely solely on raw risk scores but employs a sequential process to improve accuracy and interpretability. First, a machine learning model generates risk assessments using routinely collected clinical data. Next, SHAP-based interpretations provide personalized feature-level attributions to support transparent decision-making. Then, a hierarchical alert mechanism applies risk stratification thresholds to prioritize high-concern cases and reduce low-value notifications. Importantly, this application integrates LLM (Logical Logical Modeling) to generate natural language interpretations and, as a knowledge-based filter, assesses the clinical rationale of alerts, further reducing false alarms. This hierarchical approach ensures that alerts are not only statistically robust but also clinically relevant. The following sections detail the various components of the AI-TEW framework.

[0048] A. Machine learning was used for initial risk prediction. For the initial risk prediction component of the AI-TEW framework, this application evaluated various machine learning algorithms to determine the optimal one. The models included CatBoost, Light GBM, XGBoost, Decision Tree Classifier (DTC), Random Forest (RF), K-Nearest Neighbors (KNN), and Logistic Regression (LR). Model selection was guided by the performance of key metrics (especially AUROC, sensitivity, and PPV), with a focus on detecting high-risk cases. Hyperparameter tuning employed Bayesian optimization methods. To ensure model robustness and minimize variability caused by data partitioning, five-fold cross-validation was used for internal validation. The final model was selected based on excellent generalization ability, stability across validation folds, and computational efficiency, thereby ensuring reliable risk estimation for downstream hierarchical alerting and interpretation components.

[0049] B. Personalized Risk Interpretation Based on SHAP. To enhance model interpretability and support clinical transparency, SHAP is used to quantify the contribution of each input feature to individual risk prediction. By computing locally accurate game-theoretic attributions, SHAP enables personalized interpretation of the model's output, highlighting which clinical variables (e.g., elevated lactate, decreased systolic blood pressure) have the greatest driving effect on high-risk alerts for specific patients. This ex post-hoc interpretability approach provides both global model insights and instance-level explanations, thereby facilitating trust building and informed clinical review.

[0050] C. A hierarchical alarm system for mitigating alarm fatigue. To address the class imbalance problem and reduce the false alarm rate, this application implements a resampling strategy during model development, systematically altering the ratio of positive to negative samples ( This study aims to assess the impact on predictive performance, enabling the optimization of model calibration under real-world prevalence conditions. For clinical deployment, this application designs a stratified alert system to support risk stratification decisions and mitigate alert fatigue. Three risk categories are defined using two clinically interpretable thresholds: a low / intermediate risk threshold (k1) and an intermediate / high risk threshold (k2), resulting in: a low-risk group (P... risk ≤k1), this threshold is characterized by high net predictive value (NPV), supporting safe and rapid discharge or observation; intermediate risk group (k1 < P risk ≤k2), representing uncertain cases requiring further evaluation; and the high-risk group (P risk Thresholds >k2, with higher positive predictive value (PPV), are prioritized for intervention. Thresholds k1 and k2 are calibrated using a precision-recall tradeoff to maximize clinical usability—ensuring high sensitivity for critical cases while minimizing unnecessary alarms in low-risk patients. To enhance the granularity of intermediate-risk management, an optional refinement scheme can subdivide the group into low-intermediate, intermediate, and high-intermediate-risk subgroups, thereby achieving tiered surveillance intensity and improving the positive predictive value (PPV) of higher subgroups.

[0051] a) Utility Function Definition. To assess the net clinical benefit of the multilevel risk stratification framework, the standard decision curve analysis (DCA) was extended to include three distinct clinical factors. Pathways: immediate intervention for high-risk patients, continued observation for intermediate-risk patients, and discharge or minimal monitoring for low-risk patients. The utility function is numerically assigned based on the clinical consequences of correct and incorrect classification, where B1=1 represents the maximum benefit of correctly identifying and intervening in high-risk cases (yi=1 and pi>k2). The cost of false positives in the high-risk group is defined as C1=k2 / (1-k2), reflecting the burden of unnecessary intervention; while the cost of false negatives in the low-risk group is L1=(1-k1) / k1, representing the harm caused by premature discharge leading to disease deterioration. For intermediate-risk cases, the benefit of delayed identification is reduced, B2=αB1, where α∈(0,1) considers the time sensitivity of the condition; the lower cost of over-observation is C2=βC1, where β∈(0,1) reflects less resource utilization than full intervention. Therefore, the complete utility function i for each patient, with the true outcome yi and the predicted risk pi, is defined as follows:

[0052] b) Calculation of total net benefit, total net benefit of the multi-level model ( The following can be calculated: Where N is the total number of patients.

[0053] c) Payoff calculation for extreme strategies: The multi-level model is compared with two extreme strategies: "all treated" and "no treated". Under the "all treated" strategy, all samples are classified as high-risk (i.e., requiring intervention). =1, then it is a true instance (TP), generating Benefits; if =0, then it is a false positive (FP), causing Loss. Therefore: Under the "no treatment" strategy, all patients are classified as low-risk (i.e., no treatment is given). =1, then it is a false negative (FN), producing Loss; if If the value is 0, then it is a true negative example (TN), and its utility value is 0. Therefore: ; d) Standardized Net Benefit (SNB): SNB quantifies the clinical utility of the multilevel model relative to these extreme strategies. A positive SNB indicates that our model provides higher clinical value than both extreme strategies. The calculation formula is as follows: .

[0054] Multi-layered models are used for knowledge-based filtering and interpretability. To improve the specificity of high-risk alerts, this application employs multiple multi-layered models, including open-source models (MedGemma, DeepSeek-V3, Baichuan-M1, LLaMA-3.3, Qwen-3) and closed-source GPT-4.1—which integrates clinical language understanding capabilities. Each model processes structured input containing patient demographics, vital signs, laboratory results, and clinical history through a consistent prompting framework and is asked to answer a structured question: “Based on your clinical knowledge, please assess the risk alert level of the model: if you agree with the alert level, answer ‘Yes’; if you think the alert is inaccurate, answer ‘No’; if you are unsure, answer ‘Unsure’.” The model generates responses in a unified pattern, providing classification decisions (“Yes,” “No,” or “Unsure”) along with concise natural language reasoning. This dialogue-based approach leverages the ability of LLMs to integrate multi-source clinical data and medical reasoning within the context of application, effectively acting as a knowledge-driven filter to differentiate between clinically meaningful alerts and false alarms. Crucially, the generated explanations provide transparent and easily understandable reasons for each decision, thereby reducing the cognitive burden on clinicians to manually validate decisions and supporting more efficient and trustworthy alert management.

[0055] result like Figure 2The diagram shows the research design of this application, including sample selection ( Figure 2 A) and the AI-TEW framework for enhancing the clinical usability of predictive models ( Figure 2 B).

[0056] Performance of machine learning models The performance of seven different machine learning algorithms in predicting in-hospital mortality rates among emergency room patients was evaluated. Figure 3 A compared the performance of these models across three internal cohorts using a five-fold cross-validation method (H1, H2, and H3). Among all methods, LightGBM consistently achieved the highest AUROC: 0.84 (95% CI, 0.81–0.86) in H1, 0.87 (95% CI, 0.85–0.90) in H2, and 0.91 (95% CI, 0.90–0.91) in H3. CatBoost and XGBoost also demonstrated strong performance. XGBoost achieved an AUROC of 0.81 (95% CI, 0.78–0.84) in H1, 0.87 (95% CI, 0.85–0.89) in H2, and 0.90 (95% CI, 0.90–0.91) in H3. CatBoost achieved AUROCs of 0.81 (95% CI, 0.79–0.84), 0.87 (95% CI, 0.84–0.89), and 0.89 (95% CI, 0.89–0.90) in H1, H2, and H3, respectively. LightGBM also exhibited the highest area under the precision-recall curve (AUPRC) values: 0.24 (95% CI, 0.19–0.29) for H1, 0.33 (95% CI, 0.27–0.39) for H2, and 0.31 (95% CI, 0.29–0.32) for H3. These results demonstrate that LightGBM possesses excellent discriminative performance in identifying IHM cases.

[0057] Due to its superior performance, LightGBM was chosen for out-of-time validation (i.e., validation on independent datasets in subsequent periods), such as Figure 3 As shown in B. In the external cohort, the model maintained robust discriminative power, with an AUROC of 0.87 (95% CI, 0.83–0.90) and an AUPRC of 0.40 (95% CI, 0.31–0.50), which is highly consistent with the internal validation results. Furthermore, as... Figure 3 As shown in B(b3), decision curve analysis (DCA) demonstrates that the LightGBM model provides clinically meaningful net benefits across a wide range of threshold probabilities (0.1–0.4), significantly outperforming both the "all-treatment" and "no-treatment" strategies. This indicates that the model has significant clinical application potential in guiding risk-based decision-making.

[0058] The impact of class imbalance on prediction performance Key clinical assessment metrics—sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV)—were calculated for the final LightGBM model at different decision probability thresholds. The model was trained using a downsampled dataset with a positive-to-negative ratio of 1:3 to mitigate class imbalance during learning; however, its evaluation was conducted on a real-world, highly imbalanced test dataset reflecting the natural occurrence of events: H1 = 1:26, H2 = 1:29, and H3 = 1:39. At decision thresholds determined by the Youden index (H1 = 0.200, H2 = 0.175, H3 = 0.225), the model achieved high negative predictive values ​​(NPV) (H1 = 98.7%, H2 = 99.1%, H3 = 99.5%), demonstrating its strong ability to identify low-risk patients and support safe clinical downgrading. However, the positive predictive value (PPV) remained low (13.9% for H1, 11.0% for H2, and 9.8% for H3), indicating a high false positive rate under actual deployment conditions.

[0059] Figure 4 This limitation is explained by demonstrating how class imbalance in the training data affects the positive predictive value (PPV). The training PPV decreases significantly as the proportion of positive cases (i.e., in-hospital deaths) decreases. For example, in H1, the original training imbalance ratio was 1:26 (1 death for every 27 patients), resulting in a training PPV of only 10.1%. When the negative samples were randomly downsampled to achieve a more balanced ratio of 1:3, the training PPV significantly improved to 49.3%. Notably, while the test PPV remained low and essentially unchanged across different training ratios when evaluated on the original imbalanced test set (e.g., 1:26 in H1), AUROC exhibited robust stability and remained at a high level (e.g., >0.84), highlighting the model's consistent discriminative performance even under severe class imbalance. A consistent pattern was observed in all three hospitals (H1, H2, and H3), indicating that while downsampling improves apparent performance during training, it does not translate into a significant increase in positive predictive value (PPV) in real-world scenarios where event incidence remains low. This highlights the need to adopt complementary strategies, such as risk stratification, to reduce false alarms and improve clinical usability.

[0060] Effectiveness of tiered early warning methods Figure 5A illustrates the standardized net benefit (relative to the "all-treatment" and "no-treatment" strategies) under different combinations of risk stratification thresholds: low / intermediate risk threshold (k1) and intermediate / high risk threshold (k2). The clinical utility of the model (measured by net benefit) changes accordingly as these thresholds are adjusted. In H2, the standardized net benefit is greater than zero when (k1,k2) falls within the region bounded by coordinates (0.01,0.20), (0.19,0.20), (0.19,0.80), (0.16,0.84), (0.15,0.85), and (0.01,0.97), indicating that the stratification method adds value compared to the default strategy.

[0061] Figure 5 B illustrates the trade-offs between positive predictive value (PPV), negative predictive value (NPV), and patient coverage as the stratification threshold changes. As the threshold k1 increases, the low-risk group expands and includes more patients, but its net present value gradually decreases. Conversely, as k2 increases, the high-risk group becomes more selective: the number of patients classified as high-risk decreases, but the PPV for this group significantly increases. For example, in H3, setting a lower k2=0.22 resulted in a high-risk PPV of 9.8% and captured 21.28% of all patients. In contrast, increasing k2 to 0.79 increased the positive predictive value (PPV) to 35.7%, but at the cost of reduced coverage (2.15% of patients). Careful calibration of k1 and k2 allows for simultaneous NPV >99.0% for the low-risk group, PPV >40.0% for the high-risk group, and a positive standardized net benefit—demonstrating the practicality and clinical applicability of this framework.

[0062] Figure 6 The performance of three risk stratification schemes was compared under optimized decision thresholds: a traditional two-level model (high risk vs. low risk), a three-level model (high risk / medium risk / low risk), and an improved multi-level decision-making method that further subdivides the medium-risk group to enable tiered clinical intervention. The results were validated in an internal validation cohort (H2). Figure 6In B), the transition from a two-tier to a three-tier model increased the positive predictive value (PPV) of the high-risk group from 11.03% to 40.49%, significantly improving case enrichment and reducing the number of false positives in high-risk patients. This allows for more effective intensive surveillance and early intervention. Patients classified as intermediate-risk exhibited considerable heterogeneity and may require additional clinical evaluation or closer observation. To support more granular decision markers and reduce physician workload, this group was further subdivided into three subgroups: low-intermediate risk (M-), intermediate risk (M), and high-intermediate risk (M+). Notably, the mortality rate in the M+ subgroup (20.14%) was significantly higher than that in the M group (7.64%) and the M- group (3.22%), indicating a meaningful risk stratification in the intermediate region. This hierarchical distinction allows for a phased clinical response—from routine surveillance in the M- group to timely assessment and escalation of treatment in the M+ group—thereby aligning predictive outcomes with actionable treatment pathways.

[0063] like Figure 6 As shown in Figure C, in the external validation cohort, the positive predictive value (PPV) for the high-risk group also increased from 18.81% in the two-level scheme to 37.17% after applying the three-level stratification, indicating a consistent performance improvement across different independent populations. These findings suggest that shifting from a binary risk prediction system to a multi-level early warning system can improve the accuracy and clinical applicability of in-hospital mortality prediction in emergency settings.

[0064] LLM-based alarm filtering improves accuracy Figure 7 The performance of six LLMs in filtering high-risk false alarms generated by multilevel decision models (MLMs) from both internal and external validation cohorts was evaluated. The task of the LLMs was to review patient-level risk alerts and categorize them as "yes" (confirmation alert), "no" (rejection as a false alarm), or "uncertain" (requiring further clinical evaluation), thus serving as a secondary validation layer to improve alert accuracy. These models showed significant differences in alert bias. In the external validation cohort, DeepSeek-V3 responded "yes" in 78.8% of cases, "no" in 0.9%, and "uncertain" in 20.3%, indicating a conservative alerting strategy. In contrast, GPT-4.1 showed a higher confirmation bias, responding "yes" in 90.3% of cases, "no" in 3.5%, and "uncertain" in 6.2%, reflecting a more aggressive confirmation preference.

[0065] To describe the properties of the uncertain output of the machine learning model (LLM) based filter, this application categorized all cases marked as “uncertain” during the validation process (see [link to application]). Figure 8 A). In the external validation set (H2 extension), the most common cause of uncertainty was missing key clinical features (66.7%), followed by contradictory clinical features (26.2%) and insufficient contextual information (7.1%). These categories consistently reflect situations where LLM identifies incomplete or contradictory clinical data (e.g., missing laboratory results, contradictory vital signs) or deems existing evidence insufficient to support the high-risk predictions of the machine learning model. The explanatory narrative of LLM aligns well with the conservative, information-seeking decision-making approach employed by clinicians in situations of genuine diagnostic uncertainty, suggesting that the model's "uncertainty" constitutes a reasonable, rather than arbitrary, output. Figure 8 As shown in B, under the same clinical information, different LLMs may emphasize different risk factors based on their underlying clinical knowledge, thus arriving at different classification results for the same patient. For example, GPT-4.1 emphasizes the lack of key laboratory indicators (such as lactate or organ function markers), and therefore gives a "not certain" judgment; MedGemma focuses more on the patient's stable vital signs and normal creatinine levels, and therefore gives a "no" classification; while DeepSeek-V3 focuses on high-risk features such as fever, potential malignancy, and elevated inflammatory markers (such as white blood cell count [WBC] and lactate dehydrogenase [LDH]), and therefore gives a "yes" judgment.

[0066] In terms of clinical performance, all LLMs achieved high sensitivity (≥90%) in China, indicating a low probability of missing critical cases. Importantly, the positive predictive value (PPV) of each LLM was higher than that of the risk stratification model used alone. MedGemma achieved the highest PPV in both internal (45.8%) and external (48.7%) validations, with sensitivities of 90.9% and 90.5%, respectively. This indicates a significant reduction in false alarm rate without affecting case detection rate, outperforming traditional two-level models (PPV: internal 11.03%, external 18.81%) and three-level risk stratification models (PPV: internal 40.49%, external 37.17%). These results suggest that integrating LLMs as a post-event optimization step for risk alerts can significantly improve the accuracy of AI-driven early warning systems, thereby reducing alert fatigue and enhancing clinical trust and application rates.

[0067] Explainability and clinical consistency Risk factors for patients at different risk levels (low, intermediate, and high) were identified using the SHAP method, and personalized interpretations were provided for each patient's risk classification. Differences in feature importance and contribution to model prediction across the three risk groups showed that risk factors were more significant in the high-risk group. Results showed that elevated blood urea nitrogen (>20 mg / dL), elevated lactate (>2 mmol / L), decreased albumin (<3.5 g / dL), and decreased lymphocyte percentage (<20%) were all associated with increased risk. SHAP-based missing data analysis confirmed that the missing patterns themselves contained clinically significant signals; for example, missing lactate or triage SpO2 measurements were associated with lower predicted mortality, consistent with selective testing practices in the emergency department. Predicted risk probabilities for representative patients in each risk group were: low-risk patient A (4.97%), intermediate-risk patient B (49.5%), and high-risk patient C (85.05%), with detailed personalized interpretations highlighting key factors for each risk classification. These results highlight the significant differences in the contribution and ranking of risk factors across different risk groups, thus supporting more precise and personalized risk stratification.

[0068] Beyond the interpretability based on SHAP, LLM further enhances interpretability through natural language generation. Specifically, LLM can express complex model outputs in plain language, providing clinicians with intuitive and actionable insights. For example, when assessing high-risk alerts, LLM might generate a narrative explanation like: “This patient has multiple critical abnormalities, including severe lactic acidosis (lactate 18.5 mmol / L), significant hypoalbuminemia (albumin 1.79 g / dL), significantly decreased bicarbonate (6.8 mmol / L), and acute kidney injury (creatinine 3.35 mg / dL). These findings indicate significant organ dysfunction, therefore this patient falls into the high-risk category.” Such narratives help to better understand and trust AI-driven predictions, ultimately improving clinical decision-making.

[0069] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations of these embodiments or their features can be made without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A dynamic decision support system for reducing false alarms from imbalanced data, characterized in that, include: The data acquisition module is configured to acquire the medical record dataset of the sample to be tested; The initial risk prediction module is configured to generate an initial risk estimate based on the medical record dataset. The stratified alert system module is configured to obtain an alert prediction result that the test sample belongs to one of the low-risk group, medium-risk group, and high-risk group based on the comparison results of the initial risk estimate and at least two first stratification thresholds; the first stratification thresholds are calculated based on precision-recall tradeoff calibration with the optimization objective of maximizing standardized net benefit; The clinical decision module is configured to output a clinical decision suitable for the test sample based on the alarm prediction results.

2. The dynamic decision support system for reducing false alarms from unbalanced data according to claim 1, characterized in that, In the stratified alarm system module, if the alarm prediction result of the sample to be tested is a medium-risk group, it is also configured to obtain an alarm prediction result that the sample to be tested belongs to one of the medium-low risk, medium risk, and medium-high risk subgroups based on the comparison result of the initial risk estimate and at least two second stratification thresholds. The value range of the second stratification threshold is between the two first stratification thresholds.

3. The dynamic decision support system for reducing false alarms from unbalanced data according to claim 1 or 2, characterized in that, The system further includes a personalized risk interpretation module, configured to obtain medical record data representing the risk group to which the test sample belongs based on the alarm prediction results, calculate the contribution of each feature in the medical record data set, and filter the key risk factors for each test sample based on the contribution.

4. The dynamic decision support system for reducing false alarms from unbalanced data according to claim 1 or 2, characterized in that, The system also includes an intelligent filtering module configured to evaluate the accuracy of the alarm prediction results using a large language model and output a classification decision including yes, no, and uncertain to identify and filter false alarms.

5. The dynamic decision support system for reducing false alarms from imbalanced data according to claim 1, characterized in that, The appropriate clinical decision for the sample to be tested includes: if it is a low-risk group, outputting a discharge or minimum monitoring plan; if it is a medium-risk group, outputting a continuous observation plan; and if it is a high-risk group, outputting an immediate intervention plan.

6. The dynamic decision support system for reducing false alarms from imbalanced data according to claim 2, characterized in that, The clinical decision-making appropriate for the sample to be tested includes: if it is a medium- to high-risk subgroup, outputting a recommended plan for timely assessment; if it is a medium- to low-risk subgroup, outputting a recommended plan for routine monitoring.

7. The dynamic decision support system for reducing false alarms from unbalanced data according to claim 1, characterized in that, The standardized net benefit is obtained by calculating the difference between the total net benefit of the alarm prediction results and the maximum of the utility values ​​of the two extreme strategies. The total net benefit of the alarm prediction results is calculated using a utility function, and the two extreme strategies are all treatment and no treatment.

8. The dynamic decision support system for reducing false alarms from unbalanced data according to claim 7, characterized in that, The utility function is: Where yi represents the true outcome of the sample, pi represents the predicted in-hospital mortality rate, and k1 and k2 are two risk thresholds. This indicates the correct identification and timely intervention of high-risk patients. and The maximum benefit brought about by ) This indicates the cost of false positives in the high-risk group, reflecting the resource and risk burden caused by unnecessary intervention; The cost of false negatives in the low-risk group reflects the serious harm caused by premature discharge followed by patient deterioration; for cases in the medium-risk group, delayed identification yields only partial benefits, denoted as ,in This is used to characterize the negative impact of delayed treatment on the condition; while the cost of over-observation in the medium-risk group is denoted as... ,in This is used to reflect that the resources required for observation are lower than those required for full intervention.

9. The dynamic decision support system for reducing false alarms from unbalanced data according to claim 1, characterized in that, The initial risk estimate was obtained based on clinical experience. Alternatively, it can be predicted based on an in-hospital mortality prediction model, which is trained using a dataset of medical records from the training set samples.

10. The dynamic decision support system for reducing false alarms from unbalanced data according to claim 1, characterized in that, The data acquisition module interfaces with the hospital's electronic medical record system.

Images