Patient risk level estimation

Abstract

Disclosed are systems and methods for training a risk stratification model for estimating patient risk levels. Weights are extracted from a source model, the weights defining a relationship between patient data and patient risk level. The weights are used to calculated a score for each patient in a target population, and the source model is then refitted to the target population, using a transfer learning estimator based on the calculated scores. The refitted source model can then be outputted for use.

Description

[0001]

[0002] PATIENT RISK LEVEL ESTIMATION

[0003] Technical Field

[0004] The present invention relates, in general terms, to systems and methods for patient risk level identification, also referred to as patient risk stratification. More particularly, the present invention relates to, but is not limited to, transfer learning estimation for refitting a model trained on a general population, for use with specific sub-populations.

[0005] Background

[0006] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

[0007] Risk-scoring models are sparse models with integer point scores. Such models are used pervasively throughout medicine for risk stratification. Risk stratification is the process of specifying the risk of a patient suffering a particular condition or determining a patient prognosis. In particular, riskscoring models have been developed to identify the patients who are most at risk of adverse events or worsening health conditions.

[0008] Accurate identification of patients at risk can be useful for proper allocation of medical resources. The increasing popularity of electronic health records (EHRs) has enabled novel opportunities for creating novel risk models with both domain expert-curated approaches and advanced machine learning solutions. Although EHRs are rich data sources, the irrelevant and redundant information they contain can adversely affect model interpretability and accessibility, especially in the clinical context.

[0009] Despite being promising, EHR-based scoring systems face two major obstacles. Firstly, existing scoring systems are essentially trained using data from a single healthcare system. Due to the budget constraints and limited available labour and data, there is high demand for transportable risk scores to be used in the more generalized situation. However, generalizability is a difficult problem since accuracy is of utmost importance and there are wide variations in the demographic patient populations across institutions (sites) and geographies.

[0010] Many tools and applications exist for medical professionals in hospitals to use as a reference for predicting patient conditions (known as clinical scoring systems). Such systems consistently run into the issue that the models may no longer be valid when the patient settings change. Moreover, when heterogeneity across various demographic populations is not negligible, poor prediction performance is a bottleneck for many clinical conditions. The resulting systematic bias in risk estimation across these subgroups can lead to inappropriate or misinformed treatment decisions. For example, clinical decision system (CDS) tools for the emergency department (ED) have been created using EHR from Duke University Hospital System (DUHS) and Singapore General Hospital (SGH), separately. However, when transferring SGH-trained scores to DUHS, a CDS locally trained at one institution does not perform well when validated at another institution due to the high level of discrepancy between patients in SGH (Asian as majority) and DUHS (White and Black as majorities).

[0011] It would be desirable to provide a patient risk stratification framework employing a model that is readily customized to a specific site or demographic, without requiring sufficient data from that site or demographic to enable the model to be trained from scratch, or at least to provide a useful alternative.

[0012] Summary

[0013] To improve transportability and equity, the present teachings investigate transfer learning as a possible framework for leveraging richer source data (e.g., high quality data, exceptional research clinicians, high performance computing environments, etc.) to facilitate medical discoveries in limited target data (e.g., smaller data). Transfer learning methods are not generally considered in the context of scoring based systems, such as that proposed here.

[0014] Disclosed herein is a method for training a risk stratification model for estimating patient risk levels, comprising: extracting weights from a source model, the weights defining a relationship between patient data and patient risk level; calculating a score for each patient in a target population comprising a plurality of patients, based on the weights; refitting the source model to the target population, using a transfer learning estimator based on the calculated scores; and outputting the refitted source model.

[0015] Also disclosed is a system for training a risk stratification model for estimating patient risk levels, comprising: memory; at least one processor; a model management module comprising a source model; a refitting module, wherein the memory stores instructions that, when executed by the at least one processor, cause: the model management module to extract weights from the source model, the weights defining a relationship between patient data and patient risk level; the at least one processor to calculate a score for each patient in a target population comprising a plurality of patients, based on the weights; the refitting module to refit the source model to the target population, using a transfer learning estimator based on the calculated scores; and output the refitted source model.

[0016] Also disclosed is a method for identifying patient risk level, comprising: obtaining a new patient record; applying a refitted source model generated using the method as described above to the new patient record.

[0017] Also disclosed is a system for identifying patient risk level, comprising: an interface for inputting a new patient record; and the system as described above, for applying the refitting source model to the new patient record.

[0018] Brief description of the drawings

[0019] Embodiments of the present invention will now be described, by way of nonlimiting example, with reference to the drawings in which :

[0020] Figure 1 illustrates a workflow of a framework in accordance with present teachings, herein interchangeably referred to as the "Trans-Score" framework;

[0021] Figure 2 is a block diagram of a system for performing a method of stratifying patient risk; and

[0022] Figures 3, 4(a) and 4(b) are radar plots showing simulation results of an assessment of the performance of the present framework, relative to a target score and source score under various scenarios.

[0023] Detailed description

[0024] Propose herein are improvements to transportability and equity in patient scoring model performance. These improvements leverage transfer learning and scoring systems to develop a new framework for bias-corrected models. The models borrow information across different cohorts without the need for sharing individual patient data.

[0025] Rather than using data originating from a single site, and without needing data per se to be shared across sites, the present methodologies instead provide a more general, and potentially more practical, framework in which data from multiple populations are stored in multiple sites with privacy constraints maintained. These framework afford training of risk-scoring models based on transfer learning approaches.

[0026] The framework 100 illustrated in Figure 1 implements such a transfer learningbased scoring system, as is intended to improve clinical decision-making across multiple healthcare systems. The framework 100 applies transfer learning techniques to adapt pre-trained scoring models derived during Source learning 102, for new, diverse patient populations during Target Learning 104, to increase generalizability and transportability.

[0027] The framework 100 comprises a Source Learning phase 102 and a Transfer Learning phase 104 where information from Source Learning 102 is used to adapt models for a specific patient demographic. During Source Learning 102, a Source Distribution 106 is analysed. The Source Distribution 106 is a population of patients for whom either or both of demographic and site data has been collected. As used herein, "demographic" data is data pertaining to the specific patient.

[0028] The Source Distribution 106 is analysed by a scoring system 108. The aim is to obtain the weights from the source model, which could be beta-coefficients derived from a regression model or other types of coefficients or weights. Although Autoscore is the illustrative extracting mechanism used as an example herein, the present framework is flexible, allowing users to choose other scoring methods that best suit their needs - i.e., any algorithm-derived score / probability can be incorporated into the proposed framework. This step helps leverage the information available in the source data 106 to enhance the performance of the refitted source model - also referred to as the target model. The scoring system 108 can be any desired scoring system. AutoScore is used for present purposes, it having been designed to automate development of interpretable clinical scoring models for patient diagnosis, prognosis and other purposes.

[0029] Scoring systems typically place different weights against electronic health record data features. For example, smoker status, various age ranges, ethnicity, weight, and other features will be weighted differently depending on the condition for which patient risk is being determined - e.g., risk of a patient having, or developing heart disease. For example, a health record may be represented as a [ l]x[ / V] row matrix of N values (e.g., values for smoker status, height, weight, age and others), and weights may be represented as a [ / V]x[l] column matrix of N weights. Each value and weight may be an integer or a floating point number as required. Multiplying the matrices will result in a number that can be compared against a set of reference ranges to determine the patient scores - e.g., a score of 1 to 5 may be low risk, 6 to 10 may be medium risk, 11 to 15 may be high risk, and 16 to 20 may require clinical referral. The weights thus define a relationship between patient data and patient risk level.

[0030] The scoring system 108, also referred to as a source model, extracts the weights from the Source Distribution 106. The weights are then applied to the target population to calculate a score for each patient. The Target Learning process 104 receives the weights 110 and then calculates a score for each patient in the target population represented by the Target Distribution 112, based on the weights 110. This step can involve any number of operations, for example, multiplying the feature values of each patient by the corresponding weights and summing them up to obtain an aggregated score. By doing so, the knowledge gained from the source data 106 is utilised to make more accurate predictions for the target population 112, despite the limited availability of target data - i.e., data in the target population 112.

[0031] The scores may thus be calculated using the same source model 108, albeit applied to target data, for each patient in the Target Distribution 112 - e.g. data from a target population, which may comprise one or both of a target demographic or demographics, or a target site or sites, for which risk level estimation is desired. The scores may alternatively be calculated using a different model, where the weights from the source model can be adapted for the different model - e.g. where age ranges do not exactly overlap, or where demographics are differently split, weights can be combined, averaged and other operations, to generate adapt the weights for the different model. In this sense, a mapping from one model to another can be used to adapt weights learned using one model, for use in another model.

[0032] The source model 108 is then refitted, using a refitting module 114, to the target population. The model that was initially trained using the source data (i.e., the source model) is refined and recalibrated to better fit the target data, thus enhancing its predictive performance when applied to the target population. This is achieved using a transfer learning estimator in a regularized logistic regression setting, for the target population 112. One such transfer learning estimator is the Oracle Trans-Lasso for estimation and prediction of a target regression vector that can be applied to the patients in the target population to estimate risk - e.g. risk of developing or succumbing to a particular medical condition.

[0033] In some embodiments, a two-stage strategy is used to generate the transfer learning estimator. In the first stage, a weighted estimate is computed. This can be achieved by solving an optimization problem that aims to minimize the difference between observed and predicted values for each specific site and demographic group in the target data. A penalty term can be added to the optimization problem, to prevent overfitting and ensure the model remains sparse and interpretable. The penalty term can be based on a hyper-parameter, which may be pre-determined. The second stage involves obtaining the transfer learning estimator. This is achieved by adding the previously computed weighted estimate to an offset term. The offset term is computed by solving a similar optimization problem as above, but for the general population rather than specific demographic groups. Again, a penalty term can be added to the optimisation function. This penalty term ensures model parsimony. Usually the source population 106 has a larger sample size than target population 112, and therefore can be considered more representative for a general population of regions or countries. For example, the source data may come from all admissions of a large public hospital, while target data may come from a specific group of patients with certain diseases (therefore sample size is much smaller)

[0034] Thus, in such embodiments the target population 112 may comprise a plurality of groups, such as demographics or sites (e.g. clinics or geographical regions). In this instance, refitting the source model to the target population is not straight-forward, since heterogeneity in the target population may be non- negligible. Thus, for each group, refitting involves minimizing the difference between the calculated scores (i.e., the scores calculated using the weights 110, for patients in the Target Distribution 112) and a set of observed values corresponding to the patients in the respective group, thereby obtaining a weight estimate referred to as Target Outcome Y. There may therefore be either single Target Outcome Y, being an aggregate of all target outcomes for each group, or a Target Outcome Y for each group. The observed values may be clinically derived, or expert data (e.g., that which is produced by a clinician - such that the target distribution comprises patient data for each patient, from which the score can be calculated, and a clinical value or score for each patient against which the calculated score can be tested or compared). An offset term, referred to as Target Outcome X, is then obtained by minimizing the difference between the calculated scores and the set of observed values corresponding to the patients, across the target population, as mentioned above. Target Outcome

[0035] Y is thus a demographic-specific or site-specific transfer learning contribution, while Target Outcome X is a global weight offset that improves generalizability across the population.

[0036] The source model, which can itself have been trained on general patient population data or a data set other than the target distribution 112, is then refitted using the weight estimates and offset term.

[0037] The penalty terms are derived from hyper-parameters - e.g., the penalty terms are the hyper-parameters themselves, or may result from a mathematical operation applied to a hyper-parameter - e.g., by multiplying the hyperparameter by an empirically determined coefficient. These hyper-parameters are selected through any suitable known process, such as cross-validation. The cross-validation process seek to maximize the ability of the model to accurately differentiate between different patient risk levels - i.e., discriminatory power. The cross-validation may be performed by applying the source model to the target distribution 112 using various random splits of the target distribution 112, to test the ability of the source model to generalize across the target distribution 112.

[0038] The refitting process learns the difference between the source and target models, enabling better adaptation to the target data. The refitting step can therefore result in a refitted source model, or a target model, that has learned from the general population Source Distribution 106, while having been adapted for use with a specific group or target population. The refitted source model may therefore comprise a single model, or may comprise a plurality of estimators, each estimator comprising the source model refitted to a respective one of the groups.

[0039] The proposed framework incorporates heterogeneous data from diverse populations and multiple cohorts to improve model fitting and prediction in an underrepresented population.

[0040] Figure 2 shows a system 200 for implementing the framework 100 of Figure 1. The system 200 comprises memory 202. The memory 202 stores program code 204 that is executed by the system 200, particularly by processor(s) 206 thereof, to implement the framework 100. The memory 202 may be any appropriate non-transitory storage medium. A model management module 208 stores the source model and, if applicable, previously calculated target models. A refitting module 210 is also in communication with the memory 202, processor(s) 206 and model management module 208, for refitting the source model.

[0041] The model management module 208 extracts weights from the source model. The processor(s) 206 calculates a score for each patient in the target population 112 based on the weights extracted by the model management module 208. The refitting module 210 then refits the source model to the target population, using a transfer learning estimator, such as that described above, based on the calculated scores. The system 200 then outputs the refitted source model. Outputting the refitted source model may comprise storing it in a clinic or site system available for analysing patient data to output a risk of the patient experiencing a particular medical condition, or that the patient has developed such a medical condition. For example, outputting the refitted source model may instead comprise transmitting the refitted source model, via network interface 212, over network 214, to a remote server associated with a clinic or hospital.

[0042] The model management module 208 and the refitting module 210 may be used for training the source model on source data extracted from a source data database 218, over network 214, the source data being general patient population data or data from other clinics and hospitals. Similarly, during refitting to a target population, target population data may be extracted from target distribution 220, over network 214.

[0043] Once the target model or refitted source model is generated, a new patient record can be obtained and the refitted source model applied thereto. This can be achieved using network interface 212 for receiving the new patient record, and processor(s) 206 to executed the framework, or method embodied by that framework, to the new patient record.

[0044] The refitted source model can be customized, either during refitting or afterwards, based on user needs or requirements. To do so, users can apply customized transformation of the new weights (e.g., point-based transformations) to tailor the source model. This transformation process can involve applying various mathematical operations or functions to the weights to align them with the user's domain knowledge, preferences, or interpretability requirements. By allowing such customization, the present framework becomes more versatile and adaptable to different applications.

[0045] Performance of the present framework, namely the Trans-Score framework, will now be assessed by comparing its accuracy, privacy, and interpretability to those of other models or benchmarks. This evaluation process involves measuring key performance indicators, such as classification accuracy, Flscore, or area under the ROC curve, to ensure that the framework achieves a suitable balance between prediction performance and privacy preservation. Additionally, the interpretability of the model is evaluated by examining the sparsity of the coefficients and the significance of the features used in the model. Different sample sizes are considered for both source and target populations, various data shift settings between these populations, and other factors that may impact the performance of the framework.

[0046] The assessment involved simulating model learning used various cohorts. Relevantly:

[0047] • Sample size. Sample sizes were tested in two different configurations:

[0048] (1) the majority demographic group had a moderate sample size (2000) compared to the minority demographic group (1000); and

[0049] (2) the majority demographic group had a large sample size (5000) compared to the minority demographic group.

[0050] • Predictors. Predictor variables were generated from a normal distribution, with various means and standard deviations, to resemble the input data from different demographic groups. The dimension of these predictors varied based on the simulation, ranging from 10 to 90. Different levels of sparsity were considered. Sparsity is reflected by the proportion of zero coefficients in the model - e.g. "0" values or weights in the matrix example set out above - reflecting the level of interpretability and the ability of the framework to identify important features.

[0051] • Data shift. The study considered two types of shifts: covariate shift and model shift. Covariate shift refers to changes in the distribution of predictor variables between the training and target datasets. Model shift refers to changes in the relationship between predictors and the outcome variable. These shifts mimic the challenges faced in real-world applications when models developed in one setting are applied in another. In the present study, various different levels of data shift were considered.

[0052] To assess the performance of the present framework, the following evaluation metrics were used. For each simulation scenario, the process was repeated 1000 times and the average performance metrics were reported for each scenario.

[0053] • Area under the ROC curve (AUC-ROC): A measure of the framework's ability to discriminate between positive and negative outcomes, accounting for both sensitivity and specificity.

[0054] • Fl-score: The harmonic mean of precision and recall, providing a balanced measure of the framework's performance, especially in cases of imbalanced datasets.

[0055] The simulation results presented in Figures 3, 4(a) and 4(b), and show a robust assessment of the performance of the present framework, relative to the target score and source score, obtained from the target and source models that use the LASSO operator, under various scenarios. Specifically, AUCs of different methods are compared across changing sample sizes and varying settings of predictor dimensions (p), differences (s), and noise levels (h).

[0056] Figure 3 shows that as the dimensionality of predictors (p) is increased, while maintaining a fixed noise level (h = 4), the performance of the present framework demonstrated a marked improvement - performance is represented by the orange line. This improvement is most prominent when p is at its maximum value of 100. This suggests that the present framework effectively leverages the added complexity introduced by higher-dimensional predictor variables to improve its predictive accuracy.

[0057] Figure 4(a) focusses on a particular setting where p is fixed at 100 and s is set at 0.2p. As the value of h (which represents the noise level) changes incrementally from 4 to 7, the present method continues to outshine the target score and source score. The relative performance advantage of Trans-Score is maintained despite minor disparities at different h settings, further corroborating its robustness to noise in the data. In Figure 4(b), the noise level h is fixed at 4 and the predictor dimension p is kept at 100, while the difference (s) varies. In this scenario, the present framework consistently demonstrates superior AUC values across all sample sizes, confirming its robust performance under differing levels of dissimilarity between the source and target data. Even in scenarios where the value of s increases and the advantage of the present framework over the source score is observed only at specific sample sizes, it is notable that the present framework invariably outperforms the target score across all s settings.

[0058] In summary, these simulation results provide empirical evidence for the robust and superior performance of the present framework, Trans-Score. Its resilience to alterations in data complexity, dissimilarity, and noise levels, as well as its consistent performance advantage over the target score and source score, validate the effectiveness of Trans-Score. These findings collectively underscore the utility of Trans-Score as a reliable and potent tool in enhancing predictive performance in healthcare risk stratification.

[0059] It will be appreciated that many further modifications and permutations of various aspects of the described embodiments are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

[0060] Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

Claims

Claims1. A method fortraining a risk stratification model for estimating patient risk levels, comprising: extracting weights from a source model, the weights defining a relationship between patient data and patient risk level; calculating a score for each patient in a target population comprising a plurality of patients, based on the weights; refitting the source model to the target population, using a transfer learning estimator based on the calculated scores; and outputting the refitted source model.

2. The method of claim 1, wherein the target population comprises a plurality of groups, and wherein refitting the source model to the target population comprises: for each group, minimizing a difference between the calculated scores and a set of observed values corresponding to the patients in the respective group, thereby obtaining a weight estimate; minimizing a difference between the calculated scores and the set of observed values corresponding to the patients, across the target population, thereby obtaining an offset term; and refitting the source model using the weight estimates and offset term.

3. The method of claim 2, wherein the refitted source model comprises one or more estimators, each estimator comprising the source model refitted to a said group.

4. The method of claim 2 or 3, wherein each group comprises one or more both a group of patients from a specific site, and a group of patients from a specific demographic.

5. The method of any one of claims 2 to 4, wherein obtaining the weight estimate comprises applying an optimization function that seeks to minimize the difference between the calculated scores and the set of observed values corresponding to the patients in the respective group.

6. The method of any one of claims 2 to 5, wherein obtaining the offset term comprises applying an optimization function that seeks to minimize the difference between the calculated scores and the set of observed values corresponding to the patients in the target population.

7. The method of claim 5 or 6, wherein each optimization function comprises a penalty term based on a respective hyper-parameter, for enforcing sparseness.

8. The method of claim 7, wherein each hyper-parameter is selected through cross-validation that maximizes discrimination between risk levels in the respective patient group or population.

9. The method of any one of claims 1 to 8, wherein the source model is trained on a general population.

10. A system for training a risk stratification model for estimating patient risk levels, comprising: memory; at least one processor; a model management module comprising a source model; a refitting module, wherein the memory stores instructions that, when executed by the at least one processor, cause: the model management module to extract weights from the source model, the weights defining a relationship between patient data and patient risk level;the at least one processor to calculate a score for each patient in a target population comprising a plurality of patients, based on the weights; the refitting module to refit the source model to the target population, using a transfer learning estimator based on the calculated scores; and output the refitted source model.

11. The system of claim 10, wherein the target population comprises a plurality of groups, and wherein the refitting module refits the source model to the target population by: for each group, minimizing a difference between the calculated scores and a set of observed values corresponding to the patients in the respective group, thereby obtaining a weight estimate; minimizing a difference between the calculated scores and the set of observed values corresponding to the patients, across the target population, thereby obtaining an offset term; and refitting the source model using the weight estimates and offset term.

12. The system of claim 11, wherein the refitting module refits the source model to each said group, such that the refitted model comprises a model comprises one or more estimators, each estimator comprising the source model refitted to a said group.

13. The system of claim 11 or 12, wherein each group comprises one or more both a group of patients from a specific site, and a group of patients from a specific demographic.

14. The system of any one of claims 11 to 13, wherein the refitting module obtains the weight estimate by applying an optimization function that seeks to minimize the difference between the calculated scores and theset of observed values corresponding to the patients in the respective group.

15. The system of any one of claims 11 to 14, wherein the refitting module obtains the offset term by applying an optimization function that seeks to minimize the difference between the calculated scores and the set of observed values corresponding to the patients in the target population.

16. The system of claim 14 or 15, wherein each optimization function comprises a penalty term based on a respective hyper- para meter, for enforcing sparseness.

17. The system of claim 16, wherein the refitting module selects each hyperparameter through cross-validation that maximizes discrimination between risk levels in the respective patient group or population.

18. The system of any one of claims 10 to 17, wherein the source model is trained on a general population.

19. A method for identifying patient risk level, comprising: obtaining a new patient record; applying a refitted source model generated using the method of any one of claims 1 to 9 to the new patient record.

20. A system for identifying patient risk level, comprising : an interface for inputting a new patient record; and the system of any one of claims 10 to 18, for applying the refitting source model to the new patient record.

Images