Methods and apparatus for identifying clinical conditions suitable for categorization with machine-learning models

Abstract

In some embodiments, a method for identifying clinical conditions suitable for categorization with machine learning models includes obtaining a list of clinical conditions and obtaining a dataset of biomarkers for the clinical conditions list. The method includes for each clinical condition (1) determining if a prevalence of the clinical condition within the dataset is within an expected range; and (2) if the prevalence of the clinical condition within the dataset is within the expected range (a) performing ML training and validation on an ML model; (b) determining if one or more performance metrics are met by the trained ML model; and (c) if the one or more performance metrics are met, identifying the trained ML model as suitable for identifying the clinical condition. Numerous other embodiments are provided.

Description

METHODS AND APPARATUS FOR IDENTIFYING CLINICAL CONDITIONS SUITABLE FOR CATEGORIZATION WITH MACHINE-LEARNING MODELSFIELD

[0001] The present application relates to medical testing and more particularly to methods and apparatus for identifying clinical conditions suitable for categorization with machine learning models.BACKGROUND

[0002] Most efforts to apply machine learning (ML) fail, placing an operational burden on businesses and degrading the trust of both decision makers and end users. For example, an ML model may fail because of the dataset used to train the model, the type of ML model employed, the method of training and validation used, the specific requirements of the setting in which the model will be employed, etc. In a clinical setting the fail rate may be even higher given the rigor and complexity of predicting disease. In some cases, as many as 90% of the major diseases diagnosed in a clinical setting may not be identifiable using ML models (to an effective level for the clinical setting), with no suitable ability to determine which 10% of the diseases are detectable by ML models. For this reason, laboratory technicians, nurses, doctors, or other practitioners may resist utilizing artificial intelligence and machine learning tools to assist with decision making and diagnosis.

[0003] If given the task to predict a disease in a dataset which is resistant to producing effective results, a data scientist may resort to nonempirical methods and analysis. These nonempirical methods may be used as a last resort by companies that have unsuccessfully invested significant time and money investigating a solution. When used with real-world data, such nonempirical methods generally fail to reproduce the accuracy seen with training and test datasets.

[0004] As such, a need exists for methods of identifying clinical conditions suitable for categorization with ML models, that streamline the ML modelling process, and that maximize ML modelling opportunities without placing an operational burden on businesses or degrading the trust of end users.SUMMARY

[0005] In some embodiments, a method is provided for identifying clinical conditions suitable for categorization with ML models. The method includes obtaining, via a computer, a list of clinical conditions, and, obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions. The method further includes, for at least one clinical condition (1) performing ML training and validation on an ML model; (2) determining, via the computer, ifone or more performance metrics are met by the trained ML model; and (3) if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition.

[0006] In some embodiments, a system is provided for identifying clinical conditions suitable for categorization with ML models. The system includes a processor and a memory coupled to the processor. The memory includes computer program instructions stored in the memory that, when executed by the processor, cause the processor to obtain a list of clinical conditions, obtain a dataset of biomarkers for the list of clinical conditions, and for at least one clinical condition (1) initiate ML training and validation on an ML model; (2) determine if one or more performance metrics are met by the trained ML model; and (3) if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition.

[0007] In some embodiments, a method of identifying clinical conditions suitable for categorization with ML models includes (1) obtaining, via a computer, a list of clinical conditions of interest to an institution; (2) obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions, the dataset of biomarkers determined in a clinical setting of the institution; and (3) for each clinical condition: (a) performing ML training and validation on an ML model; (b) determining, via the computer, if one or more performance metrics are met by the trained ML model; and (c) if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition at the institution.

[0008] In some embodiments, a method is provided for identifying clinical conditions suitable for categorization with ML models. The method includes obtaining, via a computer, a list of clinical conditions, and, obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions. The method further includes for each clinical condition (1) determining, via the computer, if a prevalence of the clinical condition within the dataset is within an expected range; and (2) if the prevalence of the clinical condition within the dataset is within the expected range (a) performing ML training and validation on an ML model; (b) determining, via the computer, if one or more performance metrics are met by the trained ML model; and (c) if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition.

[0009] In some embodiments, a system is provided for identifying clinical conditions suitable for categorization with ML models. The system includes a processor and a memory coupled to the processor. The memory includes computer program instructions stored in the memory that, when executed by the processor, cause the processor to obtain a list of clinical conditions, obtain a dataset of biomarkers for the list of clinical conditions, and for each clinical condition (1) determine if a prevalence of the clinical condition within the dataset iswithin an expected range; and (2) if the prevalence of the clinical condition within the dataset is within the expected range (a) initiate ML training and validation on an ML model; (b) determine if one or more performance metrics are met by the trained ML model; and (c) if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition.

[0010] Other features and aspects of the present invention will become more fully apparent from the following detailed description, the appended claims, and the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1A illustrates an example flow diagram of a method of identifying clinical conditions suitable for categorization with machine learning models in accordance with embodiments provided herein.

[0012] FIG. 1 B illustrates an example computer in which the method of FIG. 1A may be implemented in accordance with one or more embodiments.

[0013] FIG. 2 illustrates a flowchart of a method of identifying clinical conditions suitable for categorization with machine learning models in accordance with one or more embodiments.

[0014] FIG. 3 illustrates a flowchart of an additional method of identifying clinical conditions suitable for categorization with machine learning models in accordance with one or more embodiments.DETAILED DESCRIPTION

[0015] Independent of the grammatical term usage, individuals with male, female or other gender identities are included within the term.

[0016] Prediction of various clinical conditions and diseases based on routine and specialized biomarker panels, known as clinical decision support (CDS), is a subject of active research. Rule-based methods are being used to predict and / or detect diseases, including presence / absence of disease, disease states, and severity. Efforts are also being made to apply artificial intelligence (Al) and machine-learning (ML) methods.

[0017] Detectability of various diseases is highly dependent on the clinical setting (e.g., inpatient, outpatient, primary care, emergency department, ICU, or specialized clinic). Further, the feasibility and performance of a clinical condition’s detection method is highly dependent on the clinical setting and the dataset used to develop the detection method. For example, an emergency department typically has a patient population with acute diseases such as myocardial infarction, stroke, respiratory distress, severe infection, acute kidney disease, etc., whereas a primary care setting has a significant patient population with chronicor metabolic diseases such as diabetes, cardiovascular disease, liver disease, cancer, etc. Additionally, different clinics may have different standards / protocols on how to diagnose or treat specific clinical conditions. Thus, the data available for detecting clinical conditions may vary significantly based on the clinical setting, as may the metrics used to diagnose such conditions.

[0018] Embodiments provided herein allow identification of clinical conditions suitable for categorization with ML models within any clinical setting (e.g., inpatient, outpatient, primary care, emergency department, ICU, specialized clinic, etc.). As stated, above, most ML models fail. An ML model may not be usable for its intended purpose based on numerous factors such as the clinical condition to be detected, the dataset used to train the model, the type of ML model employed, the method of training and validation used, the specific requirements of the clinical setting in which the model will be employed, etc.

[0019] Embodiments provided herein reduce the risk associated with the building of an ML-based clinical disease identification portfolio. As an example, if there are 100 major diseases (e.g., clinical conditions) to detect in a particular clinical setting, as many as 90% may not be identifiable using ML models (to an effective level for the clinical setting).Methods and apparatus provided herein may be used to determine which of the 100 clinical conditions are suitable for categorization with machine learning models. In some embodiments, given the clinical conditions to be identified and the appropriate biomarker datasets, methods and apparatus provided herein may semi-automatically or automatically train and validate ML models for each clinical condition within the datasets. While only 10% (or 10) of the clinical conditions may be identifiable in a clinical setting with trained ML models, all 10 of the conditions will be identified (and trained ML models will be provided). This is to be contrasted with a conventional approach in which 10 clinical conditions are selected (e.g., based on clinical needs) for ML modeling. Based on the 90% fail rate, on average, only one of the 10 clinical conditions will be identifiable by a trained ML model.

[0020] In some embodiments, methods and apparatus are provided that allow an institution such as a hospital, a laboratory, an ICU, an emergency room, a primary care facility, etc., to identify clinical conditions suitable for categorization with ML models based on biomarker data from the institution itself (e.g., a biomarker dataset from the hospital, laboratory, ICU, emergency room, primary care facility, etc.). The institution’s own biomarker data may be employed to train ML models based on clinical conditions of interest to the institution (e.g., a list of clinical conditions specified by the institution). These trained ML models may be validated and analyzed to determine if they meet performance requirements (e.g., of the institution). Trained ML models that meet performance requirements may be deployed at the institution for use during clinical testing (e.g., to predict and / or detect diseases, including presence / absence of disease, disease states, and severity).

[0021] In one or more embodiments, automated or semi-automated systems and methods are provided for identifying clinical conditions detectable with ML models in a specific setting (e.g., inpatient, outpatient, primary care, emergency department, ICU, specialized clinic, etc.). A biomarker dataset is obtained for a particular clinical setting. For each clinical condition of interest, the prevalence of the clinical condition within the dataset is examined. For example, a determination may be made whether the clinical condition prevalence within the dataset is consistent with published results. If the prevalence significantly deviates from what is expected, the clinical condition may still be appropriate for ML modelling with the biomarker dataset but may require further investigation (e.g., a trauma hospital might not be recording metabolic diseases whereas those diseases are still present in the trauma hospital population). Assuming the prevalence level of the clinical condition within the biomarker data is acceptable, ML training is performed on the specific clinical condition. For example, different combinations of ML models using different combinations of hyperparameters may be trained and validated. In some embodiments, subgroup analysis may be performed to improve ML model performance, such as by identifying subgroups that perform well or subgroups that perform poorly as described further below. Example subgroups may include gender, age, locality, etc. For each trained ML model, performance metrics are examined to determine whether the trained ML model is suitable for identifying the clinical condition (e.g., within a relevant subgroup). Thus, multiple clinical conditions may be checked as candidates for ML modeling and clustered into ML-detectable and ML- undetectable categories. In this manner, each clinical condition in a set of clinical conditions may be identified as detectable or undetectable with ML methods for a given clinical setting. In some embodiments, a ranking of ML models may be provided based on multiple performance metrics.

[0022] Embodiments provided herein may be employed to create a portfolio of low- risk, high-performance, ML-based CDS solutions adapted to the specific needs of a specific hospital, primary care setting, laboratory, or other clinical setting. The methods and apparatus provided herein streamline the ML modelling process and maximize ML modelling opportunities without placing an operational burden on businesses or degrading the trust of end users.

[0023] These and other embodiments are described below with reference to FIGS. 1A-3.

[0024] FIG. 1 A illustrates an example flow diagram 100 of a method of identifying clinical conditions suitable for categorization with ML models in accordance with embodiments provided herein. With reference to FIG. 1A, and as described in further detail below, the method of flow diagram 100 includes obtaining a list of clinical conditions 102 to examine for use with predictive ML models (e.g., ML models that identify diseases or otherconditions of interest to inpatient, outpatient, primary care, emergency department, ICU, a specialized clinic, etc., such as presence / absence of disease, disease states, and / or severity). A biomarker dataset relevant to each clinical condition is then obtained (e.g., biomarker datasets 104 in FIG. 1A). In some embodiments, the same biomarker dataset may be employed for multiple clinical conditions (e.g., the same biomarker dataset may be used to train different ML models, each targeting a specific clinical condition). In other embodiments, different biomarker datasets may be employed for at least some different clinical conditions. Example biomarker datasets are described below.

[0025] Thereafter, for each clinical condition listed, the relevant biomarker dataset is examined, and if determined to be suitable, is used to train and validate one or more ML models 106 for each clinical condition. In some embodiments, each trained ML model 106 may be analyzed to identify any relevant subgroups. Thereafter, the performance metrics for each ML model are examined (e.g., for the identified relevant subgroups, if appliable). ML models 106 with acceptable performance metrics may then be identified as suitable for use, and in some embodiments, deployed in a clinical setting (e.g., as deployed ML models 108). In this manner, numerous ML models may be obtained for identifying clinical conditions (for relevant within any clinical setting.

[0026] FIG. 1 B illustrates an example computer 120 in which the method of FIG. 1 A may be implemented in accordance with one or more embodiments. With reference to FIG. 1B, computer 120 includes a processor 122 coupled to a memory 124. Memory 124 may include clinical condition list 102, biomarker datasets 104, and ML models 106. Memory 124 may also include one or more programs 126 for carrying out the methods described herein when executed by processor 122, such as selecting a clinical condition for ML modeling, determining the prevalence of clinical conditions within datasets, initiating ML model training and validation, performing subgroup analysis, analyzing performance metrics, and the like. In some embodiments, processor 122, executing one or more of programs 126, may train ML models 106 based on biomarker dataset 104 and a clinical condition (e.g., generating trained ML models 108 (FIG. 1A) that may be deployed in a hospital, laboratory, etc.). In other embodiments, processor 122 may direct other computing resources to train ML models (e.g., central processing units (CPUs), graphics processing units (GPUs), etc.). Memory 124 may include multiple memory units and / or types of memory. In some embodiments, all or a portion of memory 124 may be external to and / or remote from computer 120. Additionally, in some embodiments, multiple processors may be employed. As stated, programs 126 may include computer program instructions executable by processor 122.

[0027] FIG. 2 illustrates a flowchart of a method 200 of identifying clinical conditions suitable for categorization with ML models in accordance with one or more embodiments. With reference to FIG. 2, in block 202, method 200 includes obtaining a dataset ofbiomarkers for clinical conditions of interest, and, in block 204, selecting a clinical condition for ML modeling. For example, clinical conditions may be specified by international classification of disease (ICD) codes or otherwise specified.

[0028] In some embodiments, the biomarker dataset may be obtained from a hospital, a laboratory, another clinical setting, a healthcare data provider such as Dandelion Health, Inc. of New York, NY, Prognos Health, Inc. of New York, NY, or the like, In one or more embodiments, a biomarker dataset for the clinical condition may be formed from a subset of biomarkers in a larger biomarker dataset. For example, biomarkers relevant to a complete blood count (CBC) panel, a basic metabolic panel (BMP), a comprehensive metabolic panel (CMP), a lipid panel, an immunoassay panel, a urine panel, another biomarker panel, combinations of biomarker panels, etc., may be selected from a larger dataset and used to form a biomarker dataset for the clinical condition.

[0029] In some embodiments, the biomarker dataset may contain biomarkers determined for a specific hospital (e.g., biomarkers determined for at least one of an emergency room, an in-patient clinic, an outpatient clinic, a doctor’s office, and / or a laboratory affiliated with the specific hospital). In other embodiments, the biomarker dataset may contain biomarkers determined for at least one of a plurality of hospitals and a plurality of laboratories.

[0030] In instances in which the biomarker dataset includes hospital or laboratory data, in some embodiments, the biomarkers present within the biomarker dataset may be obtained from a laboratory information system (e.g., validated biomarkers, such as biomarkers validated by a rules-based methodology). In other embodiments, the biomarker training dataset may include unvalidated biomarkers (e.g., from middleware or directly from a laboratory analyzer).

[0031] In some embodiments, biomarker datasets may include biomarkers in a biomarker panel such as biomarkers from at least one of a lipid panel, a CBC panel, a BMP, a CMP, a thyroid panel, a urine panel, and an immunoassay panel. In another example, the biomarker panel may include results from routine diagnostic tests with common biomarkers measured during yearly physicals, routine health checks, or the like (e.g., biomarkers from a lipid panel, a CBC panel, and a CMP).

[0032] In some embodiments, a biomarker dataset may include 3 or more biomarker values for each biomarker in the dataset. Other numbers of biomarker values may be employed (e.g., 4, 5, 10, 20, 50, or more).

[0033] After the biomarker dataset for the clinical condition has been obtained (e.g., a biomarker dataset 104 of FIGS. 1A and 1 B), in block 206, method 200 includes determining if the prevalence of the clinical condition within the dataset is within an expected range. For example, in some embodiments, memory 124 may include a table or databasethat includes the expected prevalence ranges for clinical conditions (e.g., based on a literature review, experimental results, etc.). Such an inquiry may include, for example, a determination of whether the prevalence of the clinical condition with the biomarker dataset will lead to an effective identification (e.g., if there are only a few examples of a disease state in a dataset then deriving an identification of that disease may lead to large error rates when applying the identification to real time data).

[0034] If it is determined that the prevalence of the clinical condition within the biomarker dataset is not within an expected range, in block 208, method 200 includes determining if the prevalence of the clinical condition within the biomarker dataset is within an acceptable range despite being outside of the expected range. The prevalence of the clinical condition within the biomarker dataset being outside the expected range may indicate a source of error in sample collection, handling or storage, in biomarker reporting, etc. Alternatively, the discrepancy may be due to differences in the population tested (e.g., chronic illnesses have a lower prevalence in an acute care setting, a dialysis clinic has more patients with kidney disease, etc.). If no error is found, and the prevalence of the clinical condition is higher than expected, the biomarker dataset may still be useable for training an ML model for identifying the clinical condition (as described below with reference to block 210). However, if no error is found and the prevalence of the clinical condition is lower than expected, the biomarker dataset may not be suitable for training an ML model for identifying the clinical condition. In such cases, the clinical condition may be marked or otherwise identified as being not suitable for ML modeling for the biomarker dataset (as described below with reference to block 216).

[0035] Returning to block 206, if the prevalence of the clinical condition within the biomarker dataset is within the expected range, in block 210, method 200 includes performing ML training and validation using the biomarker dataset as input and the clinical condition as the target output (e.g., using the presence and absence of the clinical condition as outputs).

[0036] In some embodiments, prior to training, the biomarker dataset may be examined to determine its relevance to the clinical condition and to eliminate redundant biomarkers (e.g., using maximum relevance minimum redundancy (MRMR) feature selection or another suitable algorithm). In this manner, an optimized training dataset of biomarkers may be obtained. In some embodiments, the ML model may also be trained with demographic information (e.g., age, sex, gender, etc.).

[0037] Example ML algorithms that may be employed include tree-based ML algorithms such as decision trees, gradient boosted trees (e.g., XGBoost or LightGBM), random forests, etc. Other ML algorithms may be used such as a deep learning algorithm (e.g., a deep neural network (DNN)), a convolutional neural network (CNN), a region-basedCNN (R-CNN), a fully convolutional neural network (FCN), a region-based FCN (R-FCN), etc. Other suitable neural networks may be employed. Example architectures include Inception, ResNet, ResNeXt, DenseNet, or the like, although other architectures may be employed.

[0038] In at least some embodiments, multiple ML models may be trained including multiple versions of the same ML architecture with different hyperparameters, different ML architecture models, combinations of the same, etc. In one or more embodiments, an automated machine-learning pipeline may be employed such as Databricks AutoML available from Databricks Inc. of San Francisco, CA.

[0039] In some embodiments, after training, each ML model may be fine-tuned and / or validated. For example, a subset of the biomarkers of the biomarker dataset that are altered in the presence of the clinical condition may be determined and the ML model may be retrained based on the biomarker subset. In one or more embodiments, Shapley additive explanations (SHAP) feature importance, decision tree feature importance, permutation feature importance, or a similar technique may be employed to identify the most relevant biomarkers employed during training (e.g., those that impact a clinical condition diagnosis). These identified biomarkers may then be employed to update the training of the ML model by retraining the ML model using the identified biomarkers (e.g., by giving the identified biomarkers more weight or omitting biomarkers with low importance). Other example validation techniques may include train and test split, cross-validation, holdout validation, Monte Carlo, etc.

[0040] In one or more embodiments, subgroup analysis may be employed to improve ML modeling performance. For example, if the biomarker dataset used during training of an ML model represents a large, diverse population, such as the biomarker dataset for an entire hospital, laboratory, or network population, it may be desirable to understand how the ML model behaves for subgroups within the entire population such as males versus females or patients from inpatient versus outpatient versus ICU. Further granularity may include male inpatient versus female inpatient, for example. Underrepresented subgroups within the entire population may not perform as well as larger subgroups when an ML model is trained on the entire population. Subgroup analysis may allow the identification of subgroup populations on which the ML model performs poorly (and potentially retraining of the ML model with biomarkers more specifically targeted to the subgroups, such as a subset of the entire biomarker dataset). Thus, an initial ML model trained on a large population (e.g., an entire hospital or laboratory) may be suitable for use with some subgroups but other subgroups may require separately retrained ML models. Additionally, for the more general ML model trained on the entire population, poorly performing subgroups may be removed (and the ML model retrained) to improve the generalML model’s performance.

[0041] Thus, following training and validation, in block 212, method 200 optionally includes performing analysis to identify relevant subgroups for which the trained ML model may be used to identify the clinical condition. In some embodiments, subgroups may be identified (e.g., automatically) from categorial variables and numerical variables (e.g., using binning). Other subgroups may be employed, such as multi-category sub-groups. Example subgroups include gender, age, clinical setting (e.g., emergency room versus ICU versus clinical laboratory), or the like. For instance, a trained ML model may accurately identify a clinical condition for men but not women.

[0042] In some embodiments, subgroups may be based on locality. For example, emergency room specific conditions such as acute coronary syndrome (ACS), stroke, or other emergency room specific conditions may be relevant to an emergency room but not a clinic that performs metabolic health screens. Likewise, metabolic health information is not typically relevant to an ICU. In general, if a clinical condition is relevant to a subgroup (e.g., the ICU, an emergency room, etc.), the clinical condition should be present in a biomarker dataset related to the subgroup (e.g., biomarker datasets from an ICU, emergency room, etc ). In some embodiments, separate ML models may be trained for poorly performing subgroups and / or a general ML model may be retrained with one or more poorly performing subgroups removed.

[0043] Following subgroup analysis, in block 214, method 200 includes determining if performance metrics for ML modeling have been met (e.g., in general and / or for any identified relevant subgroup(s) if the ML model is subgroup specific). Performance metrics may include how well the trained ML model identifies the clinical condition on which it was trained. Performance metrics may be based on requirements of the clinical setting in which the trained ML model will be used, a literature review, experimental results, etc.

[0044] If performance metrics have not been met (as determined in block 214), method 200 proceeds to block 216 wherein the clinical condition is identified as not suitable for ML modeling. However, if performance metrics are met by the trained ML model for the clinical condition (e.g., for the relevant subgroup(s), if applicable), in block 218, method 200 includes marking or otherwise identifying the clinical condition as suitable for modeling with the trained ML model (e.g., for the relevant subgroup(s), if applicable). In some embodiments, this may include storing the trained ML model in the memory 124 of computer 120 for subsequent use.

[0045] Following either blocks 216 or 218, in block 220, method 200 includes determining whether all clinical conditions have been tested. If so, the method 200 ends at block 222; otherwise, method 200 returns to block 204 to select another clinical condition for ML modeling (repeating steps 204-220 as described above).

[0046] Each trained ML model that satisfies the performance metrics for identifying a clinical condition may be deployed for use in clinical diagnosis (for any relevant subgroup(s), if applicable). Thus, method 200 may be employed to create a portfolio of low-risk high- performance, ML-based CDS solutions adapted to the specific needs of a hospital, primary care setting or a laboratory.

[0047] FIG. 3 illustrates a flowchart of an additional method 300 of identifying clinical conditions suitable for categorization with machine learning models in accordance with one or more embodiments. With reference to FIG. 3, in block 302, method 300 includes obtaining, via a computer such as computer 120 (FIG. 1 B), a list of clinical conditions (e.g., a list of ICD codes).

[0048] In block 304, method 300 includes obtaining, via the computer, a dataset of biomarkers for the clinical condition. As stated, in some embodiments, a single biomarker dataset may be used for multiple clinical conditions while in other embodiments, multiple biomarker datasets may be employed (e.g., effectively treating the multiple biomarker datasets as an overall dataset for the clinical conditions).

[0049] In block 306, method 300 includes determining, via the computer, if a prevalence of a clinical condition within the dataset is within an expected range (e.g, by employing the computer, such as computer 120 of FIG. 1B, to search a database or table of prevalence range values stored in memory 124 or another location).

[0050] In block 308, if the prevalence of the clinical condition within the dataset is not within the expected range, method 300 proceeds to block 310 where it is determined whether the prevalence of the clinical condition is within an acceptable range (despite not being within the expected range). If no error is found, and the prevalence of the clinical condition is higher than expected, the biomarker dataset may still be useable for training an ML model for identifying the clinical condition (as described below with reference to block 312). However, if no error is found and the prevalence of the clinical condition is lower than expected, the biomarker dataset may not be useable for training an ML model for identifying the clinical condition. In such cases, the clinical condition may be identified as being not suitable for ML modeling for the biomarker dataset (as described below with reference to block 320).

[0051] Returning to block 308, if the prevalence of the clinical condition within the biomarker dataset is within the expected range, in block 312, method 300 includes performing ML training and validation on an ML model using the biomarker dataset as input and the clinical condition as the target output (e.g., using the presence and absence of the clinical condition as outputs). Following training, the ML model may be analyzed for feature importance and / or validated. Example feature importance and validation techniques are described previously with reference to method 200.

[0052] Following training and validation, in block 314, method 200 includes optionally performing, via the computer, analysis to identify relevant subgroups for the trained ML. In some embodiments, computer 120 of FIG. 1 B may perform such subgroup analysis. Example subgroups include age, gender, locality, etc.

[0053] Following subgroup analysis, in block 316, method 300 includes determining, via the computer (e.g., computer 120 of FIG. 1 B), if one or more performance metrics for ML modeling to identify the clinical condition have been met in general and / or for any identified relevant subgroup(s) if the ML model is subgroup specific (e.g., employing the computer to search a database or table of performance metrics). Performance metrics may include how well the trained ML model identifies the clinical condition on which it was trained.

[0054] If performance metrics have not been met (as determined in block 316), in block 318 method 300 proceeds to block 320 wherein the clinical condition is identified as not suitable for ML modeling. However, if performance metrics are met by the trained ML model for the clinical condition (in general and / or for any relevant subgroup(s)), in block 322, method 300 includes identifying, via the computer, the trained ML model as suitable for identifying the clinical condition (e.g., for any identified relevant subgroup(s) if the ML model is subgroup specific). In some embodiments, this may include storing the trained ML model in the memory 124 of computer 120 for subsequent use.

[0055] Following either blocks 320 or 322, in block 324, method 300 includes determining whether all clinical conditions have been tested. If so, the method 300 ends at block 326; otherwise, method 300 proceeds to block 328 wherein the next clinical condition is selected for ML modeling. Method 300 then returns to blocks 306-324 (as described above). Any trained ML model identified as suitable for identifying a clinical condition may be stored for use in a clinical setting.

[0056] In some embodiments, the dataset of biomarkers for each clinical condition may include biomarker values from blood tests. For example, the biomarkers (e.g., biomarker values) may be determined from blood tests performed at a hospital, laboratory, or other clinical setting. A dataset may include biomarkers from other tests such as tests of urine, stool, sputum, cerebrospinal fluid, another body fluid, etc. In one or more embodiments, a dataset of biomarkers may include biomarkers in a biomarker panel. Example biomarker panels include biomarkers from at least one of a lipid panel, a CBC panel, a BMP, a CMP, a thyroid panel, a urine panel, and an immunoassay panel. In some embodiments, the biomarker panel may include biomarkers from routine diagnostic tests such as biomarkers from a lipid panel, a CBC panel, and a CMP.

[0057] In one or more embodiments, the expected range for the prevalence of each clinical condition may be user selectable. Further, in some embodiments, for each clinical condition, performing ML training and validation may include training and validating aplurality of ML models. This may include, for example, at least one of training ML models using different hyperparameters and training different types of ML models.

[0058] In one or more embodiments, for each clinical condition, performing analysis to identify a relevant subgroup for the trained ML model may include identifying subgroups based on at least one of categorial variables and numerical variables.

[0059] In some embodiments, a user of methods 200 or 300 need only input the clinical conditions of interest to received trained and validated ML models that can identify specific clinical conditions (e.g., generally and / or for subgroups identified by method 200 or 300). In one or more embodiments, a user may customize clinical conditions examined, the expected prevalence ranges, the type or thresholds of performance metrics used, etc.

[0060] In some embodiments, computer 120 may present a user interface (not shown) that allows a user to upload data (e.g., a list of clinical conditions to check, one or more biomarker datasets, etc.). Computer 120, employing the methods described herein (e.g., methods 200 or 300), may then provide trained ML models for identifying one or more of the clinical conditions, model performance information, what subgroups should use the trained models, and / or the like. Thus, in some embodiments, an institution may define its own list of clinical conditions to explore. Further, models may be developed on a specific biomarker dataset of the institution (e.g., a hospital, laboratory, etc ).

[0061] In one or more embodiments described herein, one or more ML models may be trained on data relevant to a specific population. A full catalogue of diseases may be analyzed for ML modelling (e.g., based on one or more provided datasets).

[0062] In some embodiments, a single dataset for all clinical conditions and biomarkers of a hospital or clinical laboratory may be employed. The methods described herein, such as methods 200 and 300, may be used to examine the clinical conditions to determine which are suitable for ML modeling. ML models may be trained and validated, relevant subgroups may be identified, and the trained ML models may be deployed for use within the hospital or clinical laboratory.

[0063] In some embodiments, the trained ML models described herein may be employed to predict and / or detect diseases, including presence / absence of disease, disease states, and severity.

[0064] The foregoing description discloses only example embodiments of the invention. Modifications of the above disclosed apparatus and methods which fall within the scope of the invention will be readily apparent to those of ordinary skill in the art.

[0065] Accordingly, while the present invention has been disclosed in connection with example embodiments thereof, it should be understood that other embodiments may fall within the spirit and scope of the invention, as defined by the following claims.NON-LIMITING ILLUSTRATIVE EMBODIMENTS

[0066] The following is a list of non-limiting illustrative embodiments disclosed herein:

[0067] Illustrative embodiment 1. A method of identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: obtaining, via a computer, a list of clinical conditions; obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions; and for at least one clinical condition: performing ML training and validation on an ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition.

[0068] Illustrative embodiment 2. The method of illustrative embodiment 1 further comprising, for each trained ML model identified as suitable for identifying a clinical condition, deploying the trained ML model in a clinical setting that provided the dataset of biomarkers.

[0069] Illustrative embodiment 3. The method according to one of the preceding embodiments further comprising employing each deployed trained ML model to identify the clinical condition on which it was trained.

[0070] Illustrative embodiment 4. The method according to one of the preceding embodiments wherein the list of clinical conditions comprises a list of international classification of disease (ICD) codes.

[0071] Illustrative embodiment 5. The method according to one of the preceding embodiments wherein the dataset of biomarkers for each clinical condition comprises biomarker values from blood tests.

[0072] Illustrative embodiment 6. The method according to one of the preceding embodiments wherein the dataset of biomarkers is from a hospital or laboratory.

[0073] Illustrative embodiment 7. The method according to one of the preceding embodiments wherein the dataset of biomarkers comprises biomarkers in a biomarker panel.

[0074] Illustrative embodiment 8. The method according to one of the preceding embodiments wherein the biomarker panel comprises biomarkers from at least one of a lipid panel, a complete blood count (CBC) panel, a basic metabolic panel (BMP), a comprehensive metabolic panel (CMP), a thyroid panel, a urine panel, and an immunoassay panel.

[0075] Illustrative embodiment 9. The method according to one of the preceding embodiments wherein the biomarker panel comprises biomarkers from routine diagnostic tests.

[0076] Illustrative embodiment 10. The method according to one of the preceding embodiments wherein the routine diagnostic tests comprise biomarkers from a lipid panel, a complete blood count (CBC) panel, and a comprehensive metabolic panel (CMP).

[0077] Illustrative embodiment 11 . The method according to one of the preceding embodiments further comprising, for each clinical condition, determining, via the computer, if a prevalence of the clinical condition within the dataset is within an expected range by employing the computer to search a database or table of prevalence range values.

[0078] Illustrative embodiment 12. The method according to one of the preceding embodiments wherein the expected range for the prevalence of each clinical condition is user selectable.

[0079] Illustrative embodiment 13. The method according to one of the preceding embodiments wherein, for each clinical condition, performing ML training and validation comprises training and validating a plurality of ML models.

[0080] Illustrative embodiment 14. The method according to one of the preceding embodiments wherein, for each clinical condition, training and validating a plurality of ML models comprises at least one of training ML models using different hyperparameters and training different types of ML models.

[0081] Illustrative embodiment 15. The method according to one of the preceding embodiments further comprising, for each trained ML model: performing, via the computer, analysis to identify a relevant subgroup for the trained ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model for the identified relevant subgroup; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup.

[0082] Illustrative embodiment 16. The method according to one of the preceding embodiments wherein performing, via the computer, analysis to identify a relevant subgroup for the trained ML model comprises identifying subgroups based on at least one of categorial variables and numerical variables.

[0083] Illustrative embodiment 17. The method according to one of the preceding embodiments wherein determining, via the computer, if one or more performance metrics are met by the trained ML model for the identified relevant subgroup comprises employing the computer to search a database or table of performance metrics.

[0084] Illustrative embodiment 18. The method according to one of the preceding embodiments wherein, for each clinical condition, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup comprises storing the trained ML model for use during clinical condition identification.

[0085] Illustrative embodiment 19. The method according to one of the preceding embodiments further comprising, for each trained ML model, if the one or more performance metrics are not met for a trained ML model, identifying at least one subgroup and retraining the ML model for the at least one subgroup.

[0086] Illustrative embodiment 20. A system for identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: a processor; and a memory coupled to the processor, the memory including computer program instructions stored in the memory that, when executed by the processor, cause the processor to: obtain a list of clinical conditions; obtain a dataset of biomarkers for the list of clinical conditions; and for at least one clinical condition: initiate ML training and validation on an ML model; determine if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition.

[0087] Illustrative embodiment 21 . The system according to one of the preceding embodiments wherein the list of clinical conditions comprises a list of international classification of disease (ICD) codes.

[0088] Illustrative embodiment 22. The system according to one of the preceding embodiments further comprising computer program instructions stored in the memory that, when executed by the processor, cause the processor to, for each clinical condition, store for use each trained ML model identified as suitable for identifying the clinical condition.

[0089] Illustrative embodiment 23. The system according to one of the preceding embodiments further comprising computer program instructions stored in the memory that, when executed by the processor, cause the processor to: perform analysis to identify a relevant subgroup for a trained ML model; determine if one or more performance metrics are met by the trained ML model for the identified relevant subgroup; and if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup.

[0090] Illustrative embodiment 24. A method of identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: obtaining, via a computer, a list of clinical conditions of interest to an institution; obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions, the dataset of biomarkers determined in a clinical setting of the institution; and for each clinical condition: performing ML training and validation on an ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition at the institution.

[0091] Illustrative embodiment 25. The method according to one of the preceding embodiments further comprising, for each trained ML model identified as suitable for identifying a clinical condition, deploying the trained ML model in the clinical setting that provided the dataset of biomarkers.

[0092] Illustrative embodiment 26. A method of identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: obtaining, via a computer, a list of clinical conditions; obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions; and for each clinical condition: determining, via the computer, if a prevalence of the clinical condition within the dataset is within an expected range; and if the prevalence of the clinical condition within the dataset is within the expected range: performing ML training and validation on an ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition.

[0093] Illustrative embodiment 27. The method according to one of the preceding embodiments further comprising, for each trained ML model identified as suitable for identifying a clinical condition, deploying the trained ML model in a clinical setting that provided the dataset of biomarkers.

[0094] Illustrative embodiment 28. The method according to one of the preceding embodiments further comprising employing each deployed trained ML model to identify the clinical condition on which it was trained.

[0095] Illustrative embodiment 29. The method according to one of the preceding embodiments further comprising, for each trained ML model: performing, via the computer, analysis to identify a relevant subgroup for the trained ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model for the identified relevant subgroup; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup.

[0096] Illustrative embodiment 30. The method according to one of the preceding embodiments further comprising, for each trained ML model, if the one or more performance metrics are not met for a trained ML model, identifying at least one subgroup and retraining the ML model for the at least one subgroup.

[0097] Illustrative embodiment 31 . A system for identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: a processor; and a memory coupled to the processor, the memory including computer program instructions stored in the memory that, when executed by the processor, cause the processor to: obtain a list of clinical conditions; obtain a dataset of biomarkers for the list of clinical conditions;and for each clinical condition: determine if a prevalence of the clinical condition within the dataset is within an expected range; and if the prevalence of the clinical condition within the dataset is within the expected range: initiate ML training and validation on an ML model; determine if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition.

[0098] The foregoing description discloses only example embodiments of the invention. Modifications of the above disclosed apparatus and methods which fall within the scope of the invention will be readily apparent to those of ordinary skill in the art.

[0099] Accordingly, while the present invention has been disclosed in connection with example embodiments thereof, it should be understood that other embodiments may fall within the spirit and scope of the invention, as defined by the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method of identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: obtaining, via a computer, a list of clinical conditions; obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions; and for at least one clinical condition: performing ML training and validation on an ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition.

2. The method of claim 1 further comprising, for each trained ML model identified as suitable for identifying a clinical condition, deploying the trained ML model in a clinical setting that provided the dataset of biomarkers.

3. The method of claim 3 further comprising employing each deployed trained ML model to identify the clinical condition on which it was trained.

4. The method of claim 1 wherein the list of clinical conditions comprises a list of international classification of disease (ICD) codes.

5. The method of claim 1 wherein the dataset of biomarkers for each clinical condition comprises biomarker values from blood tests.

6. The method of claim 1 wherein the dataset of biomarkers is from a hospital or laboratory.

7. The method of claim 6 wherein the dataset of biomarkers comprises biomarkers in a biomarker panel.

8. The method of claim 7 wherein the biomarker panel comprises biomarkers from at least one of a lipid panel, a complete blood count (CBC) panel, a basic metabolic panel (BMP), a comprehensive metabolic panel (CMP), a thyroid panel, a urine panel, and an immunoassay panel.

9. The method of claim 7 wherein the biomarker panel comprises biomarkers from routine diagnostic tests.

10. The method of claim 9 wherein the routine diagnostic tests comprise biomarkers from a lipid panel, a complete blood count (CBC) panel, and a comprehensive metabolic panel (CMP).11 . The method of claim 1 further comprising, for each clinical condition, determining, via the computer, if a prevalence of the clinical condition within the dataset is within an expected range by employing the computer to search a database or table of prevalence range values.

12. The method of claim 11 wherein the expected range for the prevalence of each clinical condition is user selectable.

13. The method of claim 1 wherein, for each clinical condition, performing ML training and validation comprises training and validating a plurality of ML models.

14. The method of claim 13 wherein, for each clinical condition, training and validating a plurality of ML models comprises at least one of training ML models using different hyperparameters and training different types of ML models.

15. The method of claim 1 further comprising, for each trained ML model: performing, via the computer, analysis to identify a relevant subgroup for the trained ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model for the identified relevant subgroup; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup.

16. The method of claim 2 wherein performing, via the computer, analysis to identify a relevant subgroup for the trained ML model comprises identifying subgroups based on at least one of categorial variables and numerical variables.

17. The method of claim 2 wherein determining, via the computer, if one or more performance metrics are met by the trained ML model for the identified relevant subgroup comprises employing the computer to search a database or table of performance metrics.

18. The method of claim 2 wherein, for each clinical condition, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup comprises storing the trained ML model for use during clinical condition identification.

19. The method of claim 1 further comprising, for each trained ML model, if the one or more performance metrics are not met for a trained ML model, identifying at least one subgroup and retraining the ML model for the at least one subgroup.

20. A system for identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: a processor; and a memory coupled to the processor, the memory including computer program instructions stored in the memory that, when executed by the processor, cause the processor to: obtain a list of clinical conditions; obtain a dataset of biomarkers for the list of clinical conditions; and for at least one clinical condition: initiate ML training and validation on an ML model; determine if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition.21 . The system of claim 20 wherein the list of clinical conditions comprises a list of international classification of disease (ICD) codes.

22. The system of claim 20 further comprising computer program instructions stored in the memory that, when executed by the processor, cause the processor to, for each clinical condition, store for use each trained ML model identified as suitable for identifying the clinical condition.

23. The system of claim 20 further comprising computer program instructions stored in the memory that, when executed by the processor, cause the processor to: perform analysis to identify a relevant subgroup for a trained ML model; determine if one or more performance metrics are met by the trained ML model for the identified relevant subgroup; and if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup.

24. A method of identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: obtaining, via a computer, a list of clinical conditions of interest to an institution; obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions, the dataset of biomarkers determined in a clinical setting of the institution; and for each clinical condition: performing ML training and validation on an ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition at the institution.

25. The method of claim 24 further comprising, for each trained ML model identified as suitable for identifying a clinical condition, deploying the trained ML model in the clinical setting that provided the dataset of biomarkers.

26. A method of identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: obtaining, via a computer, a list of clinical conditions; obtaining, via the computer, a dataset of biomarkers for the list of clinical conditions; and for each clinical condition: determining, via the computer, if a prevalence of the clinical condition within the dataset is within an expected range; and if the prevalence of the clinical condition within the dataset is within the expected range: performing ML training and validation on an ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model; andif the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition.

27. The method of claim 26 further comprising, for each trained ML model identified as suitable for identifying a clinical condition, deploying the trained ML model in a clinical setting that provided the dataset of biomarkers.

28. The method of claim 27 further comprising employing each deployed trained ML model to identify the clinical condition on which it was trained.

29. The method of claim 26 further comprising, for each trained ML model: performing, via the computer, analysis to identify a relevant subgroup for the trained ML model; determining, via the computer, if one or more performance metrics are met by the trained ML model for the identified relevant subgroup; and if the one or more performance metrics are met, identifying, via the computer, the trained ML model as suitable for identifying the clinical condition for the identified relevant subgroup.

30. The method of claim 26 further comprising, for each trained ML model, if the one or more performance metrics are not met for a trained ML model, identifying at least one subgroup and retraining the ML model for the at least one subgroup.31 . A system for identifying clinical conditions suitable for categorization with machine learning (ML) models, comprising: a processor; and a memory coupled to the processor, the memory including computer program instructions stored in the memory that, when executed by the processor, cause the processor to: obtain a list of clinical conditions; obtain a dataset of biomarkers for the list of clinical conditions; and for each clinical condition: determine if a prevalence of the clinical condition within the dataset is within an expected range; and if the prevalence of the clinical condition within the dataset is within the expected range: initiate ML training and validation on an ML model;determine if one or more performance metrics are met by the trained ML model; and if the one or more performance metrics are met, identify the trained ML model as suitable for identifying the clinical condition.

Images