Method and system for disease phenotyping using multimodal ehr data and weak supervision

Abstract

To provide a method, a system and software for training a model for phenotyping subjects with respect to a medical condition.SOLUTION: A method includes generating a plurality of labeling functions with respect to a medical condition. Each of labeling function within the plurality of labeling functions includes comprises a corresponding set of one or more criteria that, when satisfied, indicate presence or absence of the medical condition. The method also includes assigning, to each medical record in a first plurality of medical records, a corresponding label indicating a status of the medical condition by evaluating first information from the respective medical record using an ensemble model comprising the plurality of labeling functions to obtain as output from the ensemble model prediction for the status of the medical condition, wherein the evaluating comprises natural language processing of at least a portion of each medical record.SELECTED DRAWING: Figure 2E

Description

[Background technology]

[0001] The near-universal use of electronic health records (EHRs) in academic, clinical, and healthcare organizations has the potential to generate large data sets for studying medical conditions and their treatments across large patient populations. Such EHRs may store large amounts of patient-specific data across multiple medical visits, including large data sets from gene sequencing and digital imaging studies. Despite their potential as a valuable resource for large patient population studies and clinical decision-making, the quality and interpretability of the data contained in EHRs can be inconsistent. For example, EHRs may contain clinical and research data that is stored across multiple systems in various formats and structures, or recorded with varying levels of accuracy, completeness, and depth depending on the source of acquisition. An analysis of 10 years of EHR data from a representative clinical data provider reported incompleteness ranging from 6% to 46% for several study variables related to pancreatic cancer. Similar completeness issues were observed in EHR data related to clinical trial recruitment. See, e.g., Coorevits et al., “Electronic health records: new opportunities for clinical research.” J Intern Med. 2013;274(6):547-560, which is incorporated by reference in its entirety.

[0002] Furthermore, to conduct studies across large patient populations, the information contained in the EHR must be extracted and put into context. For example, supervised machine learning of models that can identify patient subpopulations and / or predict treatment outcomes based on EHR data requires labeled training data. Manual phenotyping of EHRs is labor-intensive and therefore impractical for the record scale required for population-wide studies. Several electronic methods have been proposed for EHR phenotyping, such as rapid phenotyping using structured medical billing codes and rule-based phenotyping, which requires confirmation of structured medical billing codes with secondary indicators of medical conditions. However, these approaches have limited accuracy. See, for example, Pendergrass and Crawford, “Using Electronic Health Records to Generate Phenotypes For Research,” Curr Protoc Hum Genet., 100(1):e80 (2019), which is incorporated herein by reference in its entirety. For example, billing codes do not always accurately reflect disease states and / or may be used when clinicians refer patients to specialists for further evaluation prior to diagnosis. Moreover, rule-based phenotyping that requires validation of such billing codes is rigorous and requires that a patient have specific testing performed. Furthermore, these phenotyping approaches do not allow for the identification of patients with undiagnosed medical conditions, and in fact obfuscate model training by labeling such patients as negative for the medical condition they have. Summary of the Invention

[0003] As explained above, EHRs hold potential for large-scale research of medical disorders and treatments across populations. However, there is a need in the art for improved systems and methods for phenotyping patients from EHRs. Advantageously, the present disclosure provides, in some aspects, methods and systems for phenotyping EHRs for one or more medical conditions using an ensemble of indicator functions that evaluate multi-modal data contained within the EHR.

[0004] In one aspect, the present disclosure provides a method, and a system and a non-transitory computer-readable medium for implementing such a method, for training a model for phenotyping a subject for a medical condition. The method is implemented in a computer system including one or more processors and a memory. The method includes generating a plurality of indicator functions for the medical condition. Each respective indicator function in the plurality of indicator functions includes a corresponding set of one or more criteria, which indicates the presence or absence of the medical condition when the corresponding set of the one or more criteria is met. The method also includes: assigning a corresponding indicator to each medical record indicative of the status of the medical condition by evaluating first information from each respective medical record in the first plurality of medical records using the ensemble model to obtain as an output a prediction of the status of the medical condition from an ensemble model including the plurality of indicator functions, wherein the evaluating includes natural language processing of at least a portion of the respective medical records.

[0005] In one aspect, the present disclosure provides a method, system, and non-transitory computer-readable medium for training a model for phenotyping a subject for a medical condition. The method is implemented in a computer system including one or more processors and a memory. The method includes generating a plurality of indicator functions for the medical condition, each respective indicator function in the plurality of indicator functions including a corresponding set of one or more criteria, the corresponding set of one or more criteria indicating the presence of the medical condition when met. The method includes assigning a corresponding indicator to each subject indicative of the status of the medical condition by evaluating first information from a corresponding medical record of each respective subject in the first plurality of subjects using the ensemble model to obtain as an output a prediction of the status of the medical condition from an ensemble model including the plurality of indicator functions, the evaluating including natural language processing of at least a portion of the respective medical record.

[0006] In some embodiments, the method also includes filtering the second plurality of medical records according to the first set of features of the medical condition prior to generating the plurality of indicator functions, wherein the filtering includes applying an extract filter to the second plurality of medical records to obtain the first plurality of medical records, wherein the first plurality of medical records is a subset of the second plurality of medical records.

[0007] In some embodiments, the method also includes, for each respective medical record in the subset of the first plurality of medical records, training a classification model to phenotype the subject for the medical condition using (i) the corresponding indicator as a dependent variable, and (ii) second information from the respective medical record as an independent variable.

[0008] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, in which only illustrative embodiments of the disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modification in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive. [Brief description of the drawings]

[0009] [Figure 1A] 1 collectively illustrate block diagrams of an exemplary computing device for training a model for phenotyping a subject for a medical condition, in accordance with some embodiments of the present disclosure. [Figure 1B] 1 collectively illustrate block diagrams of an exemplary computing device for training a model for phenotyping a subject for a medical condition, in accordance with some embodiments of the present disclosure. [Figure 1C] 1 collectively illustrate block diagrams of an exemplary computing device for training a model for phenotyping a subject for a medical condition, in accordance with some embodiments of the present disclosure. [Figure 1D] 1 collectively illustrate block diagrams of an exemplary computing device for training a model for phenotyping a subject for a medical condition, in accordance with some embodiments of the present disclosure. [Figure 2A]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Figure 2B]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Figure 2C]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Figure 2D]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Figure 2E]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Figure 2F]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Figure 2G]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Figure 2H]

[0023] FIG. 1 collectively provides a flowchart of processes and features for training a model for phenotyping a subject for a medical condition, according to several embodiments of the present disclosure, where dashed boxes represent optional portions of the method. [Diagram 3] FIG. 1 illustrates an exemplary schematic for training a model for phenotyping a subject for a medical condition, according to some embodiments of the present disclosure. [Figure 4A] 1 , collectively illustrate representation of target patient populations within a health record database and efforts to identify the populations through various search strategies, according to several embodiments of the present disclosure. [Figure 4B] 1 , collectively illustrate representation of target patient populations within a health record database and efforts to identify the populations through various search strategies, according to several embodiments of the present disclosure. [Figure 4C] 1 , collectively illustrate representation of target patient populations within a health record database and efforts to identify the populations through various search strategies, according to several embodiments of the present disclosure. [Figure 4D] 1 , collectively illustrate representation of target patient populations within a health record database and efforts to identify the populations through various search strategies, according to several embodiments of the present disclosure. [Figure 4E] 1 , collectively illustrate representation of target patient populations within a health record database and efforts to identify the populations through various search strategies, according to several embodiments of the present disclosure. [Figure 4F] 1 , collectively illustrate representation of target patient populations within a health record database and efforts to identify the populations through various search strategies, according to several embodiments of the present disclosure. [Figure 4G] 1 , collectively illustrate representation of target patient populations within a health record database and efforts to identify the populations through various search strategies, according to several embodiments of the present disclosure. [Diagram 5] 1 illustrates data from clinical and electronic review of patient records to identify cases of pulmonary hypertension, according to some embodiments of the present disclosure. [Figure 6A] 1 illustrates a majority probability graphical modeling (PGM) model, according to some embodiments of the present disclosure. [Figure 6B] 1 illustrates a generalized majority voting PGM model according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Introduction. Electronic health records (EHRs) provide a useful resource for personalized health data by combining large amounts of patient-specific data collected during the delivery of health care services. For example, EHRs may include one or more of clinical, demographic, administrative, billing (e.g., medical and pharmacy), and patient-centered (e.g., vital statistics or quality of life information derived from medical instruments or caregiver assessments) data. Examples of data types that may be included in EHRs include medication history (e.g., current prescriptions, concomitant medications, drug classes, drug codes, and / or ontology terms), disease status (e.g., pre-existing conditions, comorbidities, symptoms, diagnosis, and / or prognosis), laboratory test results or clinical data (e.g., biomarkers, genomic variants, medical images, and / or sequencing data), and free text observations and other notes (e.g., by clinicians). In some cases, EHRs include longitudinal data, such as information collected over multiple visits to a health care provider or over a period of time. See, e.g., Coorevits et al., "Electronic health records: new opportunities for clinical research." J Intern Med. 2013;274(6):547-560, Cowie et al. "Electronic health records to facilitate clinical research." Clin Res Cardiol. 2017;106(1):1-9, and Xiao et al., "Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review." J Am Med Inform Assoc. 2018;25(10):1419-1428, which are incorporated by reference in their entireties.

[0011] Existing EHR phenotypes commonly found in observational clinical studies can vary widely in methodology and quality and often suffer from several limitations. Many studies utilize diagnosis codes such as ICD-9, ICD-10, or SNOMED codes without validating the quality of these codes to determine how representative they are of their desired cohort of interest. Studies utilizing claims data have limited ability to perform detailed chart review or manual abstraction for the development of phenotype or code-based cohort definitions, but some population and cohort level metrics can be used in aggregate to confirm that the cohort appears roughly appropriate from a clinical perspective. Unfortunately, many studies do not perform any kind of cohort characterization or chart review validation and blindly trust that codes developed primarily for non-research purposes (most often for claims purposes, with research as a secondary use) accurately represent what the code descriptions should represent. However, contemporary coding in EHR systems is messy, incomplete, and often inaccurate, sometimes by design, sometimes due to changes in a patient's clinical status, provisional diagnoses as they progress along a diagnostic pathway, or sometimes due to inadvertence resulting from busy clinical schedules, or data entry or mapping errors. Thus, codes for either common diseases or diseases associated with high levels of insurance reimbursement may be accurately captured, but often have pitfalls and shortcomings. The same problems apply to procedure codes, problem lists (some institutions update them regularly and some ignore them entirely), medications, and most other sources of EHR data. Based on practice patterns or geography, the codes to best capture the same disease can vary widely, limiting the generalizability of code-based phenotypes.

[0012] Conventional methods for EHR data analysis are further limited by the lack of labels in many EHR databases. Labels provide substantial benefits to model training by providing "ground truth" indicators for clinical outcomes or phenotypes that can be used to enhance the model's ability to accurately generate desired outputs. Such labels are often not consistently recorded in EHR data and therefore typically unavailable for model training. Manual labeling of EHRs requires significant resources and expense because it relies on trained experts to read, understand, and accurately classify the information in each record according to each desired analytical goal. In addition, it is very labor- and time-intensive, especially for large databases of training EHRs (EHR databases may contain millions, tens of millions, or hundreds of millions of individual records). Some conventional approaches utilize supervised learning based on the detection of medical codes (e.g., diagnosis, procedure, or drug codes) to obtain EHR labels, but such methods do not easily generalize to all records, such as those that do not fit the coding standards of a particular institution or organization. See, e.g., Xiao et al., “Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review.” J Am Med Inform Assoc. 2018;25(10):1419-1428, which is incorporated by reference in its entirety.

[0013] Various sources of patient information can be used to construct structured phenotypes for determining a patient's PH status. For example, information available in medical claims and / or medical billing databases can indicate whether a patient has PH. Advantageously, these databases are large, easily accessible, and provide information on diverse national or international populations. However, the data can be difficult to validate, do not include drug data such as OTC drugs, and often provide biased patient samples, e.g., patients with certain insurance.

[0014] Code-based structured phenotypes can also be derived from electronic health records (EHRs). Such phenotypes can be developed relatively quickly because EHR codes have defined correspondences and are relatively simple to validate through clinician review. However, code-based structured phenotypes are more rigid, lack generalizability (e.g., due to significant variation in code usage and / or clinical practice across different geographic regions and even from site to site), and provide a view of the patient that is limited to field codes.

[0015] Thus, there is a need for improved methods and systems for phenotyping medical records. Advantageously, the present disclosure addresses these and other needs by providing methods and systems for constructing and validating disease phenotypes using multimodal EHR data and weakly supervised learning. In some embodiments, the methods and systems described herein use signal aggregation together with natural language processing (NLP) and machine learning to better identify all medical condition subgroups of interest with more specific boundaries.

[0016] Many medical conditions have a relatively low prevalence in the general population and will be represented by different subsets of multimodal data in the EHR due to different underlying disease etiologies, variability in disease symptoms, and discrepancies in medical records as described above. This is illustrated in FIG. 4A, where multiple EHRs are represented as a function of the information stored in each EHR indicating whether a subject has experienced a medical condition. EHRs from patients who have experienced a medical condition are shown as darker shapes according to the representation. The use of conventional code-based structured phenotypes, shown by dashed ellipses, to identify patients who have experienced a medical condition may initially capture one or more groups quite well, but may be strict and result in false positives, i.e., identification of EHRs within the ellipses that are not part of the darker shapes, and may miss the identification of other cases of the medical condition, i.e., darker shapes that are not enclosed by the dashed ellipses. In this analogy, adding or subtracting diagnostic codes may expand or contract the ellipse, with a concomitant trade-off in terms of PPV or sensitivity of such action, but it may not be possible to customize or reshape the boundaries of this cohort to best fit the actual boundaries of the cases themselves.

[0017] One way to improve upon these rather rigid constraints is to add additional sources, tables, or modalities of data, such as adding medications, procedures, lab values, images, or notes to the initial diagnosis code-based phenotype, as illustrated in FIG. 4B. These additional criteria may better fit or identify additional subgroups (additional darker shapes) that may not be identifiable by other types of data. However, by itself, each additional data type may still remain fairly rigid in the structured phenotype, essentially creating multiple overlapping and non-overlapping rigid ellipses of various sizes, but still unable to rigorously represent the exact unique cutoff or jagged boundary between cases and controls. In addition, there may be subgroups, such as those near the top of the figure, that may not be captured by the structured data elements at all. In short, using multiple modalities or types of data is useful, but in most cases significant limitations remain, especially for challenging, relatively rare, heterogeneous diseases such as pulmonary hypertension (PH), which has a prevalence of less than 1% in the population.

[0018] Rather than just integrating additional data elements with accompanying additional logic, the methods and systems described herein utilize signal aggregation together with NLP and machine learning models to better identify all disease subgroups of interest with more specific boundaries. In signal aggregation (FIG. 4C), multiple signals are defined from these multimodal data features using different labeling functions. Since no single final phenotype is generated from each of these multiple data elements, multiple useful rules can be created. In fact, the resulting weak labeling does not need to be complete, but rather considers multiple additional dotted ellipses in FIG. 4C.

[0019] These signals are then aggregated using statistical and technical methods to generate patient signatures that are more specifically tailored to the actual patient subgroups after the signal aggregation step, as seen by the dotted line shapes in FIG. 4D that are no longer constrained to be strictly circular. As described in more detail herein, signal aggregation represents an opportunity to generate higher performing subgroups, which can then be compared to all previous structured phenotypes based on chart review validation metrics to select the best performing model. However, despite signal aggregation, the final cohort may still improve on or still miss some subgroups that were not represented at all in the structured data elements and signature functions.

[0020] In some embodiments, the aggregation is achieved using a voting method, such as a majority voting method. That is, the indicator functions define different segments of the medical record that indicate the presence of a disease. Each indicator function is evaluated on the subject's medical record, and the result of the evaluation is used as a single vote that is aggregated together with the votes provided by all other indicator functions evaluated on the subject's medical record. In some embodiments, the aggregation is achieved, for example, by solving a probabilistic graph modeling (PGM) model that applies one or more priors associated with the performance of each individual indicator function to the results of the evaluation of each indicator on the subject's medical record. An example of the evaluation of such a PGM model is shown, for example, in Example 4. In some embodiments, the aggregation is achieved using a snorkel's probability model to aggregate the results of the evaluation of each indicator on the subject's medical record.

[0021] As explained above, the dashed shapes in Figure 4 (e.g., structured phenotypes 406) are abstract representations of phenotypes for medical conditions. When structured queries are used to identify these phenotypes, the identified cohorts are "rigid" (e.g., represented by regular shapes such as ovals in structured phenotypes 406). However, when more complex aggregation models are used to identify subjects having the phenotype, the identified signals become more precise, resulting in a more "soft" identification of subjects (e.g., as represented by the irregular shapes in Figure 4D resulting from the aggregation of indicator functions).

[0022] In some embodiments, further refinement of the phenotype is performed by training a machine learning model, such as a Transformer-based NLP model, from the aggregated weak labels after signal aggregation, or from the structured phenotype itself, as illustrated in Figure 4E. This machine learning model may then be able to identify signals or learn relationships in the unstructured clinical note EHR data, for example, that not only allows better identification of the boundary between cases and controls relative to what is represented in the structured data elements, but also can identify new subgroups, such as the top cohort of the figure, that were not previously represented by any structured data elements or label functions, as illustrated in Figure 4F.

[0023] To overcome these challenges, a method has been developed that integrates different structured phenotypes that can be identified from different medical data modalities to accurately label medical records. In some embodiments, the method begins with defining a patient population. Examples of considerations that can be used to describe the patient population include: (i) which clinical population / entity is being targeted; (ii) which research question and / or projected task is being addressed; (iii) what are the clinical and / or research use cases; and (iv) which patients should be specifically included or excluded, and over what time period. For example, a target population can be described as (adult and / or adolescent) patients with X medical condition, optionally (having / having received) Y therapeutic intervention, and optionally with (with or without) a previous medical history of Z.

[0024] In some embodiments, the methods described herein include generating a plurality of indicator functions for the medical condition, each respective indicator function in the plurality of indicator functions including a corresponding set of one or more criteria that indicates the presence or absence of the medical condition when met. The method also includes assigning a corresponding indicator to each respective medical record indicative of a status of the medical condition by evaluating first information from each respective medical record in the first plurality of medical records using an ensemble model to obtain as an output a prediction of a status of the medical condition from an ensemble model including the plurality of indicator functions, wherein the evaluating includes natural language processing of at least a portion of the respective medical records.

[0025] In some embodiments, the method also includes filtering the second plurality of medical records according to the first set of features of the medical condition prior to generating the plurality of indicator functions, wherein the filtering includes applying an extract filter to the second plurality of medical records to obtain the first plurality of medical records, wherein the first plurality of medical records is a subset of the second plurality of medical records.

[0026] In some embodiments, the method also includes, for each respective medical record in the subset of the first plurality of medical records, training a classification model to phenotype the subject for the medical condition using (i) the corresponding indicator as a dependent variable, and (ii) second information from the respective medical record as an independent variable.

[0027] Advantageously, the methods described herein are more robust than commonly used standard phenotyping techniques, which often rely on a limited number of codes, modalities, or data elements (such as focusing only on diagnostic codes or procedures). For example, many studies, especially early observational studies or studies that are less methodologically rigorous, use only a single ICD code or a limited number of codes. Such code-based structured phenotypes are overly rigorous and have a significant trade-off between capturing more cases (true positives) and limiting the number of controls (false positives). The methods and systems described herein create more flexible models to better characterize difficult and / or rare diseases. In some embodiments, this is achieved, at least in part, through the use of weakly supervised learning for model development, as further described below.

[0028] Exemplary implementation.

[0029] An illustrative implementation 300 of a method for training a model for phenotyping a subject for a medical condition according to some embodiments of the disclosure provided herein will now be described in conjunction with Figure 3. In Figure 3, a plurality of electronic health records (EHRs) 302 are partitioned into a training fold 304 and a validation fold 306. Although this partitioning is illustrated prior to filtering (354), indicator function construction (360), and weak labeling (364) in Figure 3, the partitioning into training and validation folds may occur at any time prior to model training (366).

[0030] As illustrated in FIG. 3 , the embodiment 300 includes an initial step of constructing (352) a filter and filtering (354) the medical records 302 to remove filtered records 312, thereby generating a subset of candidate training records 308 and a subset of candidate validation records 310. However, in some embodiments, no initial filtering is performed and the weak labeling method is performed using the entirety of the starting plurality of EHRs 302. In some embodiments, constructing (352) a filter includes developing a set of clinical logic to identify a target cohort or outcome of interest. In some embodiments, the set of clinical logic includes one or more first condition types, where the one or more first condition types indicate the presence of a target medical condition (e.g., a diagnosis or clinical outcome) when met. In some embodiments, the set of clinical logic includes one or more second condition types, where the one or more second condition types indicate the absence of a target medical condition when met. In some embodiments, the set of clinical logic includes both a first condition type and a second condition type. In some embodiments, the set of clinical logic includes only the first condition type.

[0031] In some embodiments, filtering is performed solely using clinical logic that includes conditions that, when met, indicate the presence of a medical condition, i.e., in some embodiments, filtering includes only inclusion criteria and not exclusion criteria.

[0032] In some embodiments, the one or more conditions used in the clinical logic are evaluated in or extracted from text in the electronic health record. For example, in some embodiments, a list of symptoms and / or related descriptions of medical conditions that may be included in the unstructured clinical notes section of the EHR is curated. The curated list may then be used to search the EHR (or a database populated from the clinical notes section of the EHR), for example, by NLP, such as regular expression searches, for indications that a subject has a medical condition. In some embodiments, the filter is further refined by constructing exclusion terms. In some embodiments, these exclusion terms are removed from the text before searching the text for the list of curated indications. In some embodiments, the exclusion terms are removed after being identified during the search of the text. For example, when searching for an indication of pulmonary hypertension, commonly abbreviated as PH, the term Ph.D. may be excluded from the search criteria.

[0033] The purpose of this filtering is to refine the subject population to a reasonable size while maintaining confidence that few cases of the medical condition in the starting database will be removed by the filter. Thus, in some embodiments, the focus of the filtering is to remove only those medical records for which there is a high degree of confidence that they do not contain cases of the target medical condition. Thus, in some embodiments, the filter reflects this by being as broad in scope as possible while still reducing the set of medical records to a reasonable size. It is therefore beneficial to include medical records for which it is unclear whether they contain cases of the medical condition in the filtered subset.

[0034] When implemented, filtering (354) reduces the total number of medical records that need to be processed, thereby generating weak labels, reducing the time required to train the phenotyping model, and / or enriching the set of medical records for cases of the target medical condition. In certain embodiments, enriching the training set for cases of the target medical condition reduces overfitting and allows the model to better generalize cases of the medical condition, especially when cases of the target medical condition are scarce in the population. Thus, in some embodiments, the methods described herein include filtering a plurality of medical records (e.g., EHR 302) according to a first set of features (e.g., criteria 52) for the medical condition. The filtering includes applying (354) an extraction filter (e.g., filter 51) to the plurality of medical records 302 to obtain a plurality of medical records (e.g., a subset of candidate training records 308 and a subset of candidate validation records 310).

[0035] Filtering also allows for random selection of patients (EHRs) from the filtered population that provide a reasonable number, e.g., at least 25%, of positive cases of the target medical condition. These randomly selected EHRs can be independently reviewed for use in evaluating labeling functions, evaluating / validating labeling ensemble models, validating downstream machine learning phenotyping models, etc. That is, filtering allows for robust validation of the phenotyping models described herein.

[0036] As illustrated in FIG. 3, in some embodiments, the embodiment 300 also includes evaluating (356) cases of the subset of candidate EHRs that created it through filtering (354), and optionally refining the filter when it is determined that more EHRs that do not include cases of the medical condition have passed through the filter than desired. For example, one or more filters may be modified and / or added to exclude EHRs when certain exclusion criteria are met, thereby allowing fewer EHRs to pass through filtering (354). Similarly, the embodiment 300 includes evaluating (356) cases of filtered EHRs 312 that have been removed from the plurality of EHRs 302 by filtering (354), and optionally refining the filter when it is determined that EHRs that include cases of the medical condition have been filtered out. For example, one or more filters may be modified and / or added to include EHRs when certain inclusion criteria are met, thereby allowing more EHRs to pass through filtering (354). In some embodiments, when a filter (e.g., filter 51) is refined according to evaluation (354 and / or 356), the plurality of EHRs 302 are filtered again (354) to update the set of EHRs (e.g., the subset of candidate training records 308 and the subset of candidate validation records 310) used to build and test the model.

[0037] In some embodiments, a machine learning model 316, e.g., trained based on weak labels output from the phenotyping pipeline, is used to evaluate (358) all or a portion of the filtered EHR 312. That is, in some embodiments, filter evaluation (358) and / or refinement (352) occurs after completion of the initial cases of the phenotyping pipeline 300. In some embodiments, the final model 316 evaluates all or a portion (e.g., at least 10%, at least 20%, at least 30%, at least 40%, at least 50%, at least 60%, at least 70%, at least 80%, at least 90%, or more) of the filtered results 312 to identify filtered records that are most likely to indicate an instance of the medical condition, e.g., using a model score. In some embodiments, the identified filtered records may be used to update (352) the filter, and the pipeline may be run again using the updated filter.

[0038] 3, embodiment 300 also includes constructing (360) a set of indicator functions (e.g., indicator function 61) and evaluating (362) whether application of the set of indicator functions results in accurate indicator of the EHR of one or more test cases, for example, in the subset of candidate training records 308. In some embodiments, the test cases of the EHR (e.g., the subset of candidate training records 308) are separately reviewed and determined to either indicate the presence of a medical condition or indicate the absence of a medical condition, which is used in the evaluation (362) to determine whether to refine one or more indicator functions (e.g., indicator function 61) in the set of indicator functions.

[0039] Building a set of indicator functions (360) is an iterative process, as illustrated in Figure 3. The process involves building phenotypes using multimodal data available in the EHR, such as diagnosis codes, procedures, problem lists, medications, encounter notes, demographics, orders, etc. This involves chart review of metrics, review, and iteration to continue developing improved versions of the structured phenotypes.

[0040] As illustrated in Figure 3, once a set of indicator functions (e.g., indicator function 61) is obtained, a combination of multiple sources of data in the set of indicator functions is used for signal aggregation to generate 364 weak indicators 47 for the dataset. That is, the indicator functions are applied to the EHRs (e.g., a subset of candidate training records 308) to label each EHR for model training, generating a labeled EHR 314. This process allows for the generation of phenotypes without the need for manual curation through the use of indicator functions and weakly supervised learning.

[0041] As illustrated in FIG. 3, the embodiment 300 also includes training 366 a model 316 using the signal aggregated weak indicators 47 as independent variables and using data from the EHR 41 as independent variables. The model may be evaluated using a subset 310 of candidate validation records. In some embodiments, the subset 310 of candidate validation records is separately reviewed and determined to either indicate the presence of a medical condition or indicate the absence of a medical condition to evaluate the performance of the model. In some embodiments, the subset 310 of candidate validation records is labeled using an ensemble model of indicator functions to evaluate the performance of the model. In some embodiments, if the performance of the model 316 is not sufficient, the set of indicator functions is further iterated 360 to improve the weak indicators 364. If the model performance is found to be sufficient, the results of the model may be used to label the EHR.

[0042] Definition. The terms used in this disclosure are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention. As used in the description of the invention and in the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. The term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises" and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, to the extent that the terms "including", "includes", "having", "has", "with" or variations thereof are used in either the detailed description and / or the claims, such terms are intended to be inclusive in the same manner as the term "comprising".

[0043] As used herein, the term "if" may be interpreted to mean "when" or "upon," "in response to detecting," or "in response to determining," depending on the context. Similarly, the phrases "when determined" or "when [a described condition or event] is detected" may be interpreted to mean "upon determining," or "in response to determining," or "upon detection [of the described condition or event]," or "in response to detecting [the described condition or event]," depending on the context.

[0044] Also, terms such as first, second, etc. are used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first subject may be referred to as a second subject, and similarly, a second subject may be referred to as a first subject, without departing from the scope of the present disclosure. A first subject and a second subject are both subjects, but are not the same subject. Furthermore, the terms "subject", "user", and "patient" are used interchangeably herein.

[0045] As used herein, the term "subject" refers to any living or non-living human. In some embodiments, the subject is a human being of any age, male or female (e.g., a man, a woman, or a child).

[0046] As used herein, the terms "mutation" and "variant" refer to detectable changes in the genetic material of one or more cells. In certain instances, one or more mutations may be found in and distinguished from cancer cells (e.g., driver mutations and passenger mutations). Mutations may be transmitted from parent cells to daughter cells. Those skilled in the art will understand that a genetic mutation in a parent cell (e.g., driver mutation) may induce additional, different mutations in daughter cells (e.g., passenger mutations). Mutations generally occur within nucleic acids. In certain instances, a mutation may be a detectable change in one or more deoxyribonucleic acids or fragments thereof. Mutations generally refer to nucleotides that are added, deleted, substituted, inverted, or transposed to a new position in a nucleic acid. In some embodiments, a variant refers to a change in the primary genomic sequence at a single or multiple nucleotide positions, such as a single nucleotide variant (SNV), multiple nucleotide variant (MNV), indel (e.g., insertion or deletion of a nucleotide), DNA rearrangement (e.g., inversion or translocation of a portion of a chromosome or multiple chromosomes), copy number variation of a locus (e.g., a large span of an exon, gene, or chromosome) (CNV), partial or complete change in the ploidy of a cell, as well as changes in the epigenetic information of a genome, such as altered DNA methylation patterns. A mutation can be a naturally occurring mutation or an experimentally induced mutation. A mutation in a sequence of a specific tissue is an example of a "tissue-specific allele." For example, a tumor may have a mutation that results in an allele at a locus that does not occur in a normal cell. Another example of a "tissue-specific allele" is a fetal-specific allele that occurs in fetal tissue but not in maternal tissue. As used herein, the term "single nucleotide variant" or "SNV" refers to a substitution of one nucleotide for a different nucleotide at a position (e.g., site) of a nucleotide sequence, e.g., a sequence read from an individual. A substitution of a first nucleobase X with a second nucleobase Y may, in some embodiments, be represented as "X>Y." For example, a cytosine to thymine SNV is, in some embodiments, represented as "C>T."

[0047] As used herein, the term "cancer," "cancer tissue," or "tumor" refers to an abnormal mass of tissue whose growth exceeds and is out of step with that of normal tissue. Cancer or tumors can be defined as "benign" and "malignant" depending on the following characteristics: degree of cell differentiation, including morphology and functionality, growth rate, local invasion, and metastasis. "Benign" tumors are well differentiated, have characteristically slower growth than malignant tumors, and may remain localized to the site of origin. In addition, in some cases, benign tumors do not have the ability to invade, invade, or metastasize to distant sites. "Malignant" tumors may be poorly differentiated (dysplastic) and have characteristically rapid growth with progressive infiltration, invasion, and destruction of surrounding tissues. Furthermore, malignant tumors may have the ability to metastasize to distant sites. Thus, cancer cells are cells found within an abnormal mass of tissue whose growth is out of step with that of normal tissue. Thus, a "tumor sample" refers to a biological sample obtained from or derived from a tumor of a subject, as described herein. Cancer tissue can refer to blood cells if the cancer is a hematologic (blood) cancer.

[0048] As used interchangeably herein, the terms "classifier" or "model" refer to a machine learning model or algorithm.

[0049] In some embodiments, the model includes an unsupervised learning algorithm. An example of an unsupervised learning algorithm is cluster analysis. In some embodiments, the model includes supervised machine learning. Non-limiting examples of supervised learning algorithms include, but are not limited to, logistic regression, neural networks, support vector machines, naive Bayes algorithms, nearest neighbor algorithms, random forest algorithms, decision tree algorithms, boosted tree algorithms, multinomial logistic regression algorithms, linear models, linear regression, gradient boosting, mixture models, hidden Markov models, Gaussian NB algorithms, linear discriminant analysis, or any combination thereof. In some embodiments, the model is a multinomial classifier algorithm. In some embodiments, the model is a two-stage stochastic gradient descent (SGD) model. In some embodiments, the model is a deep neural network (e.g., a deep extensive sample-level model).

[0050] Neural Network. In some embodiments, the model is a neural network (e.g., a convolutional neural network and / or a residual neural network). Neural network algorithms, also known as artificial neural networks (ANNs), include convolutional and / or residual neural network algorithms (deep learning algorithms). In some embodiments, the neural network is a machine learning algorithm that is trained to map an input data set to an output data set, where the neural network includes an interconnected group of nodes organized into multiple layers of nodes. For example, in some embodiments, the neural network architecture includes at least an input layer, one or more hidden layers, and an output layer. In some embodiments, the neural network includes any total number of layers and any number of hidden layers, where the hidden layers function as trainable feature extractors that allow for mapping a set of input data to an output value or set of output values. In some embodiments, the deep learning algorithm includes a neural network that includes multiple hidden layers, e.g., two or more hidden layers. In some cases, each layer of the neural network includes several nodes (or neurons). In some embodiments, a node receives inputs that come either directly from input data or from the outputs of nodes in a previous layer, and performs a particular operation, e.g., a summation operation. In some embodiments, the connections from the inputs to the nodes are associated with parameters (e.g., weights and / or weighting coefficients). In some embodiments, a node receives input x iand their associated parameters. In some embodiments, the weighted sum is offset by a bias b. In some embodiments, the output of a node or neuron is gated using a threshold or activation function f, which in some cases is a linear or non-linear function. In some embodiments, the activation function is, for example, a rectified linear unit (ReLU) activation function, a leaky ReLU activation function, or other function such as a saturated hyperbolic tangent function, an identity function, a binary step function, a logistic function, an arctangent function, a soft sine function, a parametric rectified linear unit function, an exponential linear unit function, a soft plus function, a bent identity function, a soft exponential function, a sinusoidal function, a sine function, a Gaussian function, or a sigmoid function, or any combination thereof.

[0051] In some implementations, weighting coefficients, bias values, and thresholds, or other computational parameters of the neural network are "taught" or "learned" in a training phase using one or more sets of training data. For example, in some implementations, the parameters are trained using input data from a training data set and gradient descent or backpropagation such that the output values ​​the ANN computes match the examples contained in the training data set. In some embodiments, the parameters are obtained from a backpropagation neural network training process.

[0052] Any of a variety of neural networks are suitable for use with the present disclosure. Examples include, but are not limited to, feed-forward neural networks, radial basis function networks, recurrent neural networks, residual neural networks, convolutional neural networks, residual convolutional neural networks, and the like, or any combination thereof. In some embodiments, the machine learning utilizes pre-trained and / or transfer-trained ANNs or deep learning architectures. In some implementations, convolutional and / or residual neural networks are used in accordance with the present disclosure.

[0053] For example, a deep neural network model includes an input layer, a number of individually parameterized (e.g., weighted) convolution layers, and an output scorer. Each parameter (e.g., weight) of the convolution layer and the input layer contributes to a number of parameters (e.g., weights) associated with the deep neural network model. In some embodiments, at least 100 parameters, at least 1000 parameters, at least 2000 parameters, or at least 5000 parameters are associated with the deep neural network model. Thus, deep neural network models cannot be solved by a human mind, and therefore a computer must be used. In other words, given the input to the model, the model output must be determined using a computer, not a human mind, in such embodiments. See, e.g., Krizhevsky et al., 2012, “Imagenet classification with deep convolutional neural networks,” Advances in Neural Information Processing Systems 2, Pereira, Burges, Bottou, Weinberger, eds., pp. 1097-1105, Curran Associates, Inc.; Zeiler, 2012, “ADADELTA: an adaptive learning rate method,” CoRR, vol. abs / 1212.5701; and Rumelhart et al., 1988, “Neurocomputing: Foundations of research,” ch. Learning Representations by Back-propagating Errors, pp. 696-699, Cambridge, MA, USA: MIT Press, each of which is incorporated herein by reference.

[0054] Neural network algorithms, including convolutional neural network algorithms, suitable for use as models are disclosed, for example, in Vincent et al., 2010, “Stacked denoising autoencoder: Learning useful representations in a deep network with a local denoising criterion,” J Mach Learn Res 11, pp. 3371-3408; Larochelle et al., 2009, “Exploring strategies for training deep neural networks,” J Mach Learn Res 10, pp. 1-40; and Hassoun, 1995, Fundamentals of Artificial Neural Networks, Massachusetts Institute of Technology, each of which is incorporated herein by reference. Additional exemplary neural networks suitable for use as models are disclosed in Duda et al., 2001, Pattern Classification, Second Edition, John Wiley & Sons, Inc., New York, and Hastie et al., 2001, The Elements of Statistical Learning, Springer-Verlag, New York, each of which is incorporated herein by reference in its entirety. Additional exemplary neural networks suitable for use as models are also described in Draghici, 2003, Data Analysis Tools for DNA Microarrays, Chapman & Hall / CRC, and Mount, 2001, Bioinformatics: sequence and genome analysis, Cold Spring Harbor Laboratory Press, Cold Spring Harbor, New York, each of which is incorporated herein by reference in its entirety.

[0055] Support Vector Machine. In some embodiments, the model is a support vector machine (SVM). SVM algorithms suitable for use as models are described, for example, in Cristianini and Shawe-Taylor, 2000, “An Introduction to Support Vector Machines,” Cambridge University Press, Cambridge; Boser et al., 1992, “A training algorithm for optimal margin classifiers,” Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, ACM Press, Pittsburgh, Pa., pp. 142-152; Vapnik, 1998, Statistical Learning Theory, Wiley, New York; Mount, 2001, Bioinformatics: sequence and genome analysis, Cold Spring Harbor Laboratory Press, Cold Spring Harbor, NY; Duda, Pattern Classification, Second Edition, 2001, John Wiley & Sons, Inc., pp. 259, 262-265; Hastie, 2001, The Elements of Statistical Classification, Vol. 1, No. 1, pp. 259-265; Learning, Springer, New York, and Furey et al., 2000, Bioinformatics 16, 906-914. When used for classification, SVMs separate a given set of binary labeled data with a hyperplane that is maximally far from the labeled data. In certain cases where linear separation is not possible, SVMs work in combination with the technique of "kernels" that automatically achieve a nonlinear mapping into the feature space. The hyperplane found by the SVM in the feature space corresponds in some cases to a nonlinear decision boundary in the input space.In some embodiments, multiple parameters (e.g., weights) associated with the SVM define a hyperplane. In some embodiments, the hyperplane is defined by at least 10, at least 20, at least 50, or at least 100 parameters, and the SVM model must be computed because it cannot be solved by the human mind.

[0056] Naive Bayes Algorithm. In some embodiments, the model is a naive Bayes algorithm. Naive Bayes models suitable for use as models are disclosed, for example, in Ng et al., 2002, “On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes,” Advances in Neural Information Processing Systems, 14, which is incorporated herein by reference. Naive Bayes models are any models in the family of “probabilistic models” based on applying Bayes’ theorem with the assumption of strong (naive) independence between features. In some embodiments, they are combined with kernel density estimation. See, for example, Hastie et al., 2001, The elements of statistical learning: data mining, inference, and prediction, eds. Tibshirani and Friedman, Springer, New York, which is incorporated herein by reference.

[0057] Nearest Neighbor Algorithm. In some embodiments, the model is a nearest neighbor algorithm. In some implementations, the nearest neighbor model is memory-based and does not involve model fitting. For nearest neighbor, given a query point x0 (test object), k training points x0 closest to x0 are (r), r, ..., k (here the training objects) are identified and then the point x0 is classified using its k nearest neighbors. In some embodiments, the Euclidean distance in the feature space is

number

[0058] A k-nearest neighbor model is a non-parametric machine learning method in which the input is composed of the k nearest training examples in feature space. The output is a class membership. An object is classified by the multiple votes of its neighbors, and the object is assigned to the most common class among its k nearest neighbors (k is a positive integer, typically small). If k=1, the object is simply assigned to the class of its single nearest neighbor. See Duda et al., 2001, Pattern Classification, Second Edition, John Wiley & Sons, incorporated herein by reference. In some embodiments, the number of distance calculations required to solve a k-nearest neighbor model is such that it cannot be performed by the human mind, and therefore a computer is used to solve the model for a given input.

[0059] Random Forest, Decision Tree, and Boosted Tree Algorithms. In some embodiments, the model is a decision tree. Decision trees suitable for use as models are generally described in Duda, 2001, Pattern Classification, John Wiley & Sons, Inc., New York, pp. 395-396, which is incorporated herein by reference. Tree-based methods partition the feature space into a set of rectangles and then fit a model (such as a constant) to each. In some embodiments, the decision tree is a random forest regression. For example, one specific algorithm is Classification and Regression Trees (CART). Other specific decision tree algorithms include, but are not limited to, ID3, C4.5, MART, and random forest. CART, ID3, and C4.5 are described in Duda, 2001, Pattern Classification, John Wiley & Sons, Inc., New York, pp. 396-408 and pp. 411-412, which are incorporated herein by reference. CART, MART, and C4.5 are described in Hastie et al., 2001, The Elements of Statistical Learning, Springer-Verlag, New York, Chapter 9, which is incorporated herein by reference. Random forests are described in Breiman, 1999, "Random Forests--Random Features," Technical Report 567, Statistics Department, UC Berkeley, September 1999, which is incorporated herein by reference in its entirety. In some embodiments, the decision tree model includes at least 10, at least 20, at least 50, or at least 100 parameters (e.g., weights and / or decisions) that cannot be solved by the human mind and therefore must be computed by a computer.

[0060] Regression. In some embodiments, the model uses a regression algorithm. In some embodiments, the regression algorithm is any type of regression. For example, in some embodiments, the regression algorithm is logistic regression. In some embodiments, the regression algorithm is logistic regression with lasso, L2, or elastic net regularization. In some embodiments, extracted features with corresponding regression coefficients that fail to meet a threshold are excluded (removed) from consideration. In some embodiments, a generalization of the logistic regression model that handles multi-category responses is used as the model. The logistic regression algorithm is disclosed in Agresti, An Introduction to Categorical Data Analysis, 1996, Chapter 5, pp. 103-144, John Wiley & Son, New York, which is incorporated herein by reference. In some embodiments, the model utilizes the regression model disclosed in Hastie et al., 2001, The Elements of Statistical Learning, Springer-Verlag, New York. In some embodiments, the logistic regression model includes at least 10, at least 20, at least 50, at least 100, or at least 1000 parameters (e.g., weights) that cannot be solved by the human mind and therefore must be calculated by a computer.

[0061] Linear Discriminant Analysis Algorithm. In some embodiments, Linear Discriminant Analysis (LDA), Normal Discriminant Analysis (NDA), or Discriminant Function Analysis, is a generalization of Fisher's Linear Discriminant, a method used in statistics, pattern recognition, and machine learning to find linear combinations of features that characterize or separate two or more classes of objects or events. In some embodiments, the resulting combinations are used as models (linear models) in some embodiments of the present disclosure.

[0062] Mixture Models and Hidden Markov Models. In some embodiments, the model is a mixture model such as that described in McLachlan et al., Bioinformatics 18(3):413-422, 2002. In some embodiments, particularly those that include a temporal component, the model is a hidden Markov model such as that described in Schliep et al., 2003, Bioinformatics 19(1):i255-i263.

[0063] Clustering. In some embodiments, the model is an unsupervised clustering model. In some embodiments, the model is a supervised clustering model. Clustering algorithms suitable for use as models are described, for example, in Duda and Hart, Pattern Classification and Scene Analysis, 1973, John Wiley & Sons, Inc., New York (hereinafter Duda 1973), pages 211-256, which is incorporated herein by reference in its entirety. As an illustrative example, in some embodiments, the clustering problem is described as the problem of finding natural groupings in a data set. To identify natural groupings, two problems are addressed. First, a way to measure the similarity (or heterogeneity) between two samples is determined. This metric (e.g., a similarity metric) is used to ensure that samples in one cluster are more similar to each other than samples in another cluster. Second, a mechanism is determined for partitioning the data into clusters using the similarity metric. One way to begin a clustering search is to define a distance function and calculate a matrix of distances between all pairs of samples in the training set. If distance is a good metric of similarity, then the distance between reference entries in the same cluster is significantly smaller than the distance between reference entries in different clusters. However, in some implementations, the clustering does not use a distance metric. For example, in some implementations, a non-metric similarity function s(x, x') is used to compare two vectors x and x'. In some such implementations, s(x, x') is a symmetric function whose value is large when x and x' are "similar" in some respect. Once a method for measuring "similarity" or "heterogeneity" between points in a dataset is selected, the clustering uses a criterion function that measures the clustering quality of any partition of the data. The partition of the dataset from which the criterion function is extracted is used to cluster the data.Specific exemplary clustering techniques contemplated for use in the present disclosure include, but are not limited to, hierarchical clustering (agglomerative clustering using nearest neighbor, longest distance, average linkage, centroid, or sum of squares algorithms), k-means clustering, fuzzy k-means clustering algorithms, and Jarvis-Patrick clustering. In some embodiments, the clustering comprises unsupervised clustering (e.g., the number of clusters is not predetermined and / or cluster assignments are not predetermined).

[0064] Ensembles of Models and Boosting. In some embodiments, an ensemble of models (two or more) is used. In some embodiments, boosting techniques such as AdaBoost are used in conjunction with many other types of learning algorithms to improve the performance of the models. In this approach, the outputs of any of the models disclosed herein, or their equivalents, are combined into a weighted sum that represents the final output of the boosted model. In some embodiments, the multiple outputs from the models are combined using any metric of central tendency known in the art, including but not limited to the mean, median, mode, weighted average, weighted median, weighted mode, etc. In some embodiments, the multiple outputs are combined using a voting method. In some embodiments, each model in the ensemble of models is weighted or unweighted.

[0065] As used herein, the term "parameter" refers to any coefficient, or similarly any value, of an internal or external element (e.g., weight and / or hyperparameter) in an algorithm, model, regressor, and / or classifier that can affect (e.g., can be modified, tuned, and / or adjusted) one or more inputs, outputs, and / or functions in the algorithm, model, regressor, and / or classifier. For example, in some embodiments, a parameter refers to any coefficient, weight, and / or hyperparameter that can be used to control, modify, tune, and / or adjust the behavior, learning, and / or performance of an algorithm, model, regressor, and / or classifier. In some instances, a parameter is used to increase or decrease the influence of an input (e.g., feature) to an algorithm, model, regressor, and / or classifier. As a non-limiting example, in some embodiments, a parameter is used to increase or decrease the influence of a node (e.g., of a neural network), where the node includes one or more activation functions. The assignment of parameters to specific inputs, outputs, and / or functions is not limited to any one paradigm for a given algorithm, model, regressor, and / or classifier, but may be used in any suitable algorithm, model, regressor, and / or classifier architecture for the desired performance. In some embodiments, the parameters have fixed values. In some embodiments, the values ​​of the parameters are manually and / or automatically adjustable. In some embodiments, the values ​​of the parameters are modified by a validation and / or training process for the algorithm, model, regressor, and / or classifier (e.g., by error minimization and / or backpropagation methods). In some embodiments, the algorithms, models, regressors, and / or classifiers of the present disclosure include multiple parameters.In some embodiments, the plurality of parameters is n parameters, where n>2, n>5, n>10, n>25, n>40, n>50, n>75, n>100, n>125, n>150, n>200, n>225, n>250, n>350, n>500, n>600, n>750, n>1,000, n>2,000, n>4,000, n>5,000, n>7,500, n>10,000, n>20,000, n>40,000, n>75,000, n>100,000, n>200,000, n>500,000, n>1x10. 6 , n ≥ 5 × 10 6 , or n ≧ 1×10 7 Therefore, the algorithms, models, regressors, and / or classifiers of the present disclosure cannot be implemented in a human mind. In some embodiments, n is between 10,000 and 1×10 7 , 100,000~5×10 6 , or 500,000 to 1×10 6 In some embodiments, the algorithms, models, regressors, and / or classifiers of the present disclosure operate in a k-dimensional space, where k is a positive integer greater than or equal to 5 (e.g., 5, 6, 7, 8, 9, 10, etc.). Thus, the algorithms, models, regressors, and / or classifiers of the present disclosure cannot be implemented in the human mind.

[0066] As used herein, the term "untrained model" (e.g., "untrained classifier" and / or "untrained neural network") refers to a machine learning model or algorithm, such as a classifier or neural network, that has not been trained on a target dataset. In some embodiments, "training a model" (e.g., "training a neural network") refers to the process of training an untrained or partially trained model (e.g., an "untrained or partially trained neural network"). Furthermore, it will be understood that the term "untrained model" does not exclude the possibility that transfer learning techniques may be used in such training of an untrained or partially trained model. See, for example, Fernandes et al., 2017, "Transfer Learning with Partial Observability Applied to Cervical Cancer Screening," Pattern Recognition and Image Analysis: 8, incorporated herein by reference. thIberian Conference Proceedings, 243-250 provides non-limiting examples of such transfer learning. In instances where transfer learning is used, the untrained model described above is provided with additional data beyond that of the primary training dataset. Typically, this additional data is in the form of parameters (e.g., coefficients, weights, and / or hyperparameters) learned from another auxiliary training dataset. Furthermore, while a description of a single auxiliary training dataset is disclosed, it will be understood that there is no limit to the number of auxiliary training datasets that may be used to complement the primary training dataset in training the untrained model in this disclosure. For example, in some embodiments, two or more auxiliary training datasets, three or more auxiliary training datasets, four or more auxiliary training datasets, or five or more auxiliary training datasets are used to complement the primary training dataset through transfer learning, each such auxiliary dataset being different from the primary training dataset. In some such embodiments, any manner of transfer learning is used. For example, consider the case where, in addition to the primary training dataset, there is a first auxiliary training dataset and a second auxiliary training dataset. In such cases, parameters learned from the first auxiliary training dataset (by application of a first model to the first auxiliary training dataset) are applied to the second auxiliary training dataset using transfer learning techniques (e.g., a second model that is identical to the first model or different), resulting in a trained intermediate model, the parameters of which are then applied to the primary training dataset, which, together with the primary training dataset itself, are applied to the untrained model.Alternatively, in another exemplary embodiment, a first set of parameters learned from a first auxiliary training data set (by application of a first model to the first auxiliary training data set) and a second set of parameters learned from a second auxiliary training data set (by application of the same or different second model from the first model to the second auxiliary training data set) are each individually applied (e.g., by separate and independent matrix multiplications) to separate instances of the primary training data set, and both such applications of parameters to separate instances of the primary training data set, together with the primary training data set itself (or some reduced form of the primary training data set, such as principal components or regression coefficients learned from the primary training set), are then applied to the untrained model to train the untrained model.

[0067] Some aspects are described herein with reference to example applications for illustration. It should be understood that numerous specific details, relationships, and methods are described to provide a thorough understanding of the features described herein. However, a person skilled in the relevant art will readily recognize that the features described herein can be implemented without one or more of the specific details, or with other methods. The features described herein are not limited by the illustrated order of acts or events, as some acts may occur in different orders and / or simultaneously with other acts or events. Moreover, not all illustrated acts or events are required to implement a methodology in accordance with the features described herein.

[0068] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In this disclosure, numerous specific details are described in order to provide a thorough understanding of the disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0069] Exemplary System Embodiments. Having provided an overview of some aspects of the present disclosure and some definitions used herein, details of an exemplary system for training a model for phenotyping a subject for a medical condition (e.g., a cardiovascular condition, a neurological condition, cancer, etc.) will now be described in conjunction with Figures 1A-1D, which collectively illustrate the topology of an exemplary system for training a model for phenotyping a subject for a medical condition, according to some embodiments of the present disclosure. Advantageously, the exemplary system illustrated in Figures 1A-1D improves upon conventional methods for training a model for phenotyping a subject for a medical condition by aggregating label functions that query multimodal EHR data to generate weak labels for model training.

[0070] 1A-1D collectively illustrate block diagrams illustrating a system according to some embodiments. The device 100 in some embodiments includes one or more processing units CPU 102 (also referred to as a processor), one or more network interfaces 104, a user interface 106 including, for example, a display 108 and / or an input 110 (e.g., a mouse, touchpad, keyboard, etc.), non-persistent memory 111, persistent memory 112, and one or more communication buses 114 for interconnecting these components. The one or more communication buses 114 optionally include circuitry (sometimes referred to as a chipset) that interconnects and controls communication between the system components. While non-persistent memory 111 typically includes high-speed random access memory such as DRAM, SRAM, DDR RAM, ROM, EEPROM, flash memory, etc., persistent memory 112 typically includes CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, magnetic disk storage device, optical disk storage device, flash memory device, or other non-volatile solid-state storage device. Persistent memory 112 optionally includes one or more storage devices located remotely from CPU 102. Persistent memory 112 and non-volatile memory devices within non-persistent memory 112 comprise non-transitory computer-readable storage media. In some implementations, non-persistent memory 111, or alternatively non-transitory computer-readable storage media, sometimes in conjunction with persistent memory 112, stores the following programs, modules, and data structures, or a subset thereof: An operating system 30, which includes procedures for handling various basic system services and for performing hardware-dependent tasks; A network communication module (or instructions) 30 for connecting the system 100 with other devices and / or a communication network 105; an electronic health record store 40 that stores electronic health records 41 (e.g., electronic health records 41-1, ... 41-A) relating to a subject, including one or more data types, such as structured clinical data 42, unstructured clinical data 43, prescribed medications 44, medical procedures 45, and lab results 46, as well as optional weak indicators 47 and / or classifications 48 generated using the methods described herein; an optional filter module 50 that constructs and / or applies filters 51 (e.g., criteria 52-1, ... 52-B) to the electronic health records 41 as an optional step to filter out electronic health records 41 that are not associated with the medical condition of interest before assigning a weak indicator, thereby enriching the set of electronic records for those associated with the medical condition of interest by querying the electronic health records 41 against inclusion and / or exclusion criteria 52 for features 53; a weak labeling module 60 for generating labels for training a classification model 80 by constructing and / or applying labeling functions 61 (e.g., labeling functions 61-1, ... 61-C) to the electronic health record 41 to query the electronic health record 41 against criteria 62 (e.g., criteria 62-1-1, ... 62-1-G of labeling function 1 61-1) and combining the outputs of the individual labeling functions 61 using an ensemble model 63; an optional training module 70 for training a classification model 80 to predict cases of a medical condition using information (e.g., one or more data types such as structured clinical data 42, unstructured clinical data 43, prescribed medications 44, medical procedures 45, and lab results 46) from an electronic health record (e.g., electronic health record 41) and weak labels 47 as variables, the training module optionally including a hyperparameter search algorithm 71 and a parameter optimization algorithm 72; An optional classification model 80 (e.g., trained using the optional training module 70), having a number of hyper-parameters 81 and model parameters 82; · An optional phenotyping module 90 for predicting a phenotype for a medical condition by inputting information from an electronic health record (eg, from the EHR store 40) into the classification model 80.

[0071] 1A-1D depict various components of "System 100," the figures are intended as functional illustrations of various features that may be present in a computer system in some embodiments, rather than as structural schematics of the embodiments described herein. In fact, in some embodiments, items shown separately are combined and some items are separate. Furthermore, while FIGS. 1A-1D depict certain data and modules in non-persistent memory 111, in some embodiments, some or all of these data and modules are in persistent memory 112. For example, in various embodiments, one or more of the above-identified elements are stored in one or more of the aforementioned memory devices and correspond to sets of instructions for performing the functions described above. The above-identified modules, data, or programs (e.g., sets of instructions) need not be implemented as separate software programs, procedures, data sets, or modules, and thus various subsets of these modules and data are, in some embodiments, combined or otherwise rearranged in various implementations.

[0072] In some implementations, non-persistent memory 111 optionally stores a subset of the modules and data structures identified above. Additionally, in some embodiments, the memory stores additional modules and data structures not described above. In some embodiments, one or more of the above-identified elements are stored in a computer system other than that of system 100 that is addressable by system 100 such that system 100 retrieves all or a portion of such data when needed.

[0073] 1A, system 100 is represented as a single computer that includes all of the functionality for training a model for phenotyping a subject for a medical condition. However, while a single machine is illustrated, the term "system" should also be construed to include any collection of machines that individually or jointly execute a set (or sets) of instructions for performing any one or more of the methodologies discussed herein.

[0074] For example, in some embodiments, system 100 includes one or more computers. In some embodiments, functionality for training a model for phenotyping a subject for a medical condition is spread across any number of networked computers and / or resident on each of several networked computers and / or hosted on one or more virtual machines at remote locations accessible over communications network 105. For example, different portions of the various modules and data stores illustrated in Figures 1A-1D may be stored and / or executed on different instances of processing devices and / or processing servers / databases within a distributed computing environment.

[0075] In some embodiments, the system may operate in the capacity of a server or client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or client machine in a cloud computing infrastructure or environment. In some embodiments, the system may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, server, network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine.

[0076] In another embodiment, the system comprises a virtual machine including modules for executing instructions for performing any one or more of the methodologies disclosed herein. In computing, a virtual machine (VM) is an emulation of a computer system that is based on a computer architecture and provides the functionality of a physical computer. Some such embodiments involve dedicated hardware, software, or a combination of hardware and software.

[0077] Those skilled in the art will appreciate that any of a wide variety of different computer topologies may be used in an application, and that all such topologies are within the scope of the present disclosure.

[0078] In some embodiments, the system 100 further includes a clinical trial module that evaluates the one or more electronic health records 41 with the classification 48 indicative of the presence of the medical condition to determine whether the patient is eligible for inclusion in a clinical trial for the treatment of the medical condition, e.g., a clinical trial that is currently recruiting patients, a clinical trial that has not yet begun recruiting patients, and / or an ongoing clinical trial that may recruit additional patients in the future. In some embodiments, the clinical trial module evaluates the one or more electronic health records 41 with the classification 48 indicative of the presence of the medical condition to determine whether the results of a clinical trial, e.g., the results of an ongoing clinical trial and / or the results of a completed clinical trial, are relevant to the subject. For example, in some embodiments, the system 100 queries a database related to the medical condition, e.g., a look-up table (LUT) of clinical trials, e.g., active and / or completed clinical trials, and compares information in the one or more electronic health records 41 with the inclusion criteria of the clinical trials stored in the database to identify subjects who meet or nearly meet the clinical trial inclusion criteria. In some embodiments, records of matching clinical trials, e.g., clinical trials for which the subject is eligible or likely to be eligible, and / or trials for which the subject was eligible or likely to be eligible, are further stored in system 100 and / or subject data store 120.

[0079] In some embodiments, system 100 further includes a reporting module that generates a report to a physician or healthcare organization (e.g., a hospital, clinic, healthcare provider, or health insurance company) that evaluates the one or more classifications 48 and identifies one or more undiagnosed subjects that are likely to have a medical condition. In some embodiments, the report includes one or more clinical recommendations for further evaluation and / or treatment of the identified subjects. In some embodiments, the report includes one or more treatment recommendations for the identified subjects. In some embodiments, the report includes one or more clinical trials that may be relevant to the identified patient, for example, as identified using the clinical trials module described above.

[0080] In some embodiments, the system 100 includes a model training module that includes instructions for training one or more untrained or partially trained models based on feature data from a training dataset. In some embodiments, the system 100 also includes a database of training data (e.g., multiple training electronic health records) for use in training the one or more models. In other embodiments, the model training module accesses a remote storage device that hosts the training data. In some embodiments, the training data includes a set of training features that includes multi-modal data extracted from electronic health records (e.g., structured clinical data 42, unstructured clinical data 43, medications 44, medical procedures 45, and / or lab results 46), as illustrated in Figures 1A-1B, without limitation.

[0081] In some embodiments, each of the one or more data stores and / or modules is communicatively coupled to a data bus 114 to transfer data between each data store or module for processing and / or storage. In some alternative embodiments, each of the one or more data stores and / or modules is communicatively coupled to one another for independent communication without sharing a data bus.

[0082] Further details about the system and example embodiments of the modules and feature sets are discussed in PCT application PCT / US19 / 69149, entitled "METHOD AND PROCESS FOR PREDICTING AND ANALYZING PATIENT COHORT RESPONSE, PROGRESSION, AND SURVIVAL," filed December 31, 2019, which is incorporated by reference in its entirety for all purposes.

[0083] Exemplary embodiments Now that details of the system 100 for training a model for phenotyping a subject for a medical condition have been disclosed, details regarding the processes and features of the system according to various embodiments of the present disclosure are provided below. In particular, an exemplary process is described below with reference to FIGS. 2A-2H. In some embodiments, such processes and features of the system are performed by modules 50, 60, 70, and / or 90, as illustrated in FIGS. 1A-1D. With reference to these methods, in some embodiments, the systems described herein (e.g., system 100) include instructions for training a model for phenotyping a subject for a medical condition that are improved as compared to conventional methods for training a model for phenotyping a subject for a medical condition.

[0084] In one aspect, the present disclosure provides a method 200 for training a model for phenotyping a subject for a medical condition. In some embodiments, all or a portion of the method is implemented in a computer system including at least one processor and a memory storing at least one program having instructions for execution by the at least one processor.

[0085] In some embodiments, the one or more electronic health records are pre-processed prior to analysis using the methods and systems described herein. For example, in some embodiments, an electronic health record (EHR) is obtained from an image of a health record, such as a scanned image of a physical (e.g., paper and / or handwritten) health record document. In some embodiments, the EHR is obtained in an image file format, such as a PDF. In some such embodiments, the method includes analyzing the EHR using a character recognition process to convert the image into computer-readable text. For example, in some embodiments, the method includes, prior to receiving the first electronic health record, receiving one or more images of text corresponding to the first electronic health record, and converting the one or more images of text into a computer-readable text format using text recognition. In some embodiments, the character recognition is optical character recognition (OCR). In some embodiments, the OCR converts the image into raw text.

[0086] Methods for text recognition are well known in the art and include, but are not limited to, sliding window classification, connected component analysis (CCA), bounding box regression-based methods, segmentation-based methods, and / or combinations thereof. For example, sliding window classification utilizes a convolutional classifier to detect characters in an image using a multi-scale sliding window. CCA-based methods operate by segmenting pixels in an image that have consistent local characteristics, such as color, edge, texture, and / or stroke width, into characters. Alternatively, or additionally, text line-based methods operate by first identifying lines of text and partitioning the identified text lines into smaller components, such as words and characters. In general, bounding box and segmentation-based methods operate by detecting text at a single word level, for example, using bounding boxes that isolate regions of text from the local background, with optional filtering, cleaning, and post-recognition processing. See, for example, Keerthana et al., 2020, “Text Detection and Recognition: A Review,” IRJET Vol 7(8), 2156-2169, which is incorporated by reference in its entirety.

[0087] In some embodiments, the segmentation is performed by identifying one or more regular expressions in the corresponding sections. Generally, a regular expression refers to a sequence of characters that specifies a search pattern in the text. For example, in an exemplary embodiment, the segmentation uses language pattern recognition to segment each section into sentences and search for the ends of the sentences, as specified by the corresponding regular expressions.

[0088] For example, in some embodiments, regular expression filtering is used to split a first EHR into multiple sentences. An example of a regular expression syntax that can be used to split raw text into sentences is "r'\s{2,}|(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s'". In some embodiments, regular expression filtering is used to subdivide each respective section within a plurality of sections into corresponding multiple text spans that span both sides of one or more punctuation marks. For example, in some embodiments, text spans are obtained by subdividing a string (e.g., a section) with a period, question mark, exclamation point, comma, colon, semicolon, dash, hyphen, bracket, parenthesis, apostrophe, quotation mark, and / or ellipsis. In some embodiments, certain punctuation marks are excluded from being identified as text span (e.g., sentence) boundaries. For example, in some embodiments, the last period in the doctor's abbreviation "Dr." is excluded (e.g., "dr.XX"). An example of a regular expression syntax useful for excluding the identification of certain punctuation marks as text span (e.g., sentence) boundaries can be found, for example, in Section 3.2.2 of Rokach L. et al., Information Retrieval Journal, 11(6):499-538 (2008), which is hereby incorporated by reference in its entirety for all purposes.

[0089] In some embodiments, relevant portions of the EHR are extracted and / or indexed prior to data analysis. For example, in some embodiments, structured diagnostic codes from multiple EHRs are extracted into a table or other data structure to facilitate the identification of diagnostic codes indicating the presence or absence of a target medical condition.

[0090] In some embodiments, the EHR is divided into separate episodes. For example, the EHR may be divided into a first episode related to the diagnosis and / or treatment of a first medical condition and a second episode related to the diagnosis and / or treatment of a second medical condition. For example, a first episode (e.g., related to PH diagnosis and / or treatment) may span more than one clinical visit and / or intervention, including, for example, one or more diagnostic clinical visits, one or more treatment visits, one or more follow-up visits, etc., all related to the first medical condition (e.g., PH). Similarly, a second episode (e.g., related to cancer diagnosis and / or treatment) may span more than one clinical visit and / or intervention, including, for example, one or more diagnostic clinical visits, one or more treatment visits, one or more follow-up visits, etc., all related to the second medical condition (e.g., cancer). Different episodes within a single EHR may or may not overlap in time.

[0091] In some embodiments, clinical notes in the EHR are partitioned into separate episodes. For example, in some embodiments, a visit may be defined as an interaction between a patient and a healthcare provider that results in the logging of clinical notes into the EHR system. In some embodiments, an episode includes a cluster of visits that represent one hospitalization. That is, one hospitalization may be logged as multiple visits. In some embodiments, episode boundaries are determined using one-dimensional clustering (e.g., kernel density estimation (KDE)) on the visit dates. In some embodiments, notes between boundaries are aggregated together.

[0092] In some embodiments, the clinical notes are segmented to identify distinct episodes. An exemplary method for segmenting the clinical notes includes dividing the unstructured clinical data by tokenizing the unstructured clinical data to obtain a plurality of tokens, and segmenting the plurality of tokens to obtain a plurality of segments, e.g., each segment in the plurality of segments has approximately the same number of tokens. In some embodiments, the relevant segments may be further enriched by ranking each segment in the plurality of segments based on the values ​​of the tokens in each respective segment, and removing one or more respective segments from the plurality of segments based on the ranking, thereby generating a corresponding plurality of snippets for the epitope record.

[0093] In some embodiments, the phenotyping process begins with exploratory data analysis (EDA), optionally supplemented by consultation with clinical domain experts and literature reviews of relevant clinical studies, informatics or technical articles, and / or clinical guidelines for the target medical condition. In some embodiments, the exploratory data analysis includes characterizing all relevant structured data elements in the dataset, such as diagnostic codes, procedures, medications, orders, labs, measurements, imaging studies, devices, etc. In some embodiments, the EDA also includes characterizing the data elements with respect to the extent to which they belong to relevant criteria for diagnosing the target medical condition, and / or comparing to gold standard diagnostic metrics to identify whether gaps exist, such as discrepancies between patients with relevant diagnostic codes.

[0094] In some embodiments, the method includes the use of a filter to extract a subpopulation of medical records of interest, generating an effective and unbiased sample for chart review and phenotype development. In some embodiments, the filter is developed iteratively, for example, based on domain expertise from clinicians, literature review of any and all codes or criteria used in the paper, and / or findings from initial data exploration. In some embodiments, the filter includes a combination of all potentially relevant signals for the target medical condition. In some embodiments, the filter includes a substring match of clinical notes to maximize coverage of the medical condition in multiple EHRs at the outset.

[0095] As an example, consider phenotyping an EHR for pulmonary hypertension (PH). The prevalence of PH in the general population can be estimated to be approximately 1%, so one goal of filtering is to reduce the large general population to a subpopulation of interest with a higher prevalence, e.g., greater than 2.5%, greater than 5%, greater than 10%, etc., without sacrificing sensitivity by removing any significant proportion of the disease population. Returning to the analogy of our EHR representation described above with reference to FIG. 4, this subpopulation of interest is represented by the dotted circle created by the extraction filter in FIG. 4G, with few or no cases of patients with true disease outside the circle being missed. This higher prevalence of PH in the selected subpopulation improves random, unbiased sampling of this population of interest, and retains enough true positive patients with the disease, even in a random sample of 100, to generate fairly reliable metrics from chart review and manual abstraction. Otherwise, random sampling in the general population would be expected to yield only one or two (or none) patients with true disease from a random sampling of 100 EHRs, thus necessitating either an impractical large number of chart reviews (possibly thousands) or a biased sample that could not be used across multiple stages and iterative phenotype development.

[0096] In some embodiments, the same set of unbiased samples and chart review results (ground truth labels) from this extracted subpopulation of interest can be used as training, validation, and test sets across multiple different iterations of the phenotype, including the NLP model-generated phenotype after structured signal aggregation, allowing for effective comparison between different versions and different types of models. In contrast, targeted biased samples were only relevant to a particular phenotype, such as selecting a random sample of positive patients from the structured phenotype, and could not be used to evaluate future versions of the phenotype because the sampling was anchored only to the structured phenotype. These targeted samples are useful for closely evaluating a particular phenotype, providing good confidence in the PPV metric, such as with reviewing the final phenotype at each stage of the process, and may represent how effective it is for the downstream task of labeling ECGs. However, for development, evaluating each iteration of the phenotype quickly becomes resource intensive, but metrics cannot be compared across iterations or types, such as when comparing structured vs. NLP.

[0097] The filtered random samples may be used throughout the entire phenotype development pipeline, with the training set being used for structured phenotype development, iterative improvement including false positive / negative review, and building the indicator function and ML model, while the validation set remains separate, blinded, and used for evaluation purposes only.

[0098] Thus, referring to block 202, in some embodiments, the method further includes filtering the second plurality of subjects according to the first set of features of the medical condition, where filtering includes applying an extraction filter to medical records corresponding to the second plurality of subjects to obtain the first plurality of subjects, where the first plurality of subjects is a subset of the second plurality of subjects. For example, as illustrated in FIG. 3, EHR records 302 are filtered (354) to remove filtered records 312 and pass training records 308 and validation records 310 along to the next step of the process.

[0099] In some embodiments, a medical record (e.g., an EHR) used in the methods described herein is first partitioned to form at least a training subset and a validation subset. In some embodiments, the medical records are also partitioned to include a test subset. An example of this partitioning is assigning 80% of the medical records to a training set, 10% of the medical records to a validation set, and 10% of the medical records to a test set. However, other partitions of the medical records may be used. For example, as illustrated in FIG. 3, an EHR record 302 is partitioned into a training subset 304 and a validation subset 306 prior to the filter step (354).

[0100] In some embodiments, the medical records used in the methods described herein are not split into training, validation, and / or test subsets prior to filtering. In some embodiments, the medical records are split into training, validation, and / or test subsets after initial filtering, e.g., to ensure a desired ratio of split records. That is, the medical records are split into x:y:z (training:validation:test) prior to an initial filtering step, e.g., filtering (202) of FIG. 2 or filtering step (354) of FIG. 3, and filtering removes a higher percentage of records from one set, such that the split is no longer x:y:z. With reference to the exemplary workflow of FIG. 3, in some embodiments, the EHR records 302 are filtered (354) prior to splitting into training records 308 and validation records 310 (e.g., pre-filtered training record subsets 304 and validation record subsets 306 are not formed).

[0101] Referring to block 204, in some embodiments, the extraction filter includes a plurality of criteria (e.g., criteria 52 illustrated in FIG. 1C ) related to a first set of features for a medical condition (e.g., features 53 illustrated in FIG. 1C ). Applying the extraction filter to a plurality of medical records (e.g., medical records 41 of FIG. 1 , EHR 302 of FIG. 3 , and / or training subset 304 and validation subset 306 of FIG. 3 ) includes, for each respective medical record 41 in the plurality of medical records, evaluating the respective medical record against a respective criterion 52 in the plurality of criteria, where satisfaction of a single respective criterion is sufficient to include or exclude the respective medical record in the subset of medical records (e.g., the subset of training records 308 or the subset of validation records 310 of FIG. 3 ).

[0102] In some embodiments, the filtered medical records, e.g., one or both of the records that pass the filter and the records that are removed by the filter, are independently reviewed to evaluate the performance of the filtering step. For example, referring to FIG. 3, in some embodiments, a sample of training medical records 41 from the plurality of medical records 308 are independently reviewed (356) to determine whether any records 41 that are unrelated to PH have passed the filter 51. In some embodiments, if any records 41 that passed the filtering step (354) are identified as unrelated to PH, the filter is refined (352) to exclude such records during filtering. For example, if the target medical disorder is pulmonary hypertension (commonly abbreviated as PH), a mention of "Ph.D." in a medical record may meet criterion 52 such that the record may pass the filter. By identifying such records upon independent review, the filter 51 may be modified to exclude "Ph.D." from meeting the inclusion criteria. Similarly, referring again to FIG. 3, in some embodiments, a sample of medical records 41 in the filtered subset of medical records 312 are independently reviewed (358) to determine whether any records 41 related to PH have been filtered by filter 51. In some embodiments, if any filtered records 41 are identified as related to PH during the filtering step (354), the filter is refined (352) to include such records in the filtering. For example, if the target medical disorder is PH and the medical records include a right heart catheterization (RHC) mean pulmonary artery pressure of greater than 20 mmHg but do not include sufficient mention of PH to satisfy any criteria 52, filter 51 may be modified to include criteria 52 satisfied by a mean pulmonary artery pressure of greater than 20 mmHg.

[0103] In some embodiments, this review is performed on de-identified data, which may include, for example, unstructured clinical notes and / or structured data fields from the EHR. In some embodiments, all chart reviews are blinded, such that the reviewer does not know the phenotype or model label, and is only intended to confirm the clinical ground truth of the medical diagnosis and / or index the diagnosis date. In some embodiments, the review of one or more records is performed multiple times, and the consensus on the label is used as the ground truth for the record. In some embodiments, the independent review is performed before the filtering step. For example, with reference to FIG. 3, in some embodiments, a sampling of medical records 41 in the subset of training records 304 is independently reviewed before the filtering step (354). In some embodiments, the independent review is performed after the filtering step. For example, with reference again to FIG. 3, in some embodiments, a sampling of medical records 41 and / or filtered records 312 in the subset of training records 308 is independently reviewed after the filtering step (354).

[0104] In some embodiments, at least 10 medical records are independently reviewed. In some embodiments, at least 20 medical records, at least 25 medical records, at least 30 medical records, at least 40 medical records, at least 50 medical records, at least 75 medical records, at least 100 medical records, at least 150 medical records, at least 200 medical records, at least 250 medical records, at least 300 medical records, at least 400 medical records, at least 500 medical records, at least 750 medical records, at least 1000 medical records, at least 2500 medical records, at least 5000 medical records, at least 10,000 medical records, or more medical records are independently reviewed.

[0105] In some embodiments, 50,000 or fewer medical records are independently reviewed. In some embodiments, 25,000 or fewer medical records, 10,000 or fewer medical records, 7500 or fewer medical records, 5000 or fewer medical records, 2500 or fewer medical records, 1000 or fewer medical records, 750 or fewer medical records, 500 or fewer medical records, 250 or fewer medical records, 100 or fewer medical records, or fewer medical records are independently reviewed.

[0106] In some embodiments, 10-10,000 medical records are independently reviewed. In some embodiments, 10-7500 medical records, 10-5000 medical records, 10-2500 medical records, 10-1000 medical records, 10-750 medical records, 10-500 medical records, 10-250 medical records, 10-100 medical records, 50 to 10,000 medical records, 50-7500 medical records, 50-5000 medical records, 50-2500 medical records, 50-1000 medical records, 50-750 medical records, 50-500 medical records, 50-250 medical records, 50-100 medical records, 250-10,000 medical records, 0 medical records, 250-7500 medical records, 250-5000 medical records, 250-2500 medical records, 250-1000 medical records, 250-750 medical records, 250-500 medical records, 500 to 10,000 medical records, 500-7500 medical records, 500-5000 medical records, 500-2500 medical records, 500-1000 medical records, 1000-10,000 medical records, 1000-7500 medical records, 1000-5000 medical records, or 1000-2500 medical records will be independently reviewed.

[0107] Thus, referring to block 206, in some embodiments, the method includes, for one or more respective subjects in the second plurality of subjects (e.g., medical records 302 illustrated in FIG. 3 ) excluded from the first plurality of subjects (e.g., excluded from the subset of training records 308 or the subset of validation records 310 and included in the filtered records 312, as illustrated in FIG. 3 ), evaluating whether the corresponding medical record indicates the presence of a medical condition (e.g., PH) independent of the evaluation of the respective medical records during application of the extraction filter (e.g., filtering 354 illustrated in FIG. 3 ), and upon determining that the respective medical record 41 indicates the presence of a medical condition, modifying the plurality of criteria 52 associated with the first set of features 53 to be more inclusive when applying the extraction filter (constructing a filter (352) illustrated in FIG. 3 ).

[0108] Referring to block 208, in some embodiments, applying the extraction filter (e.g., filtering 354 illustrated in FIG. 3) includes, for each respective subject in the second plurality of subjects (e.g., medical record 302 illustrated in FIG. 3), evaluating a first data type (e.g., one of structured clinical data 42, unstructured clinical data 43, medications 44, medical procedures 45, or lab results 46) against a first respective criterion 52-i in the plurality of criteria, and evaluating a second data type (e.g., a different one of structured clinical data 42, unstructured clinical data 43, medications 44, medical procedures 45, or lab results) against a second respective criterion 52-j in the plurality of criteria.

[0109] Referring to block 210, in some embodiments, the first type of data is structured data in an electronic health record (EHR) (e.g., structured clinical data 42) and the second type of data is unstructured data in the EHR (e.g., unstructured clinical data 43). In some embodiments, the structured data is medical billing codes or diagnosis codes and the unstructured data is clinical notes in the EHR.

[0110] Referring to block 212, in some embodiments, applying an extraction filter (e.g., filtering 354 illustrated in FIG. 3 ) includes evaluating at least three data types selected from structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying an extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, and lab results 46. In some embodiments, applying an extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, and prescribed medications 44. In some embodiments, applying an extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, and performed medical procedures 45. In some embodiments, applying an extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, lab results 46, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, lab results 46, and prescribed medications 44. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, lab results 46, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating lab results 46, prescribed medications 44, and performed medical procedures 45.

[0111] In some embodiments, applying the extraction filter (e.g., filtering 354 illustrated in FIG. 3 ) includes evaluating at least four data types selected from structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, and prescribed medications 44.

[0112] In some embodiments, applying an extraction filter (e.g., filtering 354 illustrated in FIG. 3 ) includes evaluating at least structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45.

[0113] In some embodiments, drugs that indicate a likelihood of a particular medical condition can be identified from a relevant directory of targeted therapies, such as those that indicate a likelihood that a patient has cancer, or a particular type of cancer, found in the National Cancer Institute, "List of Targeted Therapy Drug Approved for Specific Type of Cancer," updated on October 31, 2022, available on the Internet at cancer.gov / about-cancer / treatment / types / targeted-therapies / approved-drug-list, which is incorporated herein by reference in its entirety.

[0114] In some embodiments, the drug is identified from one or more drug databases, such as the Translational Medicine Ontology (TMO). See, for example, Luciano et al., “The Translational Medicine Ontology and Knowledge Base: driving personalized medicine by bridging the gap between bench and bedside,” Journal of Biomedical Semantics, 2011;2(2):S1, which is incorporated herein by reference in its entirety. In some embodiments, the drug is identified from the National Cancer Institute Thesaurus (NCIt). See, for example, “NCIthesaurus,” Version 22.11d (Release Date: 2022-11-28), available on the Internet at ncithesaurus.nci.nih.gov / ncitbrowser, which is incorporated herein by reference in its entirety. Suitable examples of drug databases contemplated for use in the present disclosure include, but are not limited to, CHM: CheMBL, DrugB: Drug-Bank, ODB: Ontario database, THIN: The Health Improvement Network, TCM: Traditional Chinese Medicine, TBDB: Tuberculosis Database, TTD: Therapeutic Target database, PDTD: Potential Drug-Target Database, TDR: Tropical Diseases Research, HIVRT: HIV Drug Resistance Database, TCMSP: Traditional Chinese Medicine Platform, SCYP: Super Cytochrome P450, DHUB: Drug Repurposing Hub, DSDB: Drug Signatures Database, PROM: Promiscuous, DRAR: Drug RepurposingAdverse Reaction, DMAP:Drug-Protein connectivity MAP, CMAP: Complement Map database, DMC:Drug Map Central, SIDER:Side Effect Resource, KSRPO:A platform for drug Repositioning, NNFIN:Network-based similarity finder, DSRV:Drug survival database, CHSP:anti-Cancer Herbs database for System Pharmacology, D2G:Drug to Gene, GSDB:Gene Set Database, SBIOS:Swiss BIOisostere, DTOM: Drug Target interactome database, DPTH: Drug Pathway database, DTW: Drug Target Web, DNET: Drug-disease Network database, SUT: SuperTarget database, DTC: Drug Target Commons, and / or KEGG: Kyoto Encyclopedia of Genes and Genomes. See, e.g., Masoudi-Sobhanzadeh et al., “Drug database and their contributions to drug repurposing,” Genomics. 2020;112(2):1087-1095, which is incorporated by reference in its entirety.

[0115] In some embodiments, the criteria for the plurality of features for a medical condition are one or more words, medical codes, or phrases that are automatically curated, at least in part, using, for example, an electronic medical glossary, a medical terminology, or the like. In some embodiments, the criteria are identified using natural language processing, for example, by training a machine learning model to identify electronic health records of patients affected by a health condition based on structured and / or unstructured clinical data, using techniques such as Local Interpretable Model-Agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), and then performing a post-hoc model interpretability analysis to determine the words and / or phrases that provide predictive power to the model. Similarly, NLP can be used to identify words and / or phrases that are significantly more common in the EHRs of patients affected by a medical condition than in the EHRs of patients not affected by the medical condition.

[0116] In some embodiments, the criteria are evaluated using a string matching algorithm. In general, a string matching algorithm (e.g., a string search algorithm) performs a function in which one or more substrings are identified within a larger string. The function further identifies the location where the substring is located within the larger string. A string includes, but is not limited to, a sequence of text (e.g., an EHR, a section, a text span, or a portion thereof). String matching is further described in Bulus et al., 2017, “Comparison of String-Matching Algorithms in Web Documents,” UniTech 2017, which is incorporated herein by reference in its entirety. In some embodiments, the string matching algorithm includes a naive string search, a finite-state automaton-based search, a Rabin-Karp algorithm, a Knuth-Morris-Pratt algorithm, a Boyer-Moore string search algorithm, a two-way string matching algorithm, or a backward non-deterministic directed acyclic string graph matching algorithm using a computer system including a processor coupled to a non-transient memory. In some embodiments, the string matching is an exact string match. That is, in some embodiments, the algorithm must find an exact match of a given string from the criteria in the EHR (e.g., a curated list of words, medical codes, and / or phrases) to satisfy the criteria. In some embodiments, fuzzy matching logic is used to allow some mismatch between the representation in the EHR and the representation in the curated list of words, medical codes, and / or phrases, for example, by applying a threshold to the string distance between the representation in the EHR and the representation in the curated list. Methods for determining string distances, such as Levenshtein distance, Hamming distance, Jaro-Winkler distance, and the like, are known in the art. In some embodiments, wildcards are used when matching to a curated list to account for different forms and / or inflections of a word, for example, when the curated list includes one or more regular expressions (RegEx).In some embodiments, the string distance threshold is between 0 (e.g., exact match) and 1 (e.g., most tolerant). For example, in some embodiments, the string distance threshold is at least 0.1, at least 0.2, at least 0.3, at least 0.4, at least 0.5, at least 0.6, at least 0.7, or at least 0.8. In some embodiments, the string distance threshold is 0.99 or less, 0.95 or less, 0.9 or less, 0.8 or less, 0.7 or less, 0.6 or less, 0.5 or less, 0.4 or less, 0.3 or less, or 0.2 or less. In some embodiments, the string distance threshold is between 0.1 and 0.6, 0.1 and 0.4, 0.2 and 0.8, 0.4 and 0.95, or 0.3 and 0.7. In some embodiments, the string distance threshold falls within another range between 0 and 1.

[0117] In some embodiments, the criteria related to the features for a medical condition are one or more latent variables learned by the model. For example, in some embodiments, a natural language processing model is trained to distinguish between EHRs from subjects with a particular medical condition and EHRs from subjects without the medical condition. Then, rather than matching expressions in the EHR to expressions in a curated list of words, medical codes, and / or phrases, a portion or all of the EHR is scored using the model to provide an indication (e.g., probability, likelihood, score, classification, etc.) of whether the EHR contains expressions indicative of the presence of a medical condition.

[0118] Any one of several clustering techniques may be used, including but not limited to hierarchical clustering, k-means clustering, and density-based clustering. In some embodiments, the clustering algorithm is density-based spatial clustering of noisy applications (DBSCAN). In some embodiments, a hierarchical density-based clustering algorithm is used (referred to as HDBSCAN, see, e.g., Campello et al., 2015, “Hierarchical density estimates for data clustering, visualization, and outlier detection,” ACM Trans Knowl Disc Data, 10(1), 5). In another embodiment, a community detection-based cluster algorithm such as Louvain clustering is used (see, e.g., Blondel et al., 2008, “Fast unfolding of communities in large networks,” J stat mech:theor exp, 2008(10), P10008). In yet another embodiment, Leyden clustering is used. See, e.g., Traag et al., (2019), “From Louvain to Leiden: guaranteeing well-connected communities,” Sci Rep 9:5233, doi:10.1038 / s41598-019-41695-z. In yet another embodiment, a diffusion path algorithm is used. In some embodiments, the clustering algorithm includes a pre-trained language model and / or an embedding model. In some embodiments, the clustering algorithm further includes applying a dimensionality reduction technique to the data from the EHR. In some implementations, each part in the set of parts of the EHR is represented as a vector, such as a dense representation.

[0119] In some embodiments, the language model is a pre-trained language model. In some embodiments, the language model is a masked language modeling (MLM) model. In some embodiments, the language model is BERT. See, e.g., Devlin et al., 2018, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv:1810.04805v2). In some embodiments, the corresponding evaluation metric is a multiplicity metric. In general, multiplicity refers to a metric of how well a language model predicts a sample, such as an EHR indicating the presence or absence of a medical condition. For example, in some embodiments, multiplicity is determined as the inverse probability of a test set indicating the presence or absence of a medical condition, normalized by the number of words evaluated in the EHR. A lower multiplicity indicates a higher probability such that the model can better predict an EHR indicating the presence or absence of a medical condition. In some embodiments, for each respective EHR in the plurality of EHRs, the respective scoring representation is a multiplicity metric, and the method includes filtering the plurality of EHRs to remove each respective EHR that does not meet an evaluation threshold (e.g., removing EHRs with high multiplicity scores). For example, in some embodiments, the respective scoring representation is a multiplicity metric, and the evaluation threshold is 200 or less, 100 or less, 90 or less, 80 or less, 70 or less, 60 or less, 50 or less, 40 or less, 30 or less, 20 or less, 10 or less, 5 or less, 3 or less, or 2 or less. In some embodiments, the respective scoring representation is a multiplicity metric, and the evaluation threshold is at least 1, at least 2, at least 5, at least 10, at least 20, at least 50, or at least 100. In some embodiments, the respective scoring representation is a multiplicity metric, and the evaluation threshold is 1-5, 2-10, 1-3, 2-50, 5-100, or 80-200. In some embodiments, the threshold distance falls within another range starting from 1 or more and ending at 200 or less.

[0120] In some embodiments, any one or more of a variety of dimensionality reduction techniques are used, including, but not limited to, principal component analysis (PCA), non-negative matrix factorization (NMF), linear discriminant analysis (LDA), diffusion maps, or network (e.g., neural network) techniques such as autoencoders.

[0121] In some embodiments, the dimensionality reduction is a principal components algorithm, a random projection algorithm, an independent component analysis algorithm, a feature selection method, a factor analysis algorithm, Sammon mapping, curvilinear component analysis, a stochastic neighborhood embedding (SNE) algorithm, an Isomap algorithm, a maximum variance unfolding algorithm, a locally linear embedding algorithm, a t-SNE algorithm, a non-negative matrix factorization algorithm, a kernel principal component analysis algorithm, a graph-based kernel principal component analysis algorithm, a linear discriminant analysis algorithm, a generalized discriminant analysis algorithm, a uniform manifold approximation and projection (UMAP) algorithm, a LargeVis algorithm, a Laplacian eigenmaps algorithm, or Fisher's linear discriminant analysis algorithm. See, for example, Fodor, 2002, “A survey of dimension reduction techniques,” Center for Applied Scientific Computing, Lawrence Livermore National, Technical Report UCRL-ID-148494; Cunningham, 2007, “Dimension Reduction,” University College Dublin, Technical Report UCD-CSI-2007-7; Zahorian et al., 2011, “Nonlinear Dimensionality Reduction Methods for Use with Automatic Speech Recognition,” Speech Technologies. doi:10.5772 / 16863. ISBN 978-953-307-996-7; and Lakshmi et al., 2016, “2016 IEEE 6th International Conference on Advanced Computing (IACC),” pp. 31-34. doi:10.1109 / IACC.2016.16, ISBN978-1-4673-8286-1.

[0122] In some embodiments, the dimensionality reduction is a principal component analysis (PCA) algorithm, and each respective extracted dimensionality reduced component comprises a respective principal component derived by PCA. In such embodiments, the number of principal components in the plurality of principal components may be limited to a threshold number of principal components calculated by the PCA algorithm. In some embodiments, the threshold number of principal components is, for example, at least 2, at least 3, at least 5, at least 10, at least 20, at least 50, at least 100, at least 1000, at least 1500, or any other number. In some embodiments, the threshold number of principal components is 2000 or less, 1000 or less, 100 or less, 50 or less, or 10 or less. In some embodiments, the threshold number of principal components is 2-10, 5-20, 3-50, 20-100, or 100-2000. In some embodiments, the threshold number of principal components falls within another range starting from 2 or more and ending at 2000 or less.

[0123] In some embodiments, the dimensionality reduction includes manifold learning using a respective scoring representation for each respective EHR in the plurality of EHRs. In general, manifold learning is used to account for the low-dimensional structure of high-dimensional data by determining the maximum variation in the dataset. Examples include, but are not limited to, force-directed layout (e.g., Fruchterman, TM, & Reingold, EM (1991). Graph drawing by force-directed placement. Software: Practice and experience, 21(11), 1129-1164) (e.g., Force Atlas 2), t-distributed stochastic neighbor embedding (t-SNE), locally linear embedding (e.g., Roweis, ST, & Saul, LK (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2323-2326), locally linear isometry (ISOMAP, e.g., Tenenbaum, JB, De Silva, V., & Langford, JC (2000). A global geometric framework for nonlinear dimensionality reduction.Science,290(5500),2319-2323), kernel PCA, graph-based kernel PCA, heat diffusion likelihood for affinity-based trajectory embedding (PHATE), generalized discriminant analysis (GDA), uniform manifold approximation and projection (UMAP), and / or kernel discriminant analysis.

[0124] In some embodiments, the dimensionality reduction includes discriminant analysis. In general, force-directed layout is useful in various specific embodiments due to its ability to identify new, lower dimensions that code for non-linear aspects of the underlying data that arise from underlying relationships between data elements. In some implementations, the force-directed layout uses physics-based models as a mechanism to determine the reduced dimensionality that best represents the data, makes few assumptions about the structure of the data, and does not impose a denoising approach. Manifold learning is further described, for example, in Wang et al., 2004, "Adaptive Manifold Learning," Advances in Neural Information Processing Systems 17, which is incorporated herein by reference for all purposes.

[0125] Referring to block 214, in some embodiments, the plurality of criteria includes (i) a first criterion that, when met, excludes the respective subject from the first plurality of subjects, and (ii) a second criterion that, when met, results in inclusion of the respective subject in the first plurality of subjects, provided that no criterion of the plurality of criteria excludes the respective subject from the first plurality of subjects. For example, in some embodiments, for a medical condition that shares many symptoms with another medical condition, identification of a diagnostic code for the other medical condition in the EHR may be used as an exclusion criterion, while identification of a diagnostic code for the target condition may be used as an inclusion criterion.

[0126] In some embodiments, an initial structured phenotype for a medical condition is developed based on a combination of, for example, clinical domain expertise, literature, and guideline review, along with data exploration. In some embodiments, the criteria and logic of this phenotype are discussed and agreed upon with a multidisciplinary team of clinicians, informaticians, and data scientists. In some embodiments, this initial phenotype is then evaluated with a training and validation set chart review to evaluate performance. The review of results includes a non-blind chart review of false positives and false negatives from the training set only, and the validation set is not used to avoid overfitting the model to evaluate how much the phenotype definition and query can be improved. In some embodiments, the structured phenotype is then modified to generate V2, and this cycle of evaluation and iteration continues until no further modifications result in a clear improvement of the phenotype.

[0127] Referring to block 216, in some embodiments, the second plurality of subjects includes at least 10,000 subjects (e.g., subset 308 of training records as illustrated in FIG. 3). In some embodiments, the second plurality of subjects includes at least 2500 subjects, at least 5000 subjects, at least 10,000 subjects, at least 25,000 subjects, at least 50,000 subjects, at least 100,000 subjects, at least 250,000 subjects, at least 500,000 subjects, at least 1 million subjects, at least 2.5 million subjects, at least 5 million subjects, at least 10 million subjects, at least 25 million subjects, at least 50 million subjects, at least 100 million subjects, at least 250 million subjects, at least 500 million subjects, at least 1 billion or more subjects, or more subjects.

[0128] In some embodiments, the second plurality of subjects includes up to 1 billion subjects, up to 500 million subjects, up to 250 million subjects, up to 100 million subjects, up to 50 million subjects, up to 25 million subjects, up to 10 million subjects, up to 5 million subjects, up to 2.5 million subjects, up to 1 million subjects, up to 500,000 subjects, up to 250,000 subjects, up to 100,000 subjects, up to 50,000 subjects, up to 25,000 subjects, up to 10,000 subjects, up to 5000 subjects, up to 2500 subjects, up to 1000 subjects, or fewer subjects.

[0129] In some embodiments, the second plurality of subjects includes 500 to 1 billion subjects, 1000 to 1 billion subjects, 5000 to 1 billion subjects, 10,000 to 1 billion subjects, 50,000 to 1 billion subjects, 100,000 to 1 billion subjects, 500,000 to 1 billion subjects, 1 million to 1 billion subjects, 5 million to 1 billion subjects, 10 million to 1 billion subjects, 50 million to 1 billion subjects, 100 million to 1 billion subjects, or 500 million to 1 billion subjects. In some embodiments, the second plurality of subjects includes 5-250 million subjects, 10-250 million subjects, 50-250 million subjects, 10,000-250 million subjects, 50,000-250 million subjects, 100,000-250 million subjects, 500,000-250 million subjects, 1 million-250 million subjects, 5 million-250 million subjects, 10 million-250 million subjects, 50 million-250 million subjects, or 100 million-250 million subjects. In some embodiments, the second plurality of subjects includes 5-10 million subjects, 10-10 million subjects, 50-10 million subjects, 10,000-10 million subjects, 50,000-10 million subjects, 100,000-10 million subjects, 500,000-10 million subjects, 1 million-10 million subjects, or 5 million-10 million subjects. In some embodiments, the second plurality of subjects includes 5-1 million subjects, 100-1 million subjects, 5000-1 million subjects, 10,000-1 million subjects, 50,000-1 million subjects, 100,000-1 million subjects, or 500,000-1 million subjects. In some embodiments, the second plurality of subjects includes between 500 and 250,000 subjects, between 1000 and 250,000 subjects, between 5000 and 250,000 subjects, between 10,000 and 250,000 subjects, between 50,000 and 250,000 subjects, or between 100,000 and 250,000 subjects.In some embodiments, the second plurality of subjects includes between 500 and 100,000 subjects, between 1000 and 100,000 subjects, between 5000 and 100,000 subjects, between 10,000 and 100,000 subjects, or between 50,000 and 100,000 subjects.

[0130] Referring to block 218, in some embodiments, the medical condition is pulmonary hypertension (PH). PH is high blood pressure that affects the arteries of the right side of the heart and the lungs. In one type of PH, pulmonary arterial hypertension (PAH), high blood pressure constricts the arteries in the lungs, reducing the oxygen levels in the blood. PH is estimated to affect approximately 1% of the world's population. For more information on PH, see, for example, Mandras SA, et al., "Pulmonary Hypertension: A Brief Guide for Clinicians" Mayo Clin Proc., 95(9):1978-88 (2020), which is incorporated herein by reference in its entirety.

[0131] In some embodiments, the medical condition is cancer. In some embodiments, the medical condition is a particular type of cancer and / or a particular stage of cancer. Examples of cancer include, but are not limited to, carcinoma, lymphoma, blastoma, glioblastoma, sarcoma, and leukemia. In some embodiments, non-limiting cancers include breast cancer, squamous cell carcinoma, lung cancer (including small cell lung cancer, non-small cell lung cancer (NSCLC), adenocarcinoma of the lung, squamous cell carcinoma of the lung (e.g., squamous cell NSCLC)), various types of head and neck cancer (e.g., HNSC), cancer of the peritoneum, hepatocellular carcinoma, gastric or stomach cancer (including gastrointestinal cancer), pancreatic cancer, ovarian cancer, cervical cancer, liver cancer, bladder cancer, hepatoma, colon cancer, colorectal cancer, endometrial or uterine cancer, salivary gland cancer, kidney or renal cancer, liver cancer, prostate cancer, vulvar cancer, thyroid cancer, and hepatocellular carcinoma. In some embodiments, non-limiting cancers further include B-cell lymphomas (including low-grade / follicular non-Hodgkin's lymphoma (NHL), small lymphocytic (SL) NHL, intermediate-grade / follicular NHL, intermediate-grade diffuse NHL, high-grade immunoblastic NHL, high-grade lymphoblastic NHL, high-grade small non-incisional cell NHL, mass NHL, mantle cell lymphoma, AIDS-related lymphoma, and Waldenstrom's macroglobulinemia), chronic lymphocytic leukemia (CLL), acute lymphoblastic leukemia (ALL), hairy cell leukemia, chronic myeloid leukemia, and post-transplant lymphoproliferative disorder (PTLD). In some embodiments, non-limiting cancers further include abnormal blood vessel proliferation associated with phakomatosis, edema (such as associated with brain tumors), and / or Meigs syndrome.

[0132] As will be well understood by those skilled in the art, the term "cancer" for use in the disclosed method and system is not limited to only the primary form of cancer, but also includes cancer subtypes. Some such cancer subtypes are listed above, but also include breast cancer subtypes such as Luminal A (hormone receptor (HR) + / human epidermal growth factor receptor (HER2)-), Luminal B (HR + / HER2 +), triple negative or (HR- / HER2-) and HER2 positive. Other cancer subtypes include various lung cancers listed above, and prostate cancer subtypes with subsets of alterations defined by mutations in E26 transformation-specific genes (ETS, specifically ERG, ETV1 / 4, and FLI1 genes), and FOXA1, SPOP, and IDH1 genes.

[0133] In some embodiments, the goal of method 200 is to identify instances of a medical condition by capturing a diverse set of different multimodal signals that indicate the presence of a medical condition. In some embodiments, each of these signals is represented by an individual one-class classifier, referred to herein as an "indicator function." An indicator function is a simple set of rules intended to pattern match against a particular signal that indicates the presence or absence of a medical condition. In some embodiments, if the set of rules (criteria) for a particular indicator function is met, the indicator function votes for a predefined class, e.g., class 0, indicating the absence of a medical condition or no detection of a medical condition (e.g., if the absence of a medical condition cannot be categorically confirmed), or class 1, indicating the presence of a medical condition. In some embodiments, if the set of criteria for a particular indicator function is not met, the indicator function refrains from voting, i.e., the evaluation of the particular indicator function does not contribute to the final decision of whether the medical record indicates the presence or absence of a medical condition. In some embodiments, the indicator function may only vote for or refrain from voting for a given class, e.g., a class corresponding to the presence of a medical condition, or a class corresponding to the absence of a medical condition or an indicator of a medical condition. These rules may include any number of sets of conditions. For example, for pulmonary hypertension, the criteria may include a regular expression such as "pulm.{0,10}hypertension" that matches the text data, the presence of a code indicative of pulmonary hypertension in a structured field, a test value that meets a threshold that indicates pulmonary hypertension, etc.

[0134] Thus, referring to block 220, in some embodiments, the method 200 includes generating a plurality of indicator functions 61 for the medical condition (e.g., illustrated as step 360 in FIG. 3 ), each respective indicator function in the plurality of indicator functions including a corresponding set of one or more criteria 62 that, when met, indicates the presence or absence of the medical condition.

[0135] Referring to block 222, in some embodiments, the multiple indicator functions include a respective indicator function 61 for each respective subgroup within the multiple subgroups of subjects having a medical condition. For example, as illustrated in FIG. 4B, different subgroups 404 of subjects having a medical condition will have different indicators of the medical condition in their EHR. Different indicator functions 406 may be used to identify these different subgroups 404 of subjects having a medical condition. For example, subgroup 404-1 may be identified as having a medical condition based on information in the unstructured clinical notes portion of their EHR. Thus, indicator function 406-1 having one or more criteria 62 that may be satisfied by information stored in the unstructured clinical notes may be used to identify all or a portion of subgroup 404-1, but is not used to identify patients in subgroup 404-2 that have an indicator of the medical condition in the test results stored in their EHR. Thus, one or more indicator functions (e.g., indicator functions 406-2 and 406-3) may be generated to identify subjects having a medical condition based on the test results. Similarly, one or more indicator functions (eg, LF 406-4) may be generated to identify subjects 404-3 having a medical condition based on medications recommended or prescribed for the subject.

[0136] Referring to block 224, in some embodiments, the plurality of indicator functions includes a first respective indicator function 61 that includes a corresponding set of one or more criteria 62 that, when met, indicates the presence of a medical condition. For example, the criteria may be met by identification of a medical diagnosis or a billing code for the medical condition.

[0137] Referring to block 226, in some embodiments, generating the plurality of indicator functions includes, for a first respective indicator function 61, i) defining a first set of one or more criteria 62, the set of criteria 62 indicating the presence of a medical condition when met; and ii) determining, for each respective subject in a first subset of the first plurality of subjects, whether a corresponding medical record for the respective subject satisfies the first set of one or more criteria (e.g., evaluating 362 the first respective indicator function across a subset of the plurality of training candidates 308 as illustrated in FIG. 3 ). ii) for each subject in the first subset of the first plurality of subjects determined to satisfy the first set of one or more criteria, evaluating whether the corresponding medical record indicates the absence of the medical condition, independent of determining whether the corresponding medical record satisfies the first set of one or more criteria; and iv) upon determining that the corresponding medical record indicates the absence of the medical condition, revising the first set of one or more criteria to one that is less inclusive (e.g., returning to constructing the indicator function step 360 if the indicator function misidentifies the sample EHR as indicating the presence of the medical condition, as illustrated in FIG. 3). Referring to block 228, in some embodiments, the revising includes adding an exclusionary criterion to the first set of one or more criteria. For example, if the target medical disorder is pulmonary hypertension (commonly abbreviated as PH), a mention of "Ph.D." in the medical record may satisfy criterion 62 such that the record is identified as indicating the presence of PH. By identifying such records, an indicator function 61 having criteria 62 met by "Ph.D." may be modified to exclude "Ph.D." from meeting the inclusion criteria.

[0138] Referring to block 230, in some embodiments, the plurality of indicator functions 61 includes a second respective indicator function that includes a corresponding set of one or more criteria 62 that, when met, indicates the absence of a medical condition, e.g., a criterion that is met by identification of a medical diagnosis or billing code for another medical condition that shares one or more symptoms with the target medical condition.

[0139] Referring to block 232, in some embodiments, generating the plurality of indicator functions includes, for a second respective indicator function 61, i) defining a second set of one or more criteria 62, the set of criteria 62 indicating an absence of a medical condition when met; ii) determining, for each respective subject in a second subset of the first plurality of subjects, whether a corresponding medical record satisfies the second set of one or more criteria (e.g., evaluating 362 the second respective indicator function across a subset of the plurality of training candidates 308 as illustrated in FIG. 3 ); and iii) evaluating the second respective indicator function 61. For each subject in the second subset of the first plurality of subjects determined to satisfy the second set of one or more criteria, evaluating whether the corresponding medical record indicates the presence of the medical condition independent of determining whether the respective medical record satisfies the second set of one or more criteria, and iv) upon determining that the respective medical record indicates the presence of the medical condition, revising the second set of one or more criteria to be more inclusive (e.g., returning to constructing the indicator function 360 if the indicator function misidentifies the sample EHR as indicating the absence of the medical condition as illustrated in FIG. 3). Referring to block 234, in some embodiments, the revising includes adding an inclusive criterion to the second set of one or more criteria. For example, if the target medical disorder is pulmonary hypertension (commonly abbreviated as PH), an EHR that includes a medical diagnosis or billing code for cardiac disease may meet the exclusion criterion 62 such that the record would be misidentified as indicating the absence of PH even if test results indicate the subject has a mean pulmonary artery pressure of greater than 20 mmHg, and the indicator function 61 may be modified to include the inclusion criterion 62 that is met by a mean pulmonary artery pressure of greater than 20 mmHg.

[0140] Referring to block 236, in some embodiments the plurality of indicator functions includes at least 10 indicator functions. In some embodiments the plurality of indicator functions includes at least 3 indicator functions, at least 4 indicator functions, at least 5 indicator functions, at least 6 indicator functions, at least 7 indicator functions, at least 8 indicator functions, at least 9 indicator functions, at least 10 indicator functions, at least 15 indicator functions, at least 20 indicator functions, at least 25 indicator functions, at least 30 indicator functions, at least 40 indicator functions, at least 50 indicator functions, or more indicator functions.

[0141] In some embodiments, the dimensionality reduction includes discriminant analysis. In general, force-directed layout is useful in various specific embodiments due to its ability to identify new, lower dimensions that code for non-linear aspects of the underlying data that arise from underlying relationships between data elements. In some implementations, the force-directed layout uses physics-based models as a mechanism to determine the reduced dimensionality that best represents the data, makes few assumptions about the structure of the data, and does not impose a denoising approach. Manifold learning is further described, for example, in Wang et al., 2004, "Adaptive Manifold Learning," Advances in Neural Information Processing Systems 17, which is incorporated herein by reference for all purposes.

[0142] Referring to block 236, the method 200 also includes, for each respective subject in the first plurality of subject medical records in the first plurality of medical records (e.g., the subset of training records 308 as illustrated in FIG. 3 ), assigning a corresponding indicator 47 indicative of a status of the medical condition by evaluating first information (e.g., one or more of structured electronic health record (EHR) data 42, unstructured EHR data 43, test results 46, prescribed medications 44, and performed medical procedures 45) from each subject's corresponding medical record using an ensemble model including multiple indicator functions 61, and obtaining a prediction 47 for the status of the medical condition as output from the ensemble model, wherein the evaluation includes natural language processing of at least a portion of the respective medical records.

[0143] In some embodiments, the criteria 62 are evaluated using a string matching algorithm. In general, a string matching algorithm (e.g., a string search algorithm) performs a function in which one or more substrings are identified within a larger string. The function further identifies the location where the substring is located within the larger string. A string includes, but is not limited to, a sequence of text (e.g., an EHR, a section, a text span, or a portion thereof). String matching is further described in Bulus et al., 2017, “Comparison of String-Matching Algorithms in Web Documents,” UniTech 2017, which is incorporated herein by reference in its entirety. In some embodiments, the string matching algorithm includes a naive string search, a finite-state automaton-based search, a Rabin-Karp algorithm, a Knuth-Morris-Pratt algorithm, a Boyer-Moore string search algorithm, a two-way string matching algorithm, or a backward non-deterministic directed acyclic string graph matching algorithm using a computer system including a processor coupled to a non-transient memory. In some embodiments, the string matching is an exact string match. That is, in some embodiments, the algorithm must find an exact match of a given string from the criteria in the EHR (e.g., a curated list of words, medical codes, and / or phrases) to satisfy the criteria. In some embodiments, fuzzy matching logic is used to allow some mismatch between the representation in the EHR and the representation in the curated list of words, medical codes, and / or phrases, for example, by applying a threshold to the string distance between the representation in the EHR and the representation in the curated list. Methods for determining string distances, such as Levenshtein distance, Hamming distance, Jaro-Winkler distance, and the like, are known in the art. In some embodiments, wildcards are used when matching to a curated list to account for different forms and / or inflections of a word, for example, when the curated list includes one or more regular expressions (RegEx).In some embodiments, the string distance threshold is between 0 (e.g., exact match) and 1 (e.g., most tolerant). For example, in some embodiments, the string distance threshold is at least 0.1, at least 0.2, at least 0.3, at least 0.4, at least 0.5, at least 0.6, at least 0.7, or at least 0.8. In some embodiments, the string distance threshold is 0.99 or less, 0.95 or less, 0.9 or less, 0.8 or less, 0.7 or less, 0.6 or less, 0.5 or less, 0.4 or less, 0.3 or less, or 0.2 or less. In some embodiments, the string distance threshold is between 0.1 and 0.6, 0.1 and 0.4, 0.2 and 0.8, 0.4 and 0.95, or 0.3 and 0.7. In some embodiments, the string distance threshold falls within another range between 0 and 1.

[0144] In some embodiments, the criteria 62 are evaluated by exploring one or more latent variables learned by the model. For example, in some embodiments, a natural language processing model is trained to distinguish between EHRs from subjects with a particular medical condition and EHRs from subjects without the medical condition. Then, rather than matching expressions in the EHR to expressions in a curated list of words, medical codes, and / or phrases, part or all of the EHR is scored using the model to provide an indication of whether the EHR contains expressions that indicate the presence of a medical condition.

[0145] Any one of several clustering techniques may be used, including but not limited to hierarchical clustering, k-means clustering, and density-based clustering. In some embodiments, the clustering algorithm is density-based spatial clustering of noisy applications (DBSCAN). In some embodiments, a hierarchical density-based clustering algorithm is used (referred to as HDBSCAN, see, e.g., Campello et al., 2015, “Hierarchical density estimates for data clustering, visualization, and outlier detection,” ACM Trans Knowl Disc Data, 10(1), 5). In another embodiment, a community detection-based cluster algorithm such as Louvain clustering is used (see, e.g., Blondel et al., 2008, “Fast unfolding of communities in large networks,” J stat mech:theor exp, 2008(10), P10008). In yet another embodiment, Leyden clustering is used. See, e.g., Traag et al., (2019), “From Louvain to Leiden: guaranteeing well-connected communities,” Sci Rep 9:5233, doi:10.1038 / s41598-019-41695-z. In yet another embodiment, a diffusion path algorithm is used. In some embodiments, the clustering algorithm includes a pre-trained language model and / or an embedding model. In some embodiments, the clustering algorithm further includes applying a dimensionality reduction technique to the data from the EHR. In some implementations, each part in the set of parts of the EHR is represented as a vector, such as a dense representation.

[0146] In some embodiments, the language model is a pre-trained language model. In some embodiments, the language model is a masked language modeling (MLM) model. In some embodiments, the language model is BERT. See, e.g., Devlin et al., 2018, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv:1810.04805v2). In some embodiments, the corresponding evaluation metric is a multiplicity metric. In general, multiplicity refers to a metric of how well a language model predicts a sample, such as an EHR indicating the presence or absence of a medical condition. For example, in some embodiments, multiplicity is determined as the inverse probability of a test set indicating the presence or absence of a medical condition, normalized by the number of words evaluated in the EHR. A lower multiplicity indicates a higher probability such that the model can better predict an EHR indicating the presence or absence of a medical condition. In some embodiments, for each respective EHR in the plurality of EHRs, the respective scoring representation is a multiplicity metric, and the method includes filtering the plurality of EHRs to remove each respective EHR that does not meet an evaluation threshold (e.g., removing EHRs with high multiplicity scores). For example, in some embodiments, the respective scoring representation is a multiplicity metric, and the evaluation threshold is 200 or less, 100 or less, 90 or less, 80 or less, 70 or less, 60 or less, 50 or less, 40 or less, 30 or less, 20 or less, 10 or less, 5 or less, 3 or less, or 2 or less. In some embodiments, the respective scoring representation is a multiplicity metric, and the evaluation threshold is at least 1, at least 2, at least 5, at least 10, at least 20, at least 50, or at least 100. In some embodiments, the respective scoring representation is a multiplicity metric, and the evaluation threshold is 1-5, 2-10, 1-3, 2-50, 5-100, or 80-200. In some embodiments, the threshold distance falls within another range starting from 1 or more and ending at 200 or less.

[0147] In some embodiments, any one or more of a variety of dimensionality reduction techniques are used, including, but not limited to, principal component analysis (PCA), non-negative matrix factorization (NMF), linear discriminant analysis (LDA), diffusion maps, or network (e.g., neural network) techniques such as autoencoders.

[0148] In some embodiments, the dimensionality reduction is a principal components algorithm, a random projection algorithm, an independent component analysis algorithm, a feature selection method, a factor analysis algorithm, Sammon mapping, curvilinear component analysis, a stochastic neighborhood embedding (SNE) algorithm, an Isomap algorithm, a maximum variance unfolding algorithm, a locally linear embedding algorithm, a t-SNE algorithm, a non-negative matrix factorization algorithm, a kernel principal component analysis algorithm, a graph-based kernel principal component analysis algorithm, a linear discriminant analysis algorithm, a generalized discriminant analysis algorithm, a uniform manifold approximation and projection (UMAP) algorithm, a LargeVis algorithm, a Laplacian eigenmaps algorithm, or Fisher's linear discriminant analysis algorithm. See, e.g., Fodor, 2002, “A survey of dimension reduction techniques,” Center for Applied Scientific Computing, Lawrence Livermore National, Technical Report UCRL-ID-148494; Cunningham, 2007, “Dimension Reduction,” University College Dublin, Technical Report UCD-CSI-2007-7; Zahorian et al., 2011, “Nonlinear Dimensionality Reduction Methods for Use with Automatic Speech Recognition,” Speech Technologies. doi:10.5772 / 16863. ISBN 978-953-307-996-7; and Lakshmi et al., 2016, “2016 IEEE 6th International Conference on Advanced Please refer to “International Association of Computing (IACC),” pp. 31-34. doi:10.1109 / IACC.2016.16, ISBN978-1-4673-8286-1.

[0149] In some embodiments, the dimensionality reduction is a principal component analysis (PCA) algorithm, and each respective extracted dimensionality reduced component comprises a respective principal component derived by PCA. In such embodiments, the number of principal components in the plurality of principal components may be limited to a threshold number of principal components calculated by the PCA algorithm. In some embodiments, the threshold number of principal components is, for example, at least 2, at least 3, at least 5, at least 10, at least 20, at least 50, at least 100, at least 1000, at least 1500, or any other number. In some embodiments, the threshold number of principal components is 2000 or less, 1000 or less, 100 or less, 50 or less, or 10 or less. In some embodiments, the threshold number of principal components is 2-10, 5-20, 3-50, 20-100, or 100-2000. In some embodiments, the threshold number of principal components falls within another range starting from 2 or more and ending at 2000 or less.

[0150] In some embodiments, the dimensionality reduction includes manifold learning using a respective scoring representation for each respective EHR in the plurality of EHRs. In general, manifold learning is used to account for the low-dimensional structure of high-dimensional data by determining the maximum variation in the dataset. Examples include, but are not limited to, force-directed layout (e.g., Fruchterman, TM, & Reingold, EM (1991). Graph drawing by force-directed placement. Software: Practice and experience, 21(11), 1129-1164) (e.g., Force Atlas 2), t-distributed stochastic neighbor embedding (t-SNE), locally linear embedding (e.g., Roweis, ST, & Saul, LK (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2323-2326), locally linear isometry (ISOMAP, e.g., Tenenbaum, JB, De Silva, V., & Langford, JC (2000). A global geometric framework for nonlinear dimensionality reduction.Science,290(5500),2319-2323), kernel PCA, graph-based kernel PCA, heat diffusion likelihood for affinity-based trajectory embedding (PHATE), generalized discriminant analysis (GDA), uniform manifold approximation and projection (UMAP), and / or kernel discriminant analysis.

[0151] Referring to block 238, in some embodiments, the first plurality of subjects includes at least 500 subjects. In some embodiments, the first plurality of subjects includes at least 1000 subjects, at least 2500 subjects, at least 5000 subjects, at least 10,000 subjects, at least 25,000 subjects, at least 50,000 subjects, at least 100,000 subjects, at least 250,000 subjects, at least 500,000 subjects, at least 1 million subjects, at least 2.5 million subjects, at least 5 million subjects, at least 10 million subjects, at least 25 million subjects, at least 50 million subjects, at least 100 million subjects, at least 250 million subjects, at least 500 million subjects, at least 1 billion or more subjects, or more subjects.

[0152] In some embodiments, the first plurality of subjects includes up to 1 billion subjects, up to 500 million subjects, up to 250 million subjects, up to 100 million subjects, up to 50 million subjects, up to 25 million subjects, up to 10 million subjects, up to 5 million subjects, up to 2.5 million subjects, up to 1 million subjects, up to 500,000 subjects, up to 250,000 subjects, up to 100,000 subjects, up to 50,000 subjects, up to 25,000 subjects, up to 10,000 subjects, up to 5000 subjects, up to 2500 subjects, up to 1000 subjects, or fewer subjects.

[0153] In some embodiments, the first plurality of subjects includes 500 to 1 billion subjects, 1000 to 1 billion subjects, 5000 to 1 billion subjects, 10,000 to 1 billion subjects, 50,000 to 1 billion subjects, 100,000 to 1 billion subjects, 500,000 to 1 billion subjects, 1 million to 1 billion subjects, 5 million to 1 billion subjects, 10 million to 1 billion subjects, 50 million to 1 billion subjects, 100 million to 1 billion subjects, or 500 million to 1 billion subjects. In some embodiments, the first plurality of subjects includes 5-250 million subjects, 10-250 million subjects, 50-250 million subjects, 10,000-250 million subjects, 50,000-250 million subjects, 100,000-250 million subjects, 500,000-250 million subjects, 1 million-250 million subjects, 5 million-250 million subjects, 10 million-250 million subjects, 50 million-250 million subjects, or 100 million-250 million subjects. In some embodiments, the first plurality of subjects includes 5-10 million subjects, 10-10 million subjects, 50-10 million subjects, 10,000-10 million subjects, 50,000-10 million subjects, 100,000-10 million subjects, 500,000-10 million subjects, 1 million-10 million subjects, or 5 million-10 million subjects. In some embodiments, the first plurality of subjects includes 5-1 million subjects, 100-1 million subjects, 5000-1 million subjects, 10,000-1 million subjects, 50,000-1 million subjects, 100,000-1 million subjects, or 500,000-1 million subjects. In some embodiments, the first plurality of subjects includes between 500 and 250,000 subjects, between 1000 and 250,000 subjects, between 5000 and 250,000 subjects, between 10,000 and 250,000 subjects, between 50,000 and 250,000 subjects, or between 100,000 and 250,000 subjects.In some embodiments, the first plurality of subjects includes between 500 and 100,000 subjects, between 1000 and 100,000 subjects, between 5000 and 100,000 subjects, between 10,000 and 100,000 subjects, or between 50,000 and 100,000 subjects.

[0154] In some embodiments, the corresponding prediction 47 is a probability that the medical record indicates the presence of a medical condition. In some embodiments, the corresponding prediction 47 is a binary indicator (e.g., yes / no and / or 0 / 1) that the medical record indicates the presence of a medical condition.

[0155] Each individual indicator function 61 will capture different, optionally multimodal features that correlate with the target medical condition. For example, referring again to FIG. 4B, indicator function 406-1 captures features in unstructured clinical notes that indicate the presence of a medical condition, while indicator function 406-4 captures features related to medications recommended or prescribed to the subject that indicate the presence of the medical condition. As a result, a single indicator function will not fully capture all cases, nor will it be completely accurate, meaning that the indicator function may be considered a weak indicator. These indicator functions may also contradict each other at any given data point. Structured phenotyping approaches rely on subject matter expert (SME) input to integrate these potentially contradictory views, but there are challenges in stitching all these different features together, especially as the number of indicator functions increases. For example, if a careful sampling approach is not in place, the final structured phenotype may overfit a particular data set after several iterations of fine-tuning.

[0156] In some embodiments, a majority voting model is used to aggregate signals from multiple indicator functions. In majority voting, each indicator function votes for an indicator for a given data point, and the indicator with the majority votes is accepted as the official indicator. However, simple majority voting does not take into account information about the accuracy and reliability of the recognized indicator functions when weighting these votes. However, some approaches are known that take these attributes into account. For example, Ratner A. et al., “Snorkel: Rapid Training Data Creation with Weak Supervision,” PVLDB, 11(3) (2017), which is incorporated herein by reference in its entirety, is an aggregation approach that can often provide slightly better results than majority voting.

[0157] Thus, referring to block 240, in some embodiments, the ensemble model includes an aggregate voting model based on an evaluation of each respective indicator function. Referring to block 242, in some embodiments, the aggregate voting model is a weighted voting model.

[0158] Referring to block 244, in some embodiments, the ensemble model includes a probabilistic graphical model (PGM). In some embodiments, the PGM is a generalized case of the majority voting model, where each indicator function is assigned an independent prior distribution. An example of such a PGM is provided in Example 4. For a review of PGM models, see, for example, Koller and Friedman, "Probabilistic Graphical Models," Massachusetts: MIT Press (2009), ISBN 978-0-262-01319-2, which is incorporated herein by reference in its entirety.

[0159] In some embodiments, PGM may be solved as follows:

number

number

number

[0160] Referring to block 246, in some embodiments, evaluating the ensemble model including the multiple indicator functions includes, for each respective subject in the first plurality of subjects, evaluating a first data type for a first respective criterion in the multiple criteria and evaluating a second data type for a second respective criterion in the multiple criteria.

[0161] Referring to block 248, in some embodiments, the first type of data is structured data in an electronic health record (EHR) and the second type of data is unstructured data in the EHR, in some embodiments, the structured data is medical billing codes or diagnosis codes and the unstructured data is clinical notes in the EHR.

[0162] Referring to block 250, in some embodiments, applying the extraction filter includes evaluating at least three data types selected from structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, the unstructured EHR data 43, and lab results 46. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, the unstructured EHR data 43, and prescribed medications 44. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, the unstructured EHR data 43, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, the unstructured EHR data 43, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, lab results 46, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, lab results 46, and prescribed medications 44. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, lab results 46, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating lab results 46, prescribed medications 44, and performed medical procedures 45.

[0163] In some embodiments, applying the extraction filter (e.g., filtering 354 illustrated in FIG. 3 ) includes evaluating at least four data types selected from structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating the structured electronic health record (EHR) data 42, unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, and performed medical procedures 45. In some embodiments, applying the extraction filter includes evaluating structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, and prescribed medications 44.

[0164] In some embodiments, applying an extraction filter (e.g., filtering 354 illustrated in FIG. 3 ) includes evaluating at least structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45.

[0165] With reference to block 252, in some embodiments, applying the extraction filter includes natural language processing of the unstructured clinical notes from the EHR for each subject in the plurality of subjects. With reference to block 254, in some embodiments, the natural language processing includes matching one or more regular expressions against the corresponding unstructured clinical notes. With reference to block 256, in some embodiments, the natural language processing includes inputting at least a portion of the unstructured clinical notes into a machine learning model trained to identify language associated with clinical conditions. With reference to block 258, in some embodiments, the natural language processing includes inputting at least a portion of the unstructured clinical notes into a large-scale language model.

[0166] While some degree of signaling can be achieved through the aggregation of indicator functions, they tend to generate a limited set of indicator training data, i.e., have limited coverage. In some embodiments, to further expand the set of indicators in the training dataset, weakly supervised techniques are used to train a machine learning model, e.g., a natural language processing (NLP), transformer-based deep learning phenotyping model. In some embodiments, this is an NLP model that learns to extract the same information found in the multimodal aggregate signal by looking exclusively at clinical notes. Training this model allows for matching of the aggregate signal and the results of the phenotyping model, yet still produces significantly more coverage. For example, returning to FIG. 4, the aggregation of indicator functions in FIG. 4C may lead to the identification of various subpopulations of subjects with a medical condition, but other subpopulations may be missed, as illustrated in FIG. 4D. By training a machine learning model to identify cases of a medical condition using weak indicators generated by aggregating multiple indicator functions, these other subpopulations may be identified, as illustrated in FIG. 4E.

[0167] Thus, referring to block 258, in some embodiments, the method 200 also includes, for each respective subject in the first plurality of subjects, training a classification model to phenotype the subject with respect to a medical condition using (i) the corresponding label as a dependent variable, and (ii) second information from the corresponding medical record as an independent variable.

[0168] Referring to block 260, in some embodiments, the classification model is a neural network, a support vector machine, a naive Bayes model, a nearest neighbor model, a boosted trees model, a random forest model, or a clustering model.

[0169] Referring to block 262, in some embodiments the second information includes at least two data types: Referring to block 264, in some embodiments the second information includes structured Electronic Health Record (EHR) data and unstructured EHR data.

[0170] Referring to block 266, in some embodiments, the second information includes at least three data types selected from structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, unstructured EHR data 43, and lab results 46. In some embodiments, the second information includes structured electronic health record (EHR) data 42, unstructured EHR data 43, and prescribed medications 44. In some embodiments, the second information includes structured electronic health record (EHR) data 42, unstructured EHR data 43, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, unstructured EHR data 43, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, lab results 46, and medical procedures performed 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, prescribed medications 44, and medical procedures performed 45. In some embodiments, the second information includes unstructured EHR data 43, lab results 46, and prescribed medications 44. In some embodiments, the second information includes unstructured EHR data 43, lab results 46, and medical procedures performed 45. In some embodiments, the second information includes unstructured EHR data 43, prescribed medications 44, and medical procedures performed 45. In some embodiments, the second information includes unstructured EHR data 43, prescribed medications 44, and medical procedures performed 45. In some embodiments, the second information includes lab results 46, prescribed medications 44, and medical procedures performed 45.

[0171] In some embodiments, the second information includes at least four data types selected from structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, the second information includes unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, unstructured EHR data 43, prescribed medications 44, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, and performed medical procedures 45. In some embodiments, the second information includes structured electronic health record (EHR) data 42 , unstructured EHR data 43 , lab results 46 , and prescribed medications 44 .

[0172] In some embodiments, the second information includes at least structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and medical procedures performed 45.

[0173] In some embodiments, the second information includes at least one of structured electronic health record (EHR) data 42, unstructured EHR data 43, lab results 46, prescribed medications 44, and performed medical procedures 45. In some embodiments, the second information includes at least lab results 46.

[0174] Referring to block 268, in some embodiments the second information includes a data type that is not part of the first information. Referring to block 270, in some embodiments the data type of the second information that is not part of the first information is a test result.

[0175] Referring to block 272, in some embodiments, the second information includes all or a portion of the first information.

[0176] Referring to block 274, in some embodiments, the medical condition is pulmonary hypertension and the second information includes electrocardiogram results.

[0177] Referring to block 274, in some embodiments, the method 200 also includes identifying, from the plurality of previously undiagnosed subjects, one or more respective subjects having the medical condition by, for each respective subject of the plurality of subjects, inputting the corresponding second information for the respective subject into the classification model and receiving as output a corresponding indication of whether the respective subject has the medical condition.

[0178] Referring to block 276, in some embodiments, the plurality of subjects are associated with a health care provider. For example, in some embodiments, the plurality of subjects are all associated with a health care insurance company. In some embodiments, the plurality of subjects are all associated with a health care provider (e.g., a hospital or clinic) or a chain of health care providers (e.g., a group of hospitals or clinics).

[0179] Referring to block 278, in some embodiments, the method 200 also includes predicting whether the test subject has the medical condition by inputting second information about the test subject into the classification model and receiving as an output an indication of whether the test subject has the medical condition.

[0180] In some embodiments, the methods described herein also include generating a report about the subject having one or more phenotypes, including the subject's identified phenotype (e.g., the presence or absence of PH). In some embodiments, the report further includes information regarding clinical trials for which the subject is eligible, treatments that are specific to the subject's medical condition, and / or possible adverse therapeutic effects associated with particular features of the subject's medical condition, such as the subject's genetic mutations, epigenetic abnormalities, associated oncogenic pathogenic infections, and / or pathological abnormalities, or other features of the subject's sample and / or clinical records.

[0181] In some embodiments, the methods further comprise using the identified phenotype to determine the subject's eligibility for enrollment in a clinical trial.

[0182] In one aspect, the present disclosure provides a method, system, and non-transitory computer-readable medium for training a model for phenotyping a subject for a medical condition. The method is implemented in a computer system including one or more processors and a memory. The method includes generating a plurality of indicator functions for the medical condition, each respective indicator function in the plurality of indicator functions including a corresponding set of one or more criteria, the corresponding set of one or more criteria indicating the presence of the medical condition when met. The method includes assigning a corresponding indicator to each subject indicative of the status of the medical condition by evaluating first information from a corresponding medical record of each respective subject in the first plurality of subjects using the ensemble model to obtain as an output a prediction of the status of the medical condition from an ensemble model including the plurality of indicator functions, the evaluating including natural language processing of at least a portion of the respective medical record.

[0183] Digital and Laboratory Medical Platform In some embodiments, the methods and systems described herein are utilized in combination with or as part of a digital and laboratory medical platform that is generally targeted to medical care and research. It should be understood that many applications of the methods and systems described above in combination with such platforms are possible. One example of such a platform is described in U.S. Patent Publication No. 2021 / 0090694, entitled "Data Based Cancer Research and Treatment Systems and Methods," published March 25, 2021, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0184] For example, implementations of one or more embodiments of the methods and systems described above include microservices constituting a digital and laboratory medical platform that supports the analysis of electrocardiograms (ECGs) to provide clinical support for personalized cardiovascular therapy. An embodiment may include a single microservice for performing analysis of ECGs and delivering to clinical support for personalized cardiovascular therapy, and / or may include multiple microservices, each with a specific role, that together implement one or more of the above embodiments. In one example, a first microservice performs an ECG analysis to deliver features to a second microservice to curate clinical support for personalized cardiovascular therapy based on the identified features. Similarly, the second microservice performs a medical analysis of the curated clinical support to deliver a recommended treatment modality according to various embodiments described herein.

[0185] When the above embodiments are implemented using or as part of a digital and laboratory medical platform, one or more of such microservices may, in some implementations, be part of an order management system that orchestrates the sequence of events as necessary at the appropriate time and in the appropriate sequence necessary to instantiate the above embodiments. A microservices-based order management system is disclosed, for example, in U.S. Patent Publication No. 2020 / 80365232, entitled "Adaptive Order Fulfillment and Tracking Methods and Systems," published on November 19, 2020, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0186] For example, continuing with the first and second microservices above, the order management system, in some implementations, notifies the first microservice that an order for curating clinical support for personalized cardiovascular therapy has been received and is ready for processing. In some implementations, the first microservice executes and notifies the order management system when delivery of ECG features for the patient is ready for the second microservice. Further, the order management system identifies that execution parameters (preconditions) for the second microservice have been met, including that the first microservice has completed, in accordance with various embodiments described herein, and notifies the second microservice that it may continue to process the order for curating clinical support for personalized cardiovascular therapy.

[0187] When the digital and laboratory medicine platform further comprises a genetic analyzer system, the genetic analyzer system, in some embodiments, comprises a target panel and / or a sequencing probe. Examples of target panels are disclosed, for example, in U.S. Patent Publication No. 2021 / 0090694, entitled "Data Based Cancer Research and Treatment Systems and Methods," published March 25, 2021, which is incorporated by reference in its entirety for all purposes. Examples of targeted panels for sequencing cell-free (cf)DNA and determining various characteristics of a specimen based on the sequencing can be found, for example, in U.S. patent application Ser. No. 17 / 179,086, filed Feb. 18, 2021, entitled "Methods And Systems For Dynamic Variant Thresholding In A Liquid Biopsy Assay," U.S. patent application Ser. No. 17 / 179,267, filed Feb. 18, 2021, entitled "Estimation Of Circulating Tumor Fraction Using Off-Target Reads Of Targeted-Panel Sequencing," and U.S. patent application Ser. No. 17 / 179,267, filed Feb. 18, 2021, entitled "Methods And Systems For Refining Copy Number Variation In A Liquid Biopsy Assay," and U.S. patent application Ser. No. 17 / 179,279, filed February 18, 2021, entitled "Personalized Cancer Assay," each of which is incorporated by reference in its entirety for all purposes. In one example, the targeted panel, according to various embodiments described herein, enables delivery of next-generation sequencing results to provide clinical support for personalized cancer therapy.Examples of next-generation sequencing probe designs are disclosed, for example, in U.S. Patent Publication No. 2021 / 0115511, published on June 22, 2021, entitled "Systems and Methods for Next Generation Sequencing Uniform Probe Design," and U.S. Patent Application No. 17 / 323,986, filed on May 18, 2021, entitled "Systems and Methods for Next Generation Sequencing Uniform Probe Design," each of which is incorporated by reference in its entirety into this specification for all purposes.

[0188] When the digital and laboratory medicine platform further includes an epigenetic analyzer system, the epigenetic analyzer system, in some embodiments, analyzes the specimen to determine its epigenetic signature and further uses the information to monitor the patient over time. Examples of epigenetic analyzer systems are disclosed, for example, in U.S. Patent Application No. 17 / 352,231, filed June 18, 2021, entitled "Molecular Response and Progression Detection From Circulating Cell Free DNA," which is incorporated by reference in its entirety for all purposes.

[0189] When the digital and laboratory medicine platform further includes a bioinformatics pipeline, the methods and systems described above are utilized in some embodiments after the completion or substantial completion of the systems and methods utilized in the bioinformatics pipeline. As an example, the bioinformatics pipeline receives next-generation gene sequencing results and returns a set of binary files, such as one or more BAM files, reflecting the number of nucleic acid (e.g., cfDNA, DNA, and / or RNA) reads aligned to a reference genome. The methods and systems described above are utilized in some embodiments, for example, to take the cfDNA, DNA, and / or RNA reads and generate genomic features as a result.

[0190] If the digital and laboratory medicine platform further comprises an RNA data normalizer, in some embodiments, any RNA read counts are normalized prior to processing according to the embodiments described above. Examples of RNA data normalizers are disclosed, for example, in Publication No. 2020 / 0098448, published March 26, 2020, entitled "Methods of Normalizing and Correcting RNA Expression Data," which is incorporated herein by reference in its entirety for all purposes.

[0191] Where the digital and laboratory medical platform further comprises a genetic data deconvolver, in some embodiments any system and method for deconvolution is utilized to analyze genetic data associated with a specimen having two or more biological components to determine the contribution of each component to the genetic data and / or to determine what genetic data, when refined, would be associated with any component of the specimen. Examples of genetic data deconvolvers are disclosed, for example, in U.S. Patent Publication Nos. 2020 / 0210852, published July 2, 2020, and PCT / US19 / 69161, filed December 31, 2019, both entitled "Transcriptome Deconvolution of Metastatic Tissue Samples," and U.S. Patent Application No. 17 / 074,984, filed October 20, 2020, entitled "Calculating Cell-type RNA Profiles for Diagnosis and Treatment," the contents of each of which are incorporated herein by reference in their entirety for all purposes.

[0192] When the digital and laboratory medicine platform further comprises an automated RNA expression caller, in some embodiments, the RNA expression levels are adjusted to be expressed as values ​​relative to a reference expression level, which is often done to prepare multiple RNA expression datasets for analysis to avoid artifacts caused when the datasets have differences because they were not generated by using the same methods, instruments, and / or reagents. Examples of automated RNA expression callers are disclosed, for example, in U.S. Patent No. 11,043,283, issued June 22, 2021, entitled "Systems and Methods for Automating RNA Expression Calls in a Cancer Prediction Pipeline," which is incorporated by reference in its entirety for all purposes.

[0193] In some embodiments, the RNA expression levels are adjusted to be expressed as values ​​relative to a reference expression level. Additionally, in some embodiments, multiple RNA expression datasets are adjusted, prepared, and / or combined for analysis and / or adjusted to avoid artifacts that arise when the datasets are different because they were not generated by using the same method, apparatus, and / or reagent. Examples of adjustment, preparation, and / or combination of RNA datasets are disclosed, for example, in U.S. Patent Application No. 17 / 405,025, filed August 18, 2021, entitled "Systems and Methods for Homogenization of Disparate Datasets," which is incorporated herein by reference in its entirety.

[0194] The digital and laboratory medical platform further includes one or more insight engines for delivering information, characteristics, or decisions related to a medical condition that may be based on genetic and / or clinical data associated with the patient and / or specimen. Exemplary insight engines include, without limitation, a tumor of unknown origin engine, a loss of homozygosity (LOH) of human leukocyte antigen (HLA) engine, a tumor mutational burden engine, a PD-L1 status engine, a homologous recombination deficiency engine, a cell pathway activation report engine, an immune infiltration engine, a microsatellite instability engine, a pathogen infection status engine, a T-cell receptor or B-cell receptor profiling engine, a line of treatment engine, a metastasis prediction engine, and / or an IO progression risk prediction engine. Examples of tumor of unknown origin engines are disclosed, for example, in U.S. Patent Application Serial No. 15 / 930,234, filed May 12, 2020, entitled "Systems and Methods for Multi-Label Cancer Classification," which is incorporated herein by reference in its entirety for all purposes. Examples of HLA LOH engines are disclosed, for example, in U.S. Patent No. 11,081,210, issued August 3, 2021, entitled "Detection of Human Leukocyte Antigen Class I Loss of Heterozygosity in Solid Tumor Types by NGS DNA Sequencing," which is incorporated by reference in its entirety for all purposes. Additional examples of HLA LOH engines are disclosed, for example, in U.S. Patent Application No. 17 / 304,940, filed June 28, 2021, entitled "Detection of Human Leukocyte Antigen Loss of Heterozygosity," which is incorporated by reference in its entirety for all purposes.Examples of tumor mutation burden (TMB) engines are disclosed, for example, in U.S. Patent Publication No. 2020 / 0258601, published August 13, 2020, entitled "Targeted-Panel Tumor Mutational Burden Calculation Systems and Methods," which is incorporated herein by reference in its entirety for all purposes. Examples of PD-L1 status engines are disclosed, for example, in U.S. Patent Publication No. 2020 / 0395097, published December 17, 2020, entitled "A Pan-Cancer Model to Predict The PD-L1 Status of a Cancer Cell Sample Using RNA Expression Data and Other Patient Data," which is incorporated herein by reference in its entirety for all purposes. Additional examples of PD-L1 status engines are disclosed, for example, in U.S. Patent No. 10,957,041, issued March 23, 2021, entitled "Determining Biomarkers from Histopathology Slide Images," which is incorporated herein by reference in its entirety for all purposes. Additional examples of homologous recombination deficiency engines are disclosed, for example, in U.S. Patent No. 10,975,445, issued April 13, 2021, entitled "An Integrative Machine-Learning Framework to Predict Homologous Recombination Deficiency," which is incorporated herein by reference in its entirety for all purposes. Additional examples of homologous recombination deficiency engines are disclosed, for example, in U.S. Patent Application No. 17 / 492,518, filed October 1, 2021, entitled "Systems and Methods for Predicting Homologous Recombination Deficiency Status of a Specimen," which is incorporated herein by reference in its entirety for all purposes.Examples of cellular pathway activation reporting engines are disclosed, for example, in U.S. Patent Publication No. 2021 / 0057042, published February 25, 2021, entitled "Systems And Methods for Detecting Cellular Pathway Dysregulation In Cancer Specimens," which is incorporated herein by reference in its entirety for all purposes. Examples of immune infiltration engines are disclosed, for example, in U.S. Patent Publication No. 2020 / 0075169, published March 5, 2020, entitled "A Multi-Modal Approach to Predicting Immune Infiltration Based on Integrated RNA Expression and Imaging Features," which is incorporated herein by reference in its entirety for all purposes. Examples of MSI engines are disclosed, for example, in U.S. Patent Publication No. 2020 / 0118644, published April 16, 2020, entitled "Microsatellite Instability Determination System and Related Methods," which is incorporated herein by reference in its entirety for all purposes. Additional examples of MSI engines are disclosed, for example, in U.S. Patent Publication No. 2021 / 0098078, published April 1, 2021, entitled "Systems and Methods for Detecting Microsatellite Instability of a Cancer Using a Liquid Biopsy," which is incorporated by reference in its entirety for all purposes. Examples of pathogen infection status engines are disclosed, for example, in U.S. Patent No. 11,043,304, published June 22, 2021, entitled "Systems And Methods For Using Sequencing Data For Pathogen Detection," which is incorporated by reference in its entirety for all purposes.Another example of a pathogen infection status engine is disclosed, for example, in PCT / US21 / 18619, filed February 18, 2021, entitled "Systems And Methods For Detecting Viral DNA From Sequencing," which is incorporated by reference in its entirety for all purposes. An example of a T cell receptor or B cell receptor profiling engine is disclosed, for example, in U.S. Patent Application No. 17 / 302,030, filed April 21, 2021, entitled "TCR / BCR Profiling Using Enrichment with Pools of Capture Probes," which is incorporated by reference in its entirety for all purposes. An example of a line of therapy engine is disclosed, for example, in U.S. Patent Publication No. 2021 / 0057071, published February 25, 2021, entitled "Unsupervised Learning And Prediction Of Lines Of Therapy From High-Dimensional Longitudinal Medications Data," which is incorporated herein by reference in its entirety for all purposes. An example of a metastasis prediction engine is disclosed, for example, in U.S. Patent No. 11,145,416, published October 12, 2021, entitled "Predicting Likelihood and Site of Metastasis from Patient Records," which is incorporated herein by reference in its entirety for all purposes. An example of an IO progression risk prediction engine is disclosed, for example, in U.S. Patent Application No. 17 / 455,876, filed November 19, 2021, entitled "Determination of Cytotoxic Gene Signature and Associated Systems and Methods for Response Prediction and Treatment," which is incorporated herein by reference in its entirety for all purposes.

[0195] In some embodiments, any data generated by the systems and methods and / or digital and laboratory medicine platforms is downloadable by the user. In one example, the data is downloaded as a CSV file containing clinical and / or molecular data associated with tests, data structuring, and / or other services ordered by the user. In various embodiments, this is accomplished by aggregating clinical data in a system backend and making it available via a portal. In some embodiments, this data includes not only variant and RNA expression data, but also data associated with immunotherapy markers such as MSI and TMB, and RNA fusions.

[0196] If the digital and laboratory medical platform further includes a device including a microphone and speaker for receiving audible queries or commands from a user and delivering answers or other information, in some embodiments, the methods and systems described above are utilized to add data to a database that the device can access. Examples of such devices are disclosed, for example, in U.S. Patent Publication No. 2020 / 0335102, published October 22, 2020, entitled "Collaborative Artificial Intelligence Method And System," which is incorporated herein by reference in its entirety for all purposes.

[0197] In some embodiments, when the digital and laboratory medical platform further includes a mobile application for importing patient records including genomic sequencing records and / or results even if not generated by the same digital and laboratory medical platform, the above-described method and system are utilized to receive the imported patient records. An example of such a mobile application is disclosed, for example, in U.S. Patent No. 10,395,772, issued on August 27, 2019, entitled "Mobile Supplementation, Extraction, And Analysis Of Health Records," which is incorporated herein by reference in its entirety for all purposes. Another example of such a mobile application is disclosed, for example, in U.S. Patent No. 10,902,952, issued on January 26, 2021, entitled "Mobile Supplementation, Extraction, And Analysis Of Health Records," which is incorporated herein by reference in its entirety for all purposes. Another example of such a mobile application is disclosed, for example, in U.S. Patent Publication No. 2021 / 0151192, filed May 20, 2021, entitled “Mobile Supplementation, Extraction, And Analysis Of Health Records,” which is incorporated by reference in its entirety into this specification for all purposes.

[0198] If the digital and lab medicine platform further includes a report generation engine, in some embodiments, the methods and systems described above may be utilized to generate a summary report of the patient's ECG results and / or genetic profile and the results of one or more insight engines for presentation to a physician. For example, if the digital and lab medicine platform is used to provide clinical support for personalized cancer treatment, in some embodiments, the report may provide the physician with information about the extent to which the sequenced specimen contained tumor or normal tissue from a first organ, a second organ, a third organ, etc. For example, the report, in some embodiments, provides a genetic profile for each of the tissue types, tumors, or organs in the specimen. In some embodiments, the genetic profile represents the gene sequences present in the tissue type, tumor, or organ, and includes information about variants, expression levels, gene products, or other information that may be derived from genetic analysis of the tissue, tumor, or organ.

[0199] In some embodiments, the report includes treatments and / or clinical trials matched based on some or all of the ECG, electronic health record, genetic profile, or insight engine findings and summaries. For example, in some embodiments, the treatments are matched according to the systems and methods disclosed in U.S. Patent Application No. 17 / 546,049, filed December 9, 2021, entitled "Artificial Intelligence Driven Therapy Curation and Prioritization," which is incorporated by reference in its entirety for all purposes. For example, in some embodiments, clinical trials are matched according to the systems and methods disclosed in U.S. Patent Publication No. 2020 / 0381087, published December 3, 2020, entitled "Systems and Methods of Clinical Trial Evaluation," which is incorporated by reference in its entirety for all purposes.

[0200] In some embodiments, the report includes a comparison of the results (e.g., molecular and / or clinical patient data) to a database of results from many specimens. Exemplary methods and systems for comparing the results to a database of results are disclosed in U.S. Patent Publication No. 2020 / 0135303, published April 30, 2020, entitled "User Interface, System, And Method For Cohort Analysis," and U.S. Patent Publication No. 2020 / 0211716, published July 2, 2020, entitled "A Method and Process for Predicting and Analyzing Patient Cohort Response, Progression and Survival," each of which is incorporated herein by reference in its entirety for all purposes. The information is sometimes used in conjunction with similar information and / or clinical response information from additional specimens to match therapies likely to be useful in treating patients, discover biomarkers, or design clinical trials, in some embodiments.

[0201] If the digital and laboratory medicine platform further includes organoids developed in association with the platform (e.g., from a patient specimen), in some embodiments, the method and system are used to further evaluate the genetic sequencing data derived from the organoids and / or the organoids' sensitivity to treatments that match based on some or all of the information determined by the system and method, including predicted cancer type, possible tumor origin, etc. In some embodiments, these treatments are tested on the organoids, their derivatives, and / or similar organoids to determine the sensitivity of the organoids to those treatments. In some embodiments, any of the results are included in the report. If the organoids are associated with a patient specimen, in some embodiments, any of the results are included in a report associated with the patient and / or delivered to the patient or the patient's physician or clinician. In various examples, the organoids can be cultured using methods such as those described in U.S. Patent Publication No. 2021 / 0155989, published May 27, 2021, entitled "Tumor Organoid Culture Compositions, Systems, and Methods," PCT / US20 / 56930, filed October 22, 2020, entitled "Systems and Methods for Predicting Therapeutic Sensitivity," U.S. Patent Publication No. 2021 / 0172931, published June 10, 2021, entitled "Large Scale Organoid Analysis," PCT / US2020 / 063619, filed December 7, 2020, entitled "Systems and Methods for High Throughput Drug Screening," and U.S. Patent Publication No. 2021 / 0172931, published June 10, 2021, entitled "Systems and Methods for High Throughput Drug Screening," PCT / US2020 / 063619, filed December 7, 2020, entitled "Artificial Fluorescent Image Systems and No. 17 / 301,975, filed April 20, 2021, entitled "Methods for Producing and Administering Filtration-Based Microorganisms," each of which is incorporated herein by reference in its entirety for all purposes.In one example, drug susceptibility assays are particularly beneficial where the systems and methods return results that are consistent with a variety of treatments, each of which is consistent with at least one treatment, or multiple results (e.g., multiple equivalent or similar likely cancer types or tumor origins).

[0202] Where the digital and laboratory medicine platform further includes one or more of the above applications in combination with or as part of a medical device or laboratory developed test targeted generally to medical care and research, in some embodiments, such laboratory developed test or medical device results are enhanced and personalized through the use of artificial intelligence. Examples of laboratory developed tests, particularly tests that can be enhanced by artificial intelligence, are disclosed, for example, in U.S. Patent Publication No. 2021 / 0118559, published April 22, 2021, entitled "Artificial Intelligence Assisted Precision Medicine Enhancements to Standardized Laboratory Diagnostic Testing," which is incorporated herein by reference in its entirety for all purposes.

[0203] It should be understood that the examples given above are illustrative and do not limit the use of the systems and methods described herein in combination with digital and laboratory medical platforms.

[0204] In some embodiments, the results of the bioinformatics pipeline are provided for report generation. Report generation, in some embodiments, includes variant scientific analysis, including interpretation of variants (including somatic and germline variants, if applicable) for pathogenicity and biological significance. In some embodiments, variant scientific analysis also estimates microsatellite instability (MSI) or tumor mutational burden. Targeted treatments, in some embodiments, are identified based on the gene, variant, and cancer type for further consideration and review by the ordering physician. In some aspects, in some embodiments, clinical trials for which the patient may be eligible are identified based on the mutation, cancer type, and / or medical history. In some embodiments, subsequent validation occurs, after which the report may be finalized for signature and delivery. In some embodiments, the first or second report includes additional data provided through the clinical data flow 202, such as patient progress notes, pathology reports, imaging reports, and other relevant documents. Such clinical data is ingested, reviewed, and summarized based on a predefined set of curation rules. The clinical data is then entered into the patient's medical history timeline for report generation.

[0205] Further details regarding clinical report generation are disclosed in U.S. patent application Ser. No. 16 / 789,363, filed Feb. 12, 2020 (PCT / US20 / 180002), the contents of which are incorporated by reference in their entirety into this specification for all purposes.

[0206] Example. Example 1 - Evaluation of methods for training models to identify patients with undiagnosed pulmonary hypertension. Pulmonary hypertension (PH) is a progressive, debilitating disease with increasing treatment options. Data-driven techniques to find patients eligible for therapy may help optimize population-level therapeutic benefits. This example aims to determine whether patients with PH can be accurately identified using structured electronic health record (EHR) data.

[0207] Briefly, using a randomly sampled cohort of patients with PH-related structured data or unstructured note mentions from a large EHR dataset of over 2 million patients, patients were identified with clinically diagnosed PH using two blinded clinician manual review of de-identified patient records. We then characterized the availability of structured data elements, including any occurrence of a PH diagnosis code in any visit or problem list, any sequence or match of PH medications (treprostinil, epoprostenol, iloprost, selexipag, riociguat, ambrisentan, bosentan, macitentan, sildenafil, tadalafil, and their corresponding brand names), or any right heart catheterization (RHC) with a mean PA pressure greater than 20 mmHg. We also characterized how many patients had an echocardiographic measurement of a maximum velocity of tricuspid regurgitation greater than 3.4 m / s or a maximum pressure gradient greater than 46 mmHg.

[0208] A total of 108 patients with clinically diagnosed PH were identified, of which 43% did not have any structured data elements suggestive of PH. Only 26% of patients had a PH code in their problem list, 53% had a PH consultation diagnosis code, 2% had a PH medication, and 12% had RHC findings consistent with PH (Figure 1). Adding the less commonly available echocardiographic measurements as structured fields resulted in the identification of 5 additional patients. In total, 38% of patients did not have EHR data suggestive of PH but could be identified as having clinically diagnosed PH using note text. This data is shown in Figure 5.

[0209] Structured data elements, such as those available in EHR data, identify only a subset of patients with true pulmonary hypertension. Unstructured data, such as clinical notes, may be needed to better identify cohorts of patients with PH.

[0210] Example 2 - Methods for phenotyping electronic health records for pulmonary hypertension. To extend the analysis performed in Example 1, it was suggested that multiple labeling functions could be developed to automate weak labeling of medical records to eventually train a machine learning model to identify patients with undiagnosed pulmonary hypertension (PH) based on electrocardiogram (ECG) data. This would help better identify patients with pulmonary hypertension because deep learning models could learn signals from ECGs based on pathophysiological effects that humans cannot detect. However, a large number of ECG datasets are required to train such models without prior knowledge of which ECG signals indicate pulmonary hypertension. Although electronic health records of patients with available ECG data can be used to label such ECG data for model training, manual evaluation of such medical records is time-consuming and subjective. To address this challenge, aggregated labeling functions reflecting structured and / or NLP / ML phenotypes were used to generate high-quality labels for downstream ECG model training.

[0211] As a first step, an exploratory analysis of an electronic health record database on the incidence of PH was performed. First, the database was searched for records of right heart catheterization (RHC) procedures over the past 10 years. RHC is a diagnostic procedure used to measure pulmonary artery pressure and therefore assess whether a patient has pulmonary hypertension. Within this data, PH was defined as a mean pulmonary artery pressure (mPAP) increase of at least 25 mmHg at rest. Unique patients meeting this criterion were identified. Next, records of PH diagnoses were searched for using the International Classification of Diseases (ICD) medical codes ICD-9 416 (chronic pulmonary heart disease) and ICD-10 127 (primary pulmonary hypertension). Hundreds of thousands of unique patients meeting this criterion were identified. Of the thousands of patients with records showing an mPAP increase of at least 25 mmHg at rest, more than one-third did not have a diagnosis of PH in their medical records. Of the hundreds of thousands of patients with a PH diagnosis code in their medical records, over 90% of them did not have a resting mPAP elevation test result of at least 25 mmHg in their medical records, so phenotyping medical records based on any of these indicators of PH would result in a significant number of false negatives.

[0212] Additionally, a search of the EHR for thousands of patients with mPAP elevation of at least 25 mmHg at rest revealed a wide range of other cardiovascular disease (CVD) conditions and diagnoses, as shown in Table 1 below.

[0213] [Table 1-1] [Table 1-2]

[0214] As a first step, the medical records in the database were divided into training (80%), validation (10%), and test (10%) sets. Each set was then filtered according to a first set of PH characteristics. Filtering involved applying an extraction filter to the medical records to obtain a subset of medical records. The extraction filter evaluated each medical record for multiple inclusion criteria and multiple exclusion criteria. A medical record was selected for inclusion in the subset of records if the extraction filter identified no exclusion criteria and at least one inclusion criterion. A medical record was excluded from the subset of records if the extraction filter identified at least one exclusion criterion and / or did not identify at least one inclusion criterion. Inclusion criteria included: (i) medical codes for PH diagnosis (e.g., ICD-9 416 or ICD-10 127); (ii) prescriptions for medications used to treat PH; (iii) RHC procedure codes; (iv) echocardiogram findings consistent with PH (e.g., tricuspid regurgitation (TR) peak velocity >3.4 m / s or tricuspid regurgitation peak gradient (TRPG) >46 mmHg); (v) mention of PH in echocardiogram reports; and (vi) mention of PH in unstructured clinical notes (e.g., regular expression identification of PH terms such as “pulmonary hypertension”, “pulm HTN”, “PAH”, “pHTN”, “PHT”, or “pulmonary hypertension”; regular expression identification of broad terms such as “PH”, “PASP”, “PAP”, “pulmonary artery”, “right ventricular pressure”, or “septal flattening”, excluding unrelated terms such as “PhD”, “PHurinary”, “PH artery”, “PHurinary”, “PH artery”). Exclusion criteria included (i) patients aged 18 years or younger, and (ii) incomplete provider encounters (e.g., inpatient, outpatient, or telemedicine). Filtering reduced the total number of patient records by at least a factor of 10, from millions to hundreds of thousands.

[0215] Development of the filter criteria was an iterative process based on consultation with clinicians, review of the literature on PH, data analysis, and review of medical records that were filtered out in earlier versions of the filter. The goal was to minimize loss of sensitivity due to exclusion of patients with PH from the subset of records and increase the prevalence of PH within the subset of records to allow unbiased sampling and metrics. As a starting point, the dataset was filtered using the following criteria: (i) medical code for PH diagnosis (e.g., ICD-9 416 or ICD-10 127), (ii) prescription of medication used to treat PH, and (vi) a subset of PH mentions in unstructured clinical notes, along with the exclusion criteria. Inclusion criterion (iv) addition of echocardiogram results consistent with PH added the patient. Inclusion criterion (v) addition of mention of PH in echocardiogram report added the patient. Inclusion criterion (iii) addition of RHC procedure code added the patient. Finally, inclusion criterion (vi) expansion of PH mentions in unstructured clinical notes to include the terms "PHCOPD," "CTEPH," and "PHILD" added patients. The patient population included by the filter criteria can be further expanded by running natural language processing (NLP) and / or machine learning models on excluded patients to identify those most likely to have PH.

[0216] They then developed a set of structured phenotypes of PH, each of which indicated the presence of PH based on consultation with clinicians, a review of the PH literature, and data analysis. The first version of the structured phenotypes included the following: · 2+ encounter diagnosis for PH ICD diagnosis code (only completed encounters by provider, not lab visits or imaging visits or phone calls), [OR] One problem list occurrence of a PH diagnosis code (excluding problem list entries entered and deleted on the same day), [OR] RHC mean pulmonary artery pressure >20mmHg, [OR] PH drug use in drug orders or recommendations for completed visits, provided that sildenafil is only included when used or recommended TID (three times daily) and not as needed (PRN) or "as before" use, and tadalafil is only included in 20 mg or 40 mg doses.

[0217] The 100 training samples were independently reviewed by two clinicians who discussed and reached consensus on the criteria to be used to identify PH and agreed that no reinterpretation of clinical judgment should occur during review. After reviewing the 10 initial medical records together, the reviewers independently reviewed duplicate records to allow agreement to be assessed between phenotyping analyses. There was agreement between the reviewers for 51 of 52 duplicate cases. Discordant cases were reviewed and adjudicated by the reviewers together. In addition, the reviewers identified when the cases of PH were first diagnosed.

[0218] All of the training samples were then phenotyped by computer review using the first version of the structured phenotypes shown above. In this review, identification of a single structured phenotype was sufficient to phenotype a medical record as PH. Hundreds of thousands of medical records were phenotyped as PH. The dates associated with the relevant clinical phenotypes were used to predict the initial diagnosis date. Briefly, a hierarchical evidence system was used to find the first relevant date. For each subject identified as having a medical condition, evidence types (associated diagnosis codes, notes, medications, etc.) are sorted into tiers. The first evidence date is selected as the initial diagnosis date in a cascading fashion, e.g., if tier 1 evidence is not available, then the first tier 2 evidence date, if that is also not available, then tier 3 evidence, etc. For example, tier 1 evidence could be right heart catheterization data since it is the most robust, while tier 2 evidence could be echo mention, tier 3 could be note mention, etc.

[0219] The agreement between this automated phenotyping and clinical review was evaluated using 100 independently reviewed training samples. The results showed high specificity but low sensitivity and F1 score.

[0220] Using clinician review as ground truth, medical records corresponding to false positives and false negatives were reviewed to determine why the structured phenotype mislabeled these records.

[0221] One false positive was mislabeled because the chart showed that sildenafil was recommended for TID use but was actually being used to treat erectile dysfunction. Another false positive was mislabeled because epoprostenol, reported on the medication order, was being used for hypoxia / ARDS. Another false positive was mislabeled because the patient had a past history of ventricular septal defect and mitral valve prolapse, but there was no indication on the echocardiogram that the patient had experienced PH.

[0222] Of the false negatives, approximately half were correctly labeled when the criteria were lowered to one clinical visit, and approximately one-third were correctly labeled when only the inpatient criteria were lowered to one clinical visit and the outpatient criteria were maintained at two visits. Nearly all of the false negative medical records had a mention of PH in the unstructured clinical notes.

[0223] Chart review found that approximately 37.5% of patients phenotyped for PH did not have any codes or medications suggestive of PH, but had evidence in clinical notes. This corresponded to a prevalence of 33% (95% CI: 24% to 43%) in the initial training set, even using a conservative definition of true positive. Assuming there were no further positive cases in the records filtered out in the first stage, this yields an estimate of 2.43% PH in the general population, which significantly exceeds literature estimates of approximately 1% occurrence in the general population.

[0224] To trial and improve labeling, the structured phenotype was adjusted by (i) reducing the inpatient criterion to one clinical visit while maintaining the outpatient criterion at two clinical visits (adding patients), (ii) removing epoprostenol from the medical indication criteria (removing patients), and (iii) removing erectile dysfunction medications from the medical recommendation criteria (removing patients).

[0225] Version 2 of the structured phenotype was then used to label the training set of medical records. Again, the identification of a single structured phenotype was sufficient to phenotype the medical records as PH. Hundreds of thousands of medical records were phenotyped as PH. The associated clinical phenotype and associated date were used to predict the initial diagnosis date. The agreement of this automated phenotyping with clinical review was evaluated using an independently reviewed training sample. Results showed a slightly higher sensitivity and F1 score than F1, but both were still low.

[0226] Then, approximately 200 medical records in the validation set were phenotyped using the V1 and V2 structured phenotypes, using the same criteria that the presence of a single structured phenotype was sufficient to label the medical record as PH. Independent clinician review of the approximately 200 validation samples was performed as described above for the test set. The agreement of the automated phenotyping with the clinical review was evaluated using the independently reviewed validation samples. The statistics of this phenotyping, shown in Table 2 below, using the clinically reviewed phenotypes as ground truth, maintained high specificity and had slightly higher sensitivity and F1 scores.

[0227] Finally, the validation set of independently reviewed medical records was evaluated using a probabilistic graphical modeling model of multiple indicator functions. Briefly, the results obtained from the evaluation of each structured phenotype, whether or not it indicates PH, were aggregated using a PGM model, as further described in Example 4. The agreement between the automated phenotyping using the aggregate indicator function and clinical review was evaluated against the results of the independent review of the validation sample described above. As shown in Table 2 below, the statistics of this phenotyping, using the clinically reviewed phenotypes as ground truth, significantly improves the sensitivity and F1 score while maintaining high specificity.

[0228] [Table 2]

[0229] Example 3 - Method for training a model to identify patients with undiagnosed pulmonary hypertension.

[0230] Using the weak phenotypic labels generated in Example 3, a neural network model was trained to identify pulmonary hypertension from ECG data. Briefly, the assigned labels were used as dependent variables and electrocardiogram (ECG) data were used as independent variables to predict whether a patient had PH or not. A standard procedure of backpropagation was used during training.

[0231] The label (PH or no PH) assigned to each medical record in Example 2 was further used to train a model based on the data in the medical records. For example, in one embodiment, a machine learning model is trained to predict whether a patient has PH by using the assigned label as a dependent variable and electrocardiogram (ECG) data as an independent variable.

[0232] In a related embodiment, a machine learning model is trained by predicting whether a patient has PH using the assigned labels as dependent variables and unstructured text from clinical notes in the medical record.

[0233] In yet other embodiments, the assigned label is used as the dependent variable, and any combination of multimodal data from the electronic health record may be used to train a model to predict whether a patient has PH. Examples of multimodal data that may be used include, without limitation, structured data in the electronic health record (EHR), unstructured data in the EHR, lab results, prescribed and / or recommended medications, and medical procedures performed.

[0234] Example 4 - A probabilistic graphical modeling approach to weakly supervised learning - Majority voting simplification. the goal Solving general probabilistic graphical modeling (PGM) models for weakly supervised problems is difficult and requires simplifications and approximations to be estimated in reasonable computation time. Conceptually, the present embodiment starts with a simple version of this PGM in order to build more complex models from this work.

[0235] Problem statement The general population is assumed to have a prevalence of a given medical disorder (e.g., PH), and access to an EHR system containing N samples from this population is available. Each sample is evaluated using a binary variable l i∈(O,N) As explained above, there are K independent indicator functions (LFs), which classify each sample as having a medical disorder or not having a medical disorder, as indicated by the binary voting variable λ ij We attempt to classify these as not being mediated through

[0236] Importantly, these LFs are not expected to be complete. Instead, each LF represents a true positive rate

number

number

number

number

[0237] Variables to be solved for Instead of dealing with the common problem of using all the voting data across samples to inform the estimates, when only a single sample is considered, a "no data" version of this model is used. In this case, we use the LF votes for this sample.

number

[0238] Majority Model The "majority vote" assumption is made and is assumed as follows: 1. LF consists of two sets, namely size K +a "positive" set (e.g., LFs that attempt to identify positive samples) of size K - and a "negative" set (e.g., LF that attempts to identify negative samples). 2. Each LF has equal performance compared to the sign they vote for. Note that in some embodiments, a negative is not a positive, e.g., simply indicating that the subject cannot be phenotyped as having the target medical condition. Similarly, in some embodiments, a negative is a positive that the subject does not have the target medical condition. Thus, in some embodiments, a negative set may include both types of negative indicator functions.

[0239] Since the "no data" domain is considered here, it is assumed that there is no good reason to prefer one LF over another, and no way to reason about their respective performances. This is very close to the assumption underlying pure majority voting methods, where it only matters which set of LFs (positive or negative) has more votes, regardless of their individual performances, e.g., each vote is weighted equally. Assumption 2 above, underlying both traditional majority voting methods and this model, is unlikely to hold in general and will be relaxed in a later section.

[0240] With this assumption, the PGM is significantly simplified and concludes that decisions made from the model (e.g., p>0.5) generally correspond to the same final outcome as taking a majority vote, but provide a reasonable probability score instead of a binary decision.

[0241] Specifying the statistical model The observable value is n + (number of "positive" LF votes) and n -(the number of "negative" LF votes). Since equal performance is assumed for all of these LFs, let γ denote the true positive rate of the LF (relative to the label it votes for) and ε denote its false positive rate. Note that for a "positive" LF, γ is P(λ=1|l=1), but for a "negative" LF, it is instead P(λ=1|l=0). This is merely for notational convenience at this point, and the model can be generalized to different performances and the more general s and l values ​​described above.

number

[0242] Solution for Posterior Distribution PGM is P(l=1|n + ,n - ) should be solved for, which is

number

number

number

number

number

[0243] Generalized Majority Model Here, the assumptions from the previous section can be relaxed. In particular, it is almost certain that distinct LFs will have different performances. Furthermore, it is desirable to use different prior distributions for different LFs, where applicable. For example, there may be high certainty that a given LF is very precise, e.g., when a particular drug is used only for a particular medical condition. In such a case, it is highly unlikely that the drug was recommended / prescribed for a different medical condition, and it would be beneficial to set a prior distribution for the LF to encode this knowledge.

[0244] In this model, each has a single observable value, n + or n - Grouping LFs into “positive” and “negative” LFs with λ is no longer feasible because the LF priors may vary within the set of LFs. Instead, the observable values ​​must be enumerated over each LF. The voting variables λ described above j can be chosen as this observable value, which is 1 if LF votes and 0 otherwise. Correspondingly, s j and

number

[0245] Solution for Posterior Distribution The difference between this PGM and the majority voting PGM illustrated in FIG. 6A is that the binomials with products of Bernoulli variables are replaced with potentially different probabilities. The derivation of the generalized majority voting model is similar to the majority voting model, except that the result is a product of K beta functions in numerator and denominator terms, rather than simply two beta functions. Furthermore, the identity of the beta functions

number

number

number

[0246] All references cited in this specification are herein incorporated by reference in their entirety for all purposes to the same extent as if each individual publication or patent or patent application was specifically and individually indicated to be incorporated by reference in its entirety for all purposes.

[0247] Another aspect of the present disclosure provides a computer system including one or more processors and a non-transitory computer-readable medium including computer-executable instructions that, when executed by the one or more processors, cause the processors to perform a method according to any one of the embodiments disclosed herein, and / or any combination, modification, substitution, addition, or deletion thereof that would be apparent to a person of ordinary skill in the art.

[0248] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing program code instructions that, when executed by a processor, cause the processor to perform a method according to any one of the embodiments disclosed herein, and / or any combination, modification, substitution, addition, or deletion thereof that would be apparent to a person of ordinary skill in the art.

[0249] The present invention can be implemented as a computer program product comprising a computer program mechanism embedded in a non-transitory computer readable storage medium. For example, the computer program product may include the program modules shown in any combination of FIG. 1 and / or described elsewhere in this application. These program modules may be stored on a CD-ROM, DVD, magnetic disk storage product, USB key, or any other non-transitory computer readable data or program storage product.

[0250] Many modifications and variations of the present disclosure may be made without departing from its spirit and scope, as will be apparent to those skilled in the art. The specific embodiments described herein are provided by way of example only. The embodiments have been selected and described in order to best explain the principles of the invention and its practical application, thereby enabling those skilled in the art to best utilize the invention and its various embodiments, with various modifications suited to the particular applications contemplated. The present disclosure is limited only by the terms of the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A method for training a model for phenotyping a subject for a medical condition, comprising: A computer system including one or more processors and a memory, generating a plurality of indicator functions for the medical condition, each respective indicator function in the plurality of indicator functions including a corresponding set of one or more criteria that, when met, indicates a presence of the medical condition; and assigning to each subject a corresponding indicator indicative of the status of the medical condition by evaluating first information from corresponding medical records of each respective subject in a first plurality of subjects using an ensemble model including the plurality of indicator functions to obtain as output a prediction of a status of the medical condition from the ensemble model, wherein the evaluating comprises natural language processing of at least a portion of the respective medical records.

2. The method further comprises, before generating the plurality of indicator functions:

2. The method of claim 1, further comprising filtering a second plurality of objects according to a first set of features of the medical condition, wherein the filtering comprises applying an extraction filter to medical records corresponding to the second plurality of objects to obtain the first plurality of objects, the first plurality of objects being a subset of the second plurality of objects.

3. The method further comprising:

3. The method of claim 1 or 2, further comprising, for each respective subject in the first plurality of subjects, training a classification model to phenotype the subject with respect to the medical condition using (i) the corresponding label as a dependent variable, and (ii) second information from the corresponding medical record as an independent variable.

4. the extraction filter includes a plurality of criteria related to the first set of features for the medical condition; 4. The method of claim 1, wherein applying the extraction filter to the plurality of medical records comprises, for each respective medical record in the second plurality of medical records, evaluating the respective medical record against a respective criterion in the plurality of criteria, and wherein satisfying a single respective criterion is sufficient to include or exclude the respective medical record in the first plurality of medical records.

5. For one or more respective subjects in the second plurality of subjects that are excluded from the first plurality of subjects, evaluating whether the corresponding medical records indicate the presence of the medical condition independent of the evaluation of the respective medical records during application of the extraction filter; 5. The method of claim 4, further comprising: upon determining that the respective medical record indicates the presence of the medical condition, revising the plurality of criteria associated with the first set of features to be more inclusive when applying the extraction filter.

6. 6. The method of claim 4 or 5, wherein applying the extraction filter comprises, for each respective object in the second plurality of objects, evaluating a first data type against a first respective criterion in the plurality of criteria, and evaluating a second data type against a second respective criterion in the plurality of criteria.

7. 7. The method of claim 6, wherein the first data type is structured data in an electronic health record (EHR) and the second data type is unstructured data in the EHR.

8. applying the extraction filter Structured Electronic Health Record (EHR) data; Unstructured EHR data, Test results, Prescribed medications, and The method of any one of claims 4 to 7, comprising evaluating at least three selected data types from a performed medical procedure.

9. 9. The method of any one of claims 4 to 8, wherein the plurality of criteria comprises: (i) a first criterion that, when met, excludes a respective subject from the first plurality of subjects; and (ii) a second criterion that, when met, results in inclusion of the respective subject in the first plurality of subjects, provided that no criterion in the plurality of criteria excludes the respective subject from the first plurality of subjects.

10. 10. The method of any one of claims 1-9, wherein the second plurality of subjects comprises at least 10,000 subjects.

11. 11. The method of any one of claims 1 to 10, wherein the plurality of indicator functions comprises a respective indicator function for each respective subgroup within a plurality of subgroups of subjects having the medical condition.

12. 12. The method of any one of claims 1 to 11, wherein the plurality of indicator functions includes a first respective indicator function that includes a corresponding set of one or more criteria, the corresponding set of one or more criteria indicating the presence of the medical condition when met.

13. The generating of the plurality of indicator functions includes, for each of the first indicator functions: i) defining a first set of one or more criteria, the first set of one or more criteria being satisfied to indicate the presence of the medical condition; ii) for each respective subject in a first subset of the first plurality of subjects, determining whether the corresponding medical record for the respective subject satisfies the first set of one or more criteria; iii) for each subject in the first subset of the first plurality of subjects determined to satisfy the first set of one or more criteria, evaluating whether the corresponding medical record indicates the absence of the medical condition, independent of the determination of whether the corresponding medical record satisfies the first set of one or more criteria; iv) modifying the first set of one or more criteria to be less inclusive upon determining that the corresponding medical records indicate the absence of the medical condition.

14. The method of claim 13 , wherein the modifying comprises adding an exclusion criterion to the first set of one or more criteria.

15. 15. The method of any one of claims 1 to 14, wherein the plurality of indicator functions includes a second respective indicator function that includes a corresponding set of one or more criteria, the corresponding set of one or more criteria indicating an absence of the medical condition when met.

16. The generating of the plurality of indicator functions includes, for each second indicator function: i) defining a second set of one or more criteria, said second set of one or more criteria being indicative of the absence of said medical condition when met; ii) for each respective subject in a second subset of the first plurality of subjects, determining whether the corresponding medical record satisfies the second set of one or more criteria; iii) for each subject in the second subset of the first plurality of subjects determined to meet the second set of one or more criteria, evaluating whether the corresponding medical record indicates the absence of the medical condition, independent of the determination of whether the respective medical record meets the second set of one or more criteria; iv) modifying the second set of one or more criteria to be more inclusive upon determining that the respective medical record indicates the presence of the medical condition.

17. The method of claim 16 , wherein the modifying comprises adding an inclusion criterion to the second set of one or more criteria.

18. The method of any one of claims 1 to 17, wherein the plurality of indicator functions comprises at least 10 indicator functions.

19. 19. The method of any one of claims 1-18, wherein the first plurality of subjects comprises at least 10,000 subjects.

20. The method of any one of claims 1 to 19, wherein the ensemble model comprises an aggregate voting model based on the evaluation of each respective indicator function.

21. The method of claim 20 , wherein the aggregate voting model is a weighted voting model.

22. The method of any one of claims 1 to 19, wherein the ensemble model comprises a probabilistic graphical model.

23. 23. The method of claim 1, wherein for each respective subject in the first plurality of subjects, evaluating the ensemble model including the plurality of indicator functions comprises: evaluating a first data type for a first respective criterion in the plurality of criteria; and evaluating a second data type for a second respective criterion in the plurality of criteria.

24. 24. The method of claim 23, wherein the first data type is structured data in an electronic health record (EHR) and the second data type is unstructured data in the EHR.

25. said applying said extraction filter Structured Electronic Health Record (EHR) data; Unstructured EHR data, Test results, Prescribed medications, and The method of any one of claims 1 to 24, comprising evaluating at least three selected data types from a performed medical procedure.

26. 26. The method of any one of claims 1 to 25, wherein for each subject in the plurality of subjects, said applying comprises natural language processing of unstructured clinical notes from the EHR.

27. 27. The method of claim 26, wherein the natural language processing comprises querying one or more regular expressions against the corresponding unstructured clinical notes.

28. 28. The method of claim 26 or 27, wherein the natural language processing comprises inputting at least a portion of the unstructured clinical note into a machine learning model trained to identify language associated with the clinical condition.

29. 28. The method of claim 26 or 27, wherein the natural language processing comprises inputting at least a portion of the unstructured clinical notes into a large-scale language model.

30. 30. The method of any one of claims 3 to 29, wherein the classification model is a neural network, a support vector machine, a naive Bayes model, a nearest neighbor model, a boosted trees model, a random forest model, or a clustering model.

31. The method of any one of claims 3 to 30, wherein the second information comprises at least two data types.

32. 32. The method of claim 31, wherein the second information includes structured electronic health record (EHR) data and unstructured EHR data.

33. The second information is Structured data in electronic health records (EHRs); Unstructured data in the EHR, Test results, Prescribed medications, and The method of any one of claims 3 to 32, comprising at least three data types selected from a performed medical procedure.

34. The method of any one of claims 3 to 30, wherein the second information includes a data type that is not part of the first information.

35. 35. The method of claim 34, wherein the data type in the second information that is not part of the first information is a test result.

36. A method according to any preceding claim, wherein the second information comprises all or a part of the first information.

37. The method of any one of claims 1 to 36, wherein the medical condition is pulmonary hypertension.

38. The method of any one of claims 1 to 36, wherein the medical condition is pulmonary hypertension and the second information comprises electrocardiogram results.

39. The method further comprising:

39. The method of any one of claims 1 to 38, further comprising identifying, from a plurality of previously undiagnosed subjects, one or more respective subjects having the medical condition by inputting, for each respective subject of the plurality of subjects, corresponding second information for the respective subject into the classification model, and receiving as output a corresponding indication of whether the respective subject has the medical condition.

40. 40. The method of claim 39, wherein the plurality of subjects are associated with a health care provider.

41. The method further comprising:

39. The method of any one of claims 1 to 38, further comprising predicting whether a test subject has the medical condition by inputting second information about the test subject into the classification model and receiving as output an indication of whether the test subject has the medical condition.

42. 1. A computer system comprising: one or more processors; A non-transitory computer readable medium comprising computer executable instructions that, when executed by the one or more processors, cause the processors to perform the method of any one of claims 1 to 41.

43. A non-transitory computer readable storage medium storing program code instructions that, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 41.

44. 1. A method for training a model for phenotyping a subject for a medical condition, comprising: A computer system including one or more processors and a memory, generating a plurality of indicator functions for the medical condition, each respective indicator function in the plurality of indicator functions including a corresponding set of one or more criteria that, when met, indicates a presence or absence of the medical condition; and assigning to each subject a corresponding indicator indicative of the status of the medical condition by evaluating first information from corresponding medical records of each respective subject in a first plurality of subjects using an ensemble model including the plurality of indicator functions to obtain as output a prediction of a status of the medical condition from the ensemble model, wherein the evaluating comprises natural language processing of at least a portion of the respective medical records.

Images