Clinical variant modeling

Abstract

Examples may use phenotype data including natural language text and genotype data to configure at least one machine learning model. The at least one machine learning model may generate and output patient scores. A patient score may include a machine-learned likelihood that a patient is affected with a genetic condition given the phenotype data including the natural language text. Some embodiments may use the patient scores and the genotype data to configure a Bayesian causal model to generate and output variant scores. A variant score may include a machine-learned likelihood that a variant of a gene is pathogenic with respect to the genetic condition. The patient score and / or the variant score may be provided to one or more downstream systems, devices, processes, components, or models.

Description

CLINICAL VARIANT MODELINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to United States Provisional Patent Application No. 63 / 579,939 filed August 31, 2023 and United States Provisional Patent Application No. 63 / 562,696 filed March 7, 2024, each of which is incorporated herein by this reference in its entirety.TECHNICAL FIELD

[0002] A technical field to which this application relates is genetic testing. Another technical field to which this application relates is machine learning -based variant classification systems.BACKGROUND

[0003] Genetic variants are differences in DNA sequences between individuals in a population. There are many different types of variants, including structural variations, single -nucleotide polymorphisms, insertion and deletion variations, copy number variations, and translocations and inversions.

[0004] Genetic sequencing technology continues to evolve rapidly. High-throughput sequencing technologies increasingly enable genetic testing spanning genotyping, single genes, gene panels, exomes, genomes, transcriptomes and epigenetic assays for genetic diseases. The increased complexity of analysis and interpretation of clinical genetic testing, and the increased volume of testing, have been accompanied by new challenges in the interpretation of sequence variants.

[0005] For example, clinical molecular laboratories are increasingly detecting novel sequence variants in the course of testing patient specimens for a rapidly increasing number of genes associated with genetic diseases. While some phenotypes are associated with a single gene, many are associated with multiple genes.

[0006] Variant classification refers to a process of classifying genetic variants based on evidence supporting or rejecting a causal relationship with disease. The clinical significance of any given sequence variant falls along a gradient, ranging from those in which the variant is almost certainly pathogenic for a disease to those that are almost certainly benign.

[0007] Variant classifications are not themselves diagnoses but can be used by clinicians to make diagnostic decisions.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the disclosure. The drawings are for explanation and understanding only and should not be taken to limit the disclosure to the specific embodiments shown.

[0009] FIG. 1 illustrates an example of a variant scoring process, in accordance with some embodiments of the present disclosure.

[0010] FIG. 2 illustrates an example of a process for generating patient scores using a natural language processing-based model, in accordance with some embodiments of the present disclosure.

[0011] FIG. 3A illustrates an example process for generating a text score using a natural language processing-based model, in accordance with some embodiments of the present disclosure.

[0012] FIG. 3B illustrates an example process for generating a patient score using a natural language processing-based model, in accordance with some embodiments of the present disclosure.

[0013] FIG. 4 illustrates an example process for configuring a clinical variant model using machine learning, in accordance with some embodiments of the present disclosure.

[0014] FIG. 5A illustrates an example of a patient score computation, in accordance with some embodiments of the present disclosure.

[0015] FIG. 5B illustrates an example of a patient score computation, in accordance with some embodiments of the present disclosure.

[0016] FIG. 5C illustrates an example of a patient score computation using machine-learned feature weights, in accordance with some embodiments of the present disclosure.

[0017] FIG. 6A illustrates an example of a variant score computation using a Bayesian model, in accordance with some embodiments of the present disclosure.

[0018] FIG. 6B illustrates an example of a distribution of patient scores for a variant, in accordance with some embodiments of the present disclosure.

[0019] FIG. 6D illustrates an example of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure.

[0020] FIG. 6E illustrates an example of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure.

[0021] FIG. 6F illustrates an example of a distribution of patient scores and variant scores, in accordance with some embodiments of the present disclosure.

[0022] FIG. 6G illustrates an example of a distribution of patient scores and classified variants, in accordance with some embodiments of the present disclosure.

[0023] FIG. 7A illustrates an example of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure.

[0024] FIG. 7B illustrates an example of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure.

[0025] FIG. 8A illustrates an example of performance data for a patient score generator and a variant score generator, in accordance with some embodiments of the present disclosure.

[0026] FIG. 8B illustrates an example of a clinical variant modeling and classification system, in accordance with some embodiments of the present disclosure.

[0027] FIG. 8C illustrates an example of a variant classification system including clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0028] FIG. 9A illustrates a method for clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0029] FIG. 9B illustrates a method for clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0030] FIG. 9C illustrates a method for clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0031] FIG. 10 illustrates an example computing system that includes clinical variant modeling in accordance with some embodiments of the present disclosure.

[0032] FIG. 11 is a block diagram of an example computer system in which aspects of the present disclosure can operate.DETAILED DESCRIPTION

[0033] Genetic variants may be classified as pathogenic (i.e., disease-causing), benign (not diseasecausing), likely to be pathogenic, likely to be benign, or as having uncertain significance (VUS). Many variants are currently classified as VUS today because, among other reasons, the information about those variants is insufficient to make a classification. As genetic testing is increasingly adopted into healthcare for disease diagnosis and management, the clinical genomics field is increasingly encountering novel variants including both common and rare novel variants, which are also in need of classification. At the same time, the expansion of genetic testing leads to an ever-growing wealth of data, including clinical data.

[0034] Embodiments of the clinical variant modeling approaches described herein address these and / or other challenges by including one or more of a patient score generator, which generates patient scores, or a variant score generator, which generates variant scores based on the patient scores. While this disclosure describes approaches including both a patient score generator and a variant score generator, either the patient score generator or the variant score generator may be used independently of the other component. For example, in some applications, patient scores generated by the patient score generator may be useful separately and independently of variant scores generated by the variant score generator. As another example, the variant score generator can generate variant scores based on patient scores that are obtained not from a patient score generator as described herein but from another source, such as a database or a different type of patient score generator.

[0035] The clinical variant modeling approaches described herein (e.g., a modeling pipeline including a patient score generator as described and / or a variant score generator as described) leverage diverse genotype and anonymized clinical data from a population of patients that have undergone genetic testing to classify or re-classify both novel and previously seen variants, thereby reducing variants of uncertain significance and / or improving the accuracy of prior variant classifications.

[0036] Embodiments of the described clinical variant modeling approaches include a variant score generator configured as a Bayesian causal model, which has proven to be highly accurate for incorporating clinical evidence into variant classification at scale. The described approaches can be used to improve variant classification for genes and diseases that have not yet been tested, harnessing information at scale as the clinical datasets continue to grow. The described approaches can improve variant classification and reduce uncertainty in genetic testing in a scalable way.

[0037] The described clinical variant modeling approaches are robust to dataset size. For example, the described approaches can generate reliable predictions on large datasets (e.g., three to four million patients) and also on smaller datasets. The variant score generator enables domain expertise to be leveraged in situations where data is sparse. For the patient score model, the curriculum learning approach improves performance in the sparse regime.

[0038] A goal of clinical genetic testing is to assess risk for or to confirm diagnosis of hereditary diseases. In the course of genetic testing, distinguishing pathogenic DNA (deoxyribonucleic acid) variants from benign variants is a significant challenge. Clinical genetic testing labs continue to encounter many novel rare variants as more individuals undergo genetic testing, further exacerbating the challenge. These variants are not always well-documented in published literature or in the CLINVAR database and therefore end up classified as variants of uncertain significance (VUS).

[0039] Clinical data are among the most powerful forms of evidence for distinguishing pathogenic from benign variants. For example, clinical data may include observation of a variant in patients with a clearly defined disease, observation of a variant in patients definitively unaffected for a clearly defined disease, variant co-segregating with a clearly defined disease among affected individuals within a family, or observation of de novo occurrence of a variant. However, there are often challenges with incorporating clinical data into variant classification. For instance, a complete relevant medical history is not always provided at the time of genetic testing, which makes it challenging to differentiate between affected individuals with missing data and those who are unaffected.

[0040] Second, many hereditary diseases include symptoms that can be associated with common sporadic diseases, such as cancer and cardiovascular diseases, limiting the ability to identify a molecular cause and establish a genetic etiology. Finally, not all genetic diseases exhibit complete penetrance, which makes it difficult to determine when a variant is not associated with hereditary disease, and therefore benign. These challenges are particularly acute when clinical data is reviewed on a case-by-case basis for variant classification. However, with increasing access to large sets of clinical health information and genetic testing results can help overcome these challenges.

[0041] The disclosed approaches are able to leverage accumulated genotype data and clinical data for populations that include individuals of diverse racial and ethnic backgrounds who were referred for a wide range of clinical genetic testing. The dataset is massive and clinically diverse: in some embodiments, it includes over one hundred million words of clinical descriptions (e.g., personal and family history, indications fortesting) submitted for the tested patients, as well as over two million unique variants observed across more than 3,900 genes.

[0042] The disclosed clinical variant modeling approaches can maximize the utility of clinical datasets for improving variant classification and reducing VUS. Embodiments of the described approaches for clinical variant modeling use machine learning to determine patterns from clinical data available for millions of patients and precisely apply this learning as evidence for variant classification using a Bayesian approach. The described approaches can machine-leam relationships (e.g., statistical correlations) among and outcomes associated with different variables, including but not limited to thepenetrance of disease, age at testing, potential phenocopies and missing data on the clinical patient records (e.g., test requisition forms).

[0043] Some embodiments of clinical variant modeling as described herein include two distinct, but connected, sequential machine learning (ML) steps. The first step involves estimating the probability that a patient who has undergone genetic testing is affected by a specific genetic condition. This probability is referred to as a patient score, which incorporates clinical and demographic information from ordering provider(s), including reported signs and symptoms, ICD-10 codes, age at testing and family history. The patient score is estimated by comparing and distinguishing the clinical profile of patients with a positive molecular diagnosis from those with a negative molecular diagnosis. In the second step of clinical variant modeling, a Bayesian inference model learns the distribution of patient scores that could be associated with benign or pathogenic variants. The inferred probability that a variant is pathogenic is referred to as the variant score.

[0044] In more detail, embodiments of the described clinical variant modeling approaches use a combination of natural language processing (NLP) and Bayesian inference to predict the pathogenicity of genetic variants using clinical data provided in patient records (e.g., test requisition forms). The resulting clinical variant modeling system is designed to learn which clinical features may distinguish patients with a molecular diagnosis from genotype -negative controls.

[0045] As used herein, clinical variant modeling can refer to a modeling pipeline that includes one or more of a patient score generator and a variant score generator, while clinical variant model can refer to a patient score generator, a variant score generator, or a combination of a patient score generator and a variant score generator. Either or both of the patient score generator and the variant score generator can include one or more machine learning models. For example, as described in more detail below, embodiments of the patient score generator include at least two machine learning models, such as an NLP component and a tree-based scoring model. Thus, for instance, some embodiments of the clinical variant model can include up to or at least three machine learning models (e.g., a first scoring model such as an NLP-based scoring model, a second scoring model such as a tree-based scoring model, and a third scoring model such as a Bayesian inference model).

[0046] In some embodiments, the clinical variant models are condition-specific (e.g., a Neurofibromatosis type I model, which includes NF1, or the Lynch syndrome model, which includes MLH1, MSH2, MSH6, PMS2, and EPCAM). As a result, each clinical variant model may be trained and tested separately for a particular condition and its associated gene(s), in some embodiments.

[0047] In some embodiments, the clinical variant models are gene-specific (e.g., a clinical variant model is specific to MMR, or another gene). As a result, each clinical variant model may be trained and tested separately for a particular gene or genes, in some embodiments.

[0048] The described clinical variant modeling approaches are different from existing methods in several ways. First, existing methods for incorporating clinical data for single gene conditions evaluate clinical information for each patient on a case-by-case basis. While this approach may work for certain conditions, it is challenging for a number of hereditary conditions with symptoms that can be associated with common sporadic diseases (i.e., high phenocopy rate, such as cancer and cardiovascular diseases),demonstrate incomplete penetrance, and show variable expressivity. In contrast, the described clinical variant modeling approaches can leverage a large clinical database from genetically-tested patients (e.g., more than four million patients), allowing for distinctions to be made between the number of patients who share a gene variant and appear to have the disease phenotype in question and those who do not (in case of phenocopies).

[0049] Second, existing methods are generally binary (e.g., the prediction either meets diagnostic criteria or does not meet criteria). In contrast, clinical variant models configured using the described approaches can calculate a probability that a given individual is affected based on clinical information on a continuous scale, and also can calculate the probability that a variant is pathogenic based on the continuous distribution of affected and non-affected status for all patients with the same variant. As a result, scientists can use the available clinical data in a much more nuanced way.

[0050] Next, the described clinical variant modeling approaches can learn patterns that exist in the clinical data within the patient population cohort and apply (or generalize) those learned patterns to other patients. This capability may help reveal patterns and tendencies in the clinical data that may not be apparent otherwise, such as clinicians’ use of short-hands, abbreviations, terms in other languages, as well as frequencies at which no clinical information is provided (whether the patient is affected or not), age distributions, and ICD-10 code usage patterns, among others. Consequently, predictions of pathogenicity can be generalized to a specific but broader patient population.

[0051] In contrast to existing methods, which rely on patient data reported in literature, the described clinical variant modeling approaches only rely on patients observed through testing. Thus, whereas existing methods may be biased towards cohorts of patients who are more likely to be reported in literature (who are often more severely affected, or earlier onset than in the broader population), the described approaches are not biased in this way.

[0052] Additionally, since clinical variant modeling as described is a machine learning-based approach, the models can be updated periodically as more clinician patient data and / or more variant information becomes available in the medical genetics community.

[0053] Since clinical variant modeling is a machine learning approach that learns the particular features (e.g., particular ICD-10 codes, free text clinical information, free text family history information, etc.) that are most discriminating between patients with a molecular diagnosis and patients who are genotype-negative, each feature is not necessarily equally weighted. Furthermore, the weighting of each feature can vary from one genetic condition to the next. The clinical variant model for a given condition can automatically learn which features are the most predictive of pathogenicity and weights them appropriately.

[0054] A technical challenge to using clinical data for variant classification is that the format, quantity, and / or quality of information contained in a patient record can vary dramatically from patient to patient. For instance, some patient records may have a detailed textual explanation of the indication and / or family history, while other records may have just a few vague keywords or no information at all in these fields. The described clinical variant modeling approaches can accommodate the variability of information available in-patient records including missing information. Because different features areweighted differently for each gene / condition, the effect of missing or sparse information is dependent upon the particular information that is missing for a particular model. Since the model learns at a macro level, when features are missing, the model learns that there is some level of missing information.

[0055] Embodiments of the described clinical variant models have been trained, tested, and validated for clinical variant classification based on clinical information available for patients and variants in a proprietary database. Future plans include updating the models as the clinical cohort grows and more genotype and phenotype information becomes available as additional patients become genetically tested. Future updates to the models should be subjected to rigorous training, testing, and clinical validation before implementation into variant classification processes.

[0056] The described approaches do not use or rely on external clinical data (e.g., from publications). Instead, the described clinical variant models learn from clinical data obtained during the course of genetic testing at a laboratory and can be applied to new patients and variants observed by the lab. Experimental results have shown that clinical variant models as described are better at making predictions for individuals in a cohort because these models understand data patterns in the sampling and allow for the generalizability of the predictions to a specific patient population. External clinical data from publications will not have these same data patterns in patient sampling as a specific patient population. However, evaluation of external clinical data, such as those from publications, can still be used to supplement clinical variant modeling as described.

[0057] The described approaches can produce meaningful / accurate patient scores and variant scores from clinical data provided on the clinical patient record (e.g., test requisition form or TRF) even when the clinical patient records are incomplete or blank. This is because the clinical variant models as described have been trained, tested, and validated using a large volume of clinical records.

[0058] To compute the variant score, embodiments of the clinical variant model analyze the distribution of patient scores as a whole for a given variant and compare that distribution to the distribution of patient scores that are seen in known pathogenic versus known benign variants. These known variants can have a similar amount of missing or incomplete clinical information for the patients tested. Since the clinical variant model is looking at patterns in a large number of patients and variants, the impact of incomplete clinical information is much less than if the modeling were applied to a single patient or a small number of patients. Additionally, since the variant scores can be calculated for all variants, even variants classified as pathogenic and benign variants, the described approaches can help identify potentially misclassified variants, e.g., for further review by experts.

[0059] The clinical variant models described are capable of capturing the distribution of patient scores across the spectrum of unaffected to affected status. In contrast, with existing methods, clinical data from seemingly unaffected individuals are often omitted from the variant classification process due to concerns that perhaps the patient clinical record is simply incompletely filled out. Unlike the prior approaches, clinical variant models actually examine how frequently a patient appears unaffected among the molecular diagnosis cohort (either due to incomplete penetrance or incompletely filled patient clinical records) and the genotype-negative cohort. The described approaches learn how much weight a given observationshould be assigned. Even if a single observation is not very meaningful, across dozens or hundreds of observations, it can become significant enough to confidently reclassify a variant.

[0060] These clinical variant models can be incorporated into a variant classification process given their high performance in distinguishing known benign from known pathogenic variants for specific genes and / or gene-disease combinations. However, seemingly well-performing clinical variant models have been developed for more genes and conditions, but these models have not been fully vetted and validated. The set of available clinical variant models can be expanded once the models have been thoroughly evaluated for clinical validity.

[0061] To help show the consistency of clinical variant model predictions with known pathogenic variants and build trust that the clinical variant model is working appropriately, pathogenic evidence for known pathogenic variants is collected and included in the modeling when available. Furthermore, recording this evidence may help in the future if contradictory evidence emerges at a later time, so clinical scientists have all possible information to determine if a reclassification to LP (likely pathogenic), VUS (variant of uncertain significance), LB (likely benign), or B (benign) is warranted. It is anticipated that applying pathogenic evidence from clinical variant modeling to pathogenic variants will further increase the stability of these pathogenic classifications.

[0062] For genes that are associated with multiple conditions, if the molecular mechanisms of disease are the same (e.g., loss of function [LOF] or gain of function [GOF]), clinical variant models as described may not need to be trained separately for discrete gene-disease associations. Clinical variant models as described may be able to learn that there are distinct combinations of clinical features seen in patients with different conditions caused by variations in the same gene. While genes associated with multiple conditions (due to different molecular mechanisms, or due to different inheritance patterns) is a complex issue for which investigation is ongoing, there have been examples where the described modeling approaches appear to be working reliably for genes associated with multiple diseases of the same mechanism but different inheritance patterns (e.g., MSH2, MSH6, MLH1, PMS2 associated with dominant LoF Lynch and recessive LoF Constitutional Mismatch Repair Deficiency). Additionally, there have been examples in which the described approaches have appeared to work reliably using a single model for genes associated with multiple diseases of the same inheritance patterns, but with different molecular mechanisms (e.g., LoF CASR and GoF CASR).

[0063] To ensure that only the best performing clinical variant models are used for variant classification, the area under the receiver operating characteristics curve (AUROC) is calculated for each model to measure the model’s performance at distinguishing between benign and pathogenic variants. In some embodiments, only models with an AUROC > 0.8 are selected for further evaluation. In some embodiments, continued refinement of the set of clinical variant models is accomplished by a combination of further validation metrics and expert review. In some embodiments, additional steps are performed to establish the weighting of the variant scores for incorporation as evidence into a variant classification framework such as the INVITAE Sherloc framework (see, e.g., Nykamp K, Anderson M, Powers M, Garcia J, Herrera B, Ho YY, Kobayashi Y, Patil N, Thusberg J, Westbrook M; Invitae Clinical Genomics Group; Topper S. Sherloc: a comprehensive refinement of the ACMG-AMP variant classification criteria.Genet Med. 2017 Oct;19(10): 1105-1117. doi: 10.1038 / gim.2017.37. Epub 2017 May 11. Erratum in: Genet Med. 2020 Jan;22(l):240-242. PMID: 28492532; PMCID: PMC5632818).

[0064] Embodiments of clinical variant models (CVMs) as described have currently been developed and validated for, e.g., eleven genetic conditions associated with seventeen genes in which they demonstrate high performance in distinguishing known benign from known pathogenic variants (> 0.8 AUROC curve). Predictions from these CVMs have been used as evidence to resolve over 1,000 unique VUS, impacting > 45,000 individuals. While > 99% of these reclassifications corresponded to downgrades of VUS to benign or likely benign, the < 1% upgrades to pathogenic or likely pathogenic impacted -160 individuals. Of note, 91% (10 / 11) of variant upgrades were in genes associated with conditions that have established guidelines for screening and treatment, highlighting the potential to change an individual’s medical management and identify at-risk relatives.

[0065] The disclosure will be understood more fully from the detailed description given below, which references the accompanying drawings. The detailed description of the drawings is for explanation and understanding and should not be taken to limit the disclosure to the specific embodiments described.

[0066] In the drawings and the following description, references may be made to components that have the same name but different reference numbers in different figures. The use of different reference numbers in different figures indicates that the components having the same name can represent the same embodiment or different embodiments of the same component. For example, components with the same name but different reference numbers in different figures can have the same or similar functionality such that a description of one of those components with respect to one drawing can apply to other components with the same name in other drawings, in some embodiments.

[0067] Also, in the drawings and the following description, components shown and described in connection with some embodiments can be used with or incorporated into other embodiments. For example, a component illustrated in a certain drawing is not limited to use in connection with the embodiment to which the drawing pertains but can be used with or incorporated into other embodiments, including embodiments shown in other drawings.

[0068] FIG. 1 illustrates an example of a variant scoring process, in accordance with some embodiments of the present disclosure.

[0069] In FIG. 1, each patient of a patient population 102 has an associated clinical patient profile 104 and genetic test results 106. The clinical patient profile 104 includes unstructured data, e.g., natural language textual descriptions such as clinical descriptions, indications for genetic testing, and family history, as well as structured data, such as ICD-10 codes (International Classification of Diseases codes), gender indicators, age, and / or other demographic data. The genetic test results 106 may identify a variant or multiple variants detected as a result of the genetic testing. In cases where multiple variants are identified, the genetic test results may be filtered or sorted by variant to identify positive and negative patients for each variant.

[0070] The clinical patient profiles 104 and genetic test results 106 are input to a patient score generator 108. The patient score generator 108 generates and outputs predicted patient scores 112 for the patients associated with the clinical patient profiles 104 and corresponding genetic test results 106, usingvariant data 110. The variant data 110 includes reference or ground-truth labels for variants that are known / established in the scientific community as pathogenic or benign with respect to a given genetic condition. Each patient score includes a probability that the patient is affected with a genetic condition, based on the clinical evidence extracted from the clinical patient profile 104, including unstructured textual descriptions.

[0071] An embodiment of the patient score generator 108 includes a machine learning model that has been trained to distinguish between affected (molecular diagnosis positive) and unaffected (genetic negative) -appearing combinations of clinical features and estimate the probability that a patient is affected, given just the clinical evidence. The patient score generator 108 can include one or more machine learning models. An embodiment of the patient score generator 108 including two machine learning models is described with reference to FIG. 2.

[0072] The patient scores 112 are input to a variant score generator 114. The variant score generator 114 uses the variant data 110 and combines the patient scores 112 for the relevant patients to generate and output a variant score 116. The variant scores 116 estimate the probability that any given variant is causal of any of diseases associated with the gene, given the patient scores.

[0073] The patient scores 112 and / or the variant scores 116 can be sent, passed, provided, or otherwise made accessible to one or more downstream systems, processes, components, models, or frameworks, such as a variant classification framework or clinical data system. As a result, the described computing system 100 can generate and output either or both patient-level predictive scores 112 and variant-level estimates of pathogenicity (variant scores) 116. For instance, the patient scores 112 may be useful on their own in some applications. Examples of applications in which the patient scores 112 may be useful independently of the variant scores include the following:

[0074] 1. for Medical Necessity Review (reimbursement) to automatically identify patients who likely meets testing / reimbursement criteria;

[0075] 2. to determine which condition a patient is most likely affected with to determine what genes should be ordered, or which genes to prioritize in exome / genome testing;

[0076] 3. to assist variant classification operations by prioritizing patients that should undergo more thorough review by expert scientist vs flagging patients who do not need any additional review;

[0077] 4. for discovery of new diseases / phenotype association, e.g., the clinical variant model could learn that certain phenotypes are correlated with pathogenic variants in a given gene that previously was not known or proven;

[0078] 5. to refine pathognomonic criteria for an association; or

[0079] 6. to aid the identification of cohorts for clinical trials or other projects.

[0080] Also, while specific implementations of the patient score generator 108 and variant score generator 114 are described, either the patient score generator 108 or the variant score generator 114 may be implemented using different approaches. For example, other techniques for computing either the patient scores 112 or the variant scores 116 may be used, in some embodiments.

[0081] In some particular embodiments, the development and application of clinical variant models for a specific gene follow several general steps: (1) first, a patient score is generated to represent theprobability that a patient is affected with the molecular condition of interest. This patient score is used for the next step. (2) Second, a variant score is calculated to represent the probability that a variant is pathogenic based on the distribution of patient scores for that variant. (3) Next, the performance of the CVM (Clinical Variant Modeling) system is determined by using a holdout set of known phenotypegenotype relationship data points. (4) CVMs that perform well are calibrated by measuring positive and negative predictive values (PPV and NPV) from the previous step and then integrated with appropriate weight into a variant classification framework such as Sherloc. (5) Optionally, a subset of the variant classifications is reviewed by a panel of clinical genomic experts to ensure that the CVMs are performing as expected.

[0082] In some specific embodiments, the following details describe aspects of the implementation. In those embodiments, a dataset of interest is defined. For example, the Monarch Disease Ontology (MonDO) may be used to discover the set of all disease associations for genes that have been tested. Each condition is assigned a set of genes and their associated modes of inheritance using the Gene Curation Coalition (GenCC) database.

[0083] A cohort of patients is identified. For each condition, positive and negative (control) cohorts of patients from the population 102 are identified as follows. First, affected individuals are identified as patients who had positive molecular diagnoses in at least one of the included genes. For example, an autosomal recessive association would require either compound heterozygosity or homozygosity for existing pathogenic or likely pathogenic variants. Control cohorts can be identified the same way for all conditions-defined as patients who had only benign variants in the included genes.

[0084] For data preprocessing and feature selection, features used for patient score modeling can include both structured patient information (e.g., ICD-10 codes, age at accessioning, clinical area, etc.) and unstructured textual (e.g., indication fortesting, family history, clinician-reported ancestry) patient information. Other sources of patient clinical data can also or alternatively be used, including but not limited to clinic notes in PDF (portable document format) files, EMR (electronic medical records), and / or media recordings such as audio or video recordings of clinician notes or patient records, as well as non- clinical data. ICD-10 codes can be preprocessed by first abbreviating to the category level and then selecting the codes most enriched in the positive cohort per the chi-square test. For each patient clinical record, a string combining both indication for testing and family history can be created by concatenating these substrings with special tokens to demarcate the beginning and end of each span of text.

[0085] The configuration of a patient score model can follow the following sequence: a pre-trained large language model (e.g., UFNLP Gatortron) can be fine-tuned to the entire corpus of labeled patients across conditions (TL1). For each condition, TL1 is further fine-tuned to that condition's examples (TL2). This model is used to predict a text score for every patient, which is then used as a feature along with other patient information (e.g., age at testing, ICD-10 codes) to train a final model (PS). This model is then used to predict an overall score for each patient, aka the patient score.

[0086] The variant model utilizes the same variant labels that were used in the genotype filtering for cohorts of genotype-positive and genotype -negative patients and / or other sets of patients that are informative for interpretation, such as alternative bellwether patients. In addition, the set of patients whoare informative for interpretation-the bellwether patients-is determined in the following manner. For each gene included in a condition, the annotated mode of inheritance (see dataset definition above), is used to identify patients who could be expected to manifest disease if the variant were causal, who should not have a relevant disease if the variant were benign, and who would not have the disease that would be explained by another known variant in an included gene. For example, the bellwether patients for a variant in a single-gene autosomal dominant model would be those who have the variant of interest and no other non-benign variants in the gene. In a multi -gene model, these patients should also have no non-benign variants in the other gene(s).

[0087] The variant score model can be a partially pooled, hierarchical Bayesian inference model for interpreting genetic variants across multiple genes. It estimates parameters like pathogenic rate, penetrance, and gene probabilities based on input tensors representing gene, patient scores, and variant labels. The variant score, a probability that the variant is pathogenic, is sampled from the posterior predictive distribution of the partially observed Bernoulli variable.

[0088] For validation, an estimate of generalization performance can be attained by evaluating the model against a holdout set of 20% of labeled variants, which were not used for training. Metrics assessed included area under the receiver operating characteristic (AUROC) curve, average precision (AP), mean squared error (MSE), and classification metrics including Fl score, accuracy, and PPV and NPV. For high-performance models (AUROC>0.8) and high-performance genes (AUROC>0.8), variants with a posterior probability of pathogenicity <0.05 or >0.99 AND with >2 affected-appearing observations in unrelated individuals, were nominated for evidence via variant classification framework (e.g., Sherloc point assignment).

[0089] FIG. 2 illustrates an example of a process for generating patient scores using a natural language processing-based model, in accordance with some embodiments of the present disclosure.

[0090] The process 200 is performed by processing logic that includes hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method is performed by components of a computing system, including, in some embodiments, components or flows shown in FIG. 2 that may not be specifically shown in other figures and / or including, in some embodiments, components or flows shown in other figures that may not be specifically shown in FIG. 2. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, at least one process can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0091] In FIG. 2, the process 200 starts by, for each gene or set of genes, identifying two cohorts of patients - 1) patients with confirmed molecular diagnoses and 2) a control cohort. In machine learning, these patient cohorts provide reference labels to guide the model training. For each cohort, a first set of clinical features 202 is identified. The features 202 include the reference labels and unstructured text, e.g.,natural language descriptions such as indication descriptions and / or family history. The features 202 are extracted from clinical patient records associated with the patients in the cohort.

[0092] A machine learning model training approach called curriculum learning can be used to train a language model Ml to classify patients based on the textual descriptions provided in the features 202, alone (e.g., an instance of the features 202 includes only the textual descriptions and the reference labels). Examples of curriculum learning are described in more detail, for example with reference to FIG. 3A below. In general, curriculum learning can include sorting or grouping the instances of features 202 in a particular way to make the model training of the language model Ml more efficient. For example, curriculum learning may be used to select instances of features that have more complete textual descriptions first for input to the model Ml so that the model Ml is trained on the instances that have more available unstructured textual data before the model Ml is trained on the instances that have less available unstructured textual data. The described modeling approaches are not limited to the use of curriculum learning. Other machine learning training approaches can be used, including traditional supervised or semi-supervised training approaches.

[0093] The Ml model outputs a text score, which is the likelihood that a patient associated with an instance of the features 202 is affected with the genetic condition. The text score produced by the model Ml is combined with the patient's other clinical features (e.g., structured data extracted from the patient’s clinical record, which do not include the unstructured textual descriptions) to form a second set of clinical features 204. The second set of clinical features 204 is used to train a second model M2. The model M2 produces a final probability, the patient score. If the holdout sets are used, this step can use the same holdout datasets as used for the Ml model.

[0094] FIG. 3A illustrates an example process 300 for generating a text score using a natural language processing-based model, in accordance with some embodiments of the present disclosure.

[0095] In FIG. 3A, first model input includes a first clinical feature set 302. An instance of the first clinical features includes phenotype data associated with a patient identifier, a disease identifier, and a variant identifier. More specifically, the phenotype data includes unstructured text, such as natural language text; for example, a textual clinical description, an indication for clinical testing, a description of family history related to a disease, and / or a description of demographic data. First clinical feature set 302 can include many instances (e.g., hundreds, thousands, or more) of the first clinical features. For example, first clinical feature set 302 can include one instance of first clinical features for each patient who is a member of a selected cohort.

[0096] A first Sub-model Ml 306 receives as input the first clinical feature set 302. In response to the first clinical feature set 302, the first Sub-model Ml 306 generates and outputs text scores 314. That is, for each instance of first clinical features, the first Sub-model Ml 306 generates and outputs a corresponding text score 314. A text score 314 includes a machine-learned prediction as to whether the patient associated with the instance of first clinical features is affected with the identified disease. The text score 314 is based on the unstructured text extracted from the patient’s clinical record, excluding other, structured data that may also be contained in the patient’s clinical record, as to whether the patient associated with the instance of first clinical features is affected with the identified disease. The text score 314 is based on theunstructured text extracted from the patient’s clinical record, excluding other, structured data that may also be contained in the patient’s clinical record.

[0097] The first Sub-model Ml 306 includes a machine learning algorithm 308, one or more model parameters (including but not limited to feature weights or coefficients, such as clinical feature weights) 310, and one or more model hyperparameters 312. The first Sub-model Ml 306, including the machine learning algorithm 308, one or more model parameters 310, and one or more model hyperparameters 312, can be implemented as a language model. In some implementations, the first Sub-model Ml 306 is constructed using a neural network-based machine learning model architecture. In some implementations, the neural network-based machine learning model architecture includes one or more self-attention layers that allow the model to assign different weights to different words or phrases included in the model input (e.g., to weight different portions of the unstructured text differently).

[0098] Alternatively or in addition, the neural network architecture includes feed-forward layers and residual connections that allow the model to machine-leam complex data patterns including relationships between different words or phrases of the model input in multiple different contexts. In some implementations, the first Sub-model Ml 306 is constructed using a transformer-based architecture that includes self-attention layers, feed-forward layers, and residual connections between the layers. The exact number and arrangement of layers of each type as well as the hyperparameter values used to configure the model are determined based on the requirements of a particular design or implementation of the clinical variant modeling system.

[0099] In some examples, the neural network-based machine learning model architecture includes or is based on one or more generative transformer models, one or more generative pre-trained transformer (GPT) models, one or more bidirectional encoder representations from transformers (BERT) models, one or more large language models (LLMs), and / or one or more other natural language processing (NLP) models.

[0100] The first sub-model Ml 306 is an NLP-based machine learning model trained on a dataset of natural language text extracted from patient clinical records. Lor example, training samples of natural language text extracted from patient forms submitted to a genetic testing service are used to train the first sub-model Ml 306. The size and composition of the dataset used to train the first Sub-model Ml 306 can vary according to the requirements of a particular design or implementation of the clinical variant modeling system. In some implementations, the dataset used to train the first sub-model Ml 306 includes hundreds of thousands to millions or more different natural language text training samples.

[0101] In some implementations, the machine learning algorithm 308 can include a logistic function. A logistic function can model the relationship between X (input) and Y (predicted output), where the probability of Y is a linear combination of the independent variables in the input X. Mathematically, a simplified form of the logistic function can be expressed as P(X)=f(x)=l / (l+eA(-(P_O+p_l x) ) ), where e is the exponential constant and P_0 and P_1 are feature coefficients. During training of the first sub-model Ml 306, logistic regression estimates and iteratively adjusts the values of the coefficients in the linear combination based on the feature values in the training data set.

[0102] In FIG. 3A, first sub-model Ml 306 has been configured via supervised machine learning training, calibration, and validation processes described. During training, the values of one or more of the model parameters (e.g., weights or feature coefficients, such as clinical feature weights) can be iteratively adjusted to test the relative effect of a particular feature input x of the first clinical feature set 302 on the predicted outcome P(Y|X), e.g., a predicted text score 314, based on the values of the feature inputs x in the first clinical feature set 302. The values of the model parameters (e.g., weights or feature coefficients, such as clinical feature weights) are initialized and adjusted during model training and calibration.

[0103] The first sub-model Ml 306 also includes model hyperparameters 312, which are selected or tuned at a global level and generally are not modified based on specific instances of training data. Examples of model hyperparameters 312 can include learning rate (the rate at which an algorithm updates estimates), learning rate decay (a gradual reduction in the learning rate over time to speed up learning), momentum (the direction of the next step with respect to the previous step), regularization constant, number of branches in a decision tree, and neural network nodes (the number of nodes in each hidden layer of the neural network).

[0104] The first sub-model Ml 306 can be configured either as a binary classifier or as a scoring model. In a binary classification mode, the output of the first sub-model Ml 306 indicates whether the predicted outcome is pathogenic or benign as a discrete or binary value, e.g., zero indicates benign and one indicates pathogenic, for a given set of input features. In a scoring mode, the output of the first submodel Ml 306 includes a score, which corresponds to a continuous probability that the predicted outcome is pathogenic or benign (e.g., a floating point value between zero and 1, inclusive).

[0105] At inference time, the first sub-model Ml 306 can receive a first clinical feature set 302 that includes unstructured text and / or a variant identifier that the first sub-model Ml 306 has not previously seen as input (i.e., the first sub-model Ml 306 has not previously analyzed the particular combination of first clinical features), and generate a predicted text score 314 for the unseen instance of first clinical features.

[0106] FIG. 3B illustrates an example process 350 for generating a patient score using a natural language processing-based model, in accordance with some embodiments of the present disclosure.

[0107] In FIG. 3B, second model input includes a second clinical feature set 352. An instance of the second clinical features includes patient scores 364 and the corresponding structured phenotype data associated with a patient identifier, a disease identifier, and a variant identifier. More specifically, the phenotype data of the second clinical feature set 352 includes structured data, such as ICD codes, demographic data, and other structured data obtained from the patient’s clinical record, excluding the unstructured phenotype data (e.g., excluding the natural language text descriptions obtained from the patient’s clinical record). Second clinical feature set 352 can include many instances (e.g., hundreds, thousands, or more) of the second clinical features. For example, second clinical feature set 352 can include one instance of second clinical features for each patient for whom a text score is generated by first Sub-model Ml 306.

[0108] A second sub-model M2 356 is a machine learning-based scoring model such as a tree-based model, e.g., a boosted tree model. The second sub-model M2 356 receives as input the second clinicalfeatures set 352. In response to the second clinical feature set 352, the second sub-model M2 356 generates and outputs patient scores 364. That is, for each instance of second clinical features, the second sub-model M2 356 generates and outputs a corresponding patient score 364. A patient score 364 includes a machine-learned prediction as to whether the patient associated with the instance of second clinical features is affected with the identified disease. The patient score 364 is based on the structured data extracted from the patient’s clinical record, excluding the unstructured text that may also be contained in the patient’s clinical record. For example, the patient score 364 is computed based on the text score 314 and other structured data, excluding the textual descriptions used by the first sub-model Ml 306 to generate the text score 314.

[0109] The second sub-model M2 356 includes a machine learning algorithm 358, one or more model parameters (including but not limited to feature coefficients or weights) 360, and one or more model hyperparameters 362. The second sub-model M2 356, including the machine learning algorithm 358, one or more feature coefficients (or weights) 360, and one or more model hyperparameters 362, can be implemented using a neural network-based machine learning model architecture. For example, the second sub-model M2 356 can be implemented using a boosted tree model, a regression model, or another type of machine learning model configured as a classification model or as a scoring model.

[0110] In some implementations, the second sub-model M2 356 is constructed using a transformerbased architecture that includes self-attention layers, feed-forward layers, and residual connections between the layers. The exact number and arrangement of layers of each type as well as the hyperparameter values used to configure the model are determined based on the requirements of a particular design or implementation of the clinical variant modeling system. The second sub-model M2 356 is trained on a dataset of second clinical features 352. For example, structured data extracted from patient forms submitted to a genetic testing service are used, along with text scores 314, to train the second sub-model M2 356. The size and composition of the dataset used to train the second sub-model M2 356 can vary according to the requirements of a particular design or implementation of the clinical variant modeling system. In some implementations, the dataset used to train the second sub-model M2 35 6 includes hundreds of thousands to millions or more different training samples, with the size of the dataset corresponding to the size of the first clinical feature set 302. In other words, for each instance of first clinical features 302, there is a corresponding instance of second clinical features 352.

[0111] In some implementations, the machine learning algorithm 358 can include one or more of a logistic function, a decision tree, or a boosting function (e.g., gradient boosting). A logistic function can model the relationship between X (input) and Y (predicted output), where the probability of Y is a linear combination of the independent variables in the input X. Mathematically, a simplified form of the logistic function can be expressed as P(X)=f(x)=l / (l+eA(-(P_0+p_l x) ) ), where e is the exponential constant and P_0 and P_1 are feature coefficients. During training of the second sub-model M2 356, logistic regression estimates and iteratively adjusts the values of the coefficients in the linear combination based on the feature values in the training data set.

[0112] In FIG. 3B, second sub-model M2 356 has been configured via supervised machine learning training, calibration, and validation processes described. During training, the values of the featurecoefficients can be iteratively adjusted to test the relative effect of a particular feature input x of the feature set 352 on the predicted outcome P(Y|X), e.g., a predicted patient score 364, based on the values of the feature inputs x in the feature set 352.The values of the feature coefficients are initialized and adjusted during model training and calibration.

[0113] The second sub-model M2 356 also includes model hyperparameters 362, which are selected or tuned at a global level and generally are not modified based on specific instances of training data. Examples of model hyperparameters 362 can include learning rate (the rate at which an algorithm updates estimates), learning rate decay (a gradual reduction in the learning rate over time to speed up learning), momentum (the direction of the next step with respect to the previous step), regularization constant, number of branches in a decision tree, and neural network nodes (the number of nodes in each hidden layer of the neural network).

[0114] The second sub-model M2 356 can be configured as a binary classifier or as a scoring model. In a binary classification mode, the output of the second sub-model M2 356 indicates whether the predicted outcome is pathogenic or benign as a discrete or binary value, e.g., zero indicates benign and one indicates pathogenic, for a given set of input features. In a scoring mode, the output of the second submodel M2 356 includes a score, which corresponds to a continuous probability that the predicted outcome is pathogenic or benign (e.g., a floating point value between zero and 1, inclusive).

[0115] At inference time, the second sub-model M2 356 can receive a second clinical feature set 352 that includes structured clinical data, text score 314, and / or a variant identifier that the second sub-model M2 356 has not previously seen as input (i.e., the second sub-model M2 356 has not previously analyzed the particular combination of second clinical features), and generate a predicted patient score 364 for the unseen instance of second clinical features.

[0116] FIG. 4 illustrates an example process for configuring a clinical variant model using machine learning, in accordance with some embodiments of the present disclosure.

[0117] In FIG. 4, a model training process 400 applies a machine learning algorithm to a labeled feature set 402 using, e.g., supervised machine learning. For example, to train the first sub-model Ml 306, the labeled feature set 402 includes instances of first clinical features 302, and for each instance of clinical features 302, a ground-truth or reference label that indicates whether the associated variant is known benign, known pathogenic, or known variant of uncertain significance (VUS). As another example, to train the second sub-model M2 356, the labeled feature set 402 includes instances of second clinical features 352, and for each instance of clinical features 352, the corresponding ground-truth or reference label that indicates whether the associated variant is known benign, known pathogenic, or known variant of uncertain significance (VUS).

[0118] In some implementations, the model training process 400 implements one or more aspects of the curriculum learning machine learning model training technique. In curriculum learning, a machine learning model is trained in a meaningful order, e.g., from the easy training samples to the hard samples. For instance, curriculum learning can be implemented by a sorter / scheduler component 404 to group, sort, or rank training instances according to some objective measure of difficulty, and then feed the training instances to the model in the predetermined order (e.g., so that the model receives easier training instancesbefore it receives the harder training instances). In the case of training the first sub-model Ml 306, the sorter / scheduler 404 can, for example, group instances of first clinical features 302 based on the amount of unstructured text contained in the patient clinical records such that training instances that have more unstructured text are received by the model before training instances that have little or no unstructured text. In the case of training the second sub-model Ml 356, the sorter / scheduler 404 can, for example, rank instances of second clinical features 352 based on the values of the text scores 314 such that training instances that have higher text scores are received by the second sub-model Ml 356 before training instances that have lower text scores.

[0119] The sorter / scheduler 404 can also or alternatively implement a pacing function, which can control how many easy training examples are input to the model before the harder training examples are input to the model, or how quickly the model is introduced to harder training examples.

[0120] Alternatively or in addition to organizing and pacing the input of training examples to the model, curriculum learning can also be used to adjust the model architecture or one or more model parameters in an incremental way; for example, gradually increasing the model capacity (adding more neural units) or gradually increasing the complexity of the model’s tasks.

[0121] The use of curriculum learning can be particularly helpful in increasing the efficiency of the model training process and increasing the predictive reliability of the model in cases where the amount of data available varies from training instance to training instance. In particular, curriculum learning can help solve the problem of using patient records that have sparse, blank, or incomplete textual descriptions for clinical variant modeling.

[0122] On a first training iteration, feature coefficients (or weights) are initialized or assigned to each feature input of a given input feature, at sub-process 406. Feature coefficients are initialized by setting the coefficient values randomly, for example. The feature coefficient values assigned at sub-process 406 operate as weights that are applied to the respective features to produce weighted features 410. The machine learning algorithm is applied to the weighted features at sub-process 412 to produce predicted output 414. The reference or ground-truth labels contained in the training instances provide the expected output 408 for supervised machine learning. The predicted output 414 is evaluated by computing a loss (or error) based on the expected output 408 and the predicted output 414, at sub-process 416. The loss is computed using a loss function, such as a gradient descent algorithm.

[0123] On each iteration, the decision sub-process 418 evaluates the difference between the predicted output and the expected output, for example using a comparison of output of the loss function against a stopping condition that depends on the change in the output of the loss function. The change in the output of the loss function is compared to an error tolerance threshold (which may be referred to herein as a model performance criterion or model convergence criterion). If the model performance criteria, e.g.,. error tolerance threshold, is not satisfied (e.g., not greater than or equal to a threshold performance level, or exceeds a maximum permitted error value, or the loss has not stopped improving by more than the tolerance, or the error has not stopped decreasing by more than the tolerance threshold), the model training continues for another iteration. If the loss has stopped improving by more than the tolerance threshold, then the model has converged and the training ends.

[0124] On subsequent training iterations, the values of one or more of the feature coefficients are adjusted, the machine learning algorithm is applied to additional instances of the feature set, and the output of the machine learning algorithm is evaluated using the loss function and error tolerance as described above. The training process 400 ends when the model performance criteria, i.e., the error tolerance threshold, is satisfied and the model converges (e.g., the output of the comparison, e.g., loss function, is greater than or equal to a threshold performance level, or is within or below the maximum permitted error value, or the loss has stopped improving by more than the tolerance threshold).

[0125] FIG. 5A illustrates an example of a patient score computation, in accordance with some embodiments of the present disclosure. In FIG. 5A, a clinical record associated with a patient 1 for a gene NF1 includes structured data (e.g., age, sex, ICD codes) and unstructured data (e.g., indication and family history). Portions of these structured and unstructured data are extracted from the clinical record and input to the patient score generator (e.g., the combination of models Ml and M2, described above). The patient score generator generates a patient score based on the extracted structured and unstructured data. In the example of FIG. 5A, the patient score indicates, based on the clinical data, a high likelihood that the patient is affected with the genetic condition.

[0126] FIG. 5B illustrates an example of a patient score computation, in accordance with some embodiments of the present disclosure.

[0127] FIG. 5B illustrates an example of a patient score computation, in accordance with some embodiments of the present disclosure. In FIG. 5B, a clinical record associated with a patient 2 for a gene NF1 includes structured data (e.g., age, sex, ICD codes) and unstructured data (e.g., indication and family history). Portions of these structured and unstructured data are extracted from the clinical record and input to the patient score generator (e.g., the combination of models Ml and M2, described above). The patient score generator generates a patient score based on the extracted structured and unstructured data. In the example of FIG. 5B, the patient score indicates, based on the clinical data, a low likelihood that the patient is affected with the genetic condition.

[0128] FIG. 5C illustrates an example of a patient score computation using machine-learned feature weights, in accordance with some embodiments of the present disclosure.

[0129] To generate the patient score: (A) First, for a given molecular disease, which may be defined by a single gene (e.g., Neurofibromatosis type I) or multiple genes (e.g., Lynch syndrome), clinical- related patient information is gathered for all patients with a molecular diagnosis for the condition as well as for all patients who have only benign variation in the genes of interest and no other molecular diagnosis (i.e., genotype-negative cohort). An ML model learns the appropriate evidence weights for pieces of clinical information that distinguish individuals with molecular diagnosis from genotype-negative individuals. (B) Next, the learned evidence strengths for the clinical symptoms seen in the cohorts are then applied to each individual in the entire cohort to generate a Patient Score for each individual.

[0130] FIG. 6A illustrates an example 600 of a variant score computation using a Bayesian model, in accordance with some embodiments of the present disclosure.

[0131] To generate a variant score, the probability that a given variant is pathogenic is derived from a set of patient scores associated with the variant. Molecular diagnostic results are used to identify arelevant cohort of patients for any given variant. This is done for both known pathogenic and known benign variants. In the example of FIG. 6A, each stack of person icons represents a single variant, with each icon in the stack representing a patient score for a patient having that variant. Using this distribution, the probability that a variant's set of observations resulted from a pathogenic variant can be inferred (e.g., machine-learning based inference or statistical correlations. The distribution can be generated using a Bayesian inference model that takes into account several clinical genetics phenomena, including incomplete penetrance, age of onset, and phenocopy. This model formulation can naturally represent the uncertainty inherent in having limited observations, and this extends to its estimate of apparent variant penetrance, the fundamental signal the model uses to classify variants. The posterior of the model can be sampled to estimate a probability that a particular variant is pathogenic.

[0132] FIG. 6B illustrates an example 610 of a distribution of patient scores for a variant, in accordance with some embodiments of the present disclosure.

[0133] The column of squares represents a distribution of patient scores for a variant of a gene (e.g., MSH2). Each square in the column represents a patient score, which has been generated using a clinical patient record. In the example of FIG. 6B, a first patient has a patient score of 0.76 and a second patient has a patient score of 0.04. The higher patient score of 0.76 correlates with a higher likelihood that the first patient is affected with the genetic condition while the lower patient score of 0.04 correlates with a lower likelihood that the second patient is affected with the genetic condition. In the example of FIG. 6B, it is shown that a patient score can still be generated for the second patient even though the unstructured textual information is missing (indications field is blank).

[0134] For generation of the variant score, an example distribution of patient scores for 11 patients with MSH2 c.942+3 A>G. Each box in the figure represents a patient. Patients with low patient scores are shown in white and patients with high patient scores are shown in black. An example patient with a high patient score, based on the clinical evidence, is shown on the left side of the figure and an example patient with a low patient score, based on clinical evidence, is shown on the right side of the figure. In the example of FIG. 6B, it is shown that a patient score can still be generated for the second patient even though a portion of the unstructured textual information is missing (family history field is blank).

[0135] FIG. 6D illustrates an example 630 of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure.

[0136] In FIG. 6D, an example distribution of patient scores for known benign and pathogenic variants in MSH2 is shown. A second ML model calculates a variant score for each variant based on the distribution of patient scores. In the example of FIG. 6D, the known benign variants, which predominantly have individuals with low patient scores, all have low variant scores, while the known pathogenic variants on the right, which have more enrichment for patients with high Patient Scores, all have high variant scores.

[0137] FIG. 6E illustrates an example 640 of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure. In FIG. 6E, variant scores are generated for VUS in MSH2 and can be compared to the variant scores of known benign and known pathogenic variants.

[0138] FIG. 6F illustrates an example of a distribution of patient scores and variant scores, in accordance with some embodiments of the present disclosure. In FIG. 6F, an example of CVM results for NF1 is shown. In this cell plot, each stack of boxes represents a single genetic variant, and each box represents a single patient. Each box is shaded according to the individual's patient score, the inferred probability that the patient is affected with the condition. Patient score boxes 652, along the y axis on the left side of the plot, represents a low patient score; while patient score boxes 654, along the y axis on the right side of the plot, represent a high patient score. In the strip below the plot of patient scores, each box is shaded according to the variant score resulting from the stack of observations of patient scores from that variant. Boxes in the area 656 indicate a low variant score; while boxes in the area 658 indicate , a high variant score. Note that the y-axis is zoomed in to enable visualization of the patient boxes on the right side. The actual stacks of patient boxes on the left extend much higher than what is shown, as these are higher frequency variants.

[0139] FIG. 6G illustrates an example 660 of a distribution of patient scores and classified variants, in accordance with some embodiments of the present disclosure. FIG. 6G shows that using the described approaches, the expected distribution of patient scores for pathogenic and benign variants in MMR genes can be machine -learned. Using the expected distribution, a quantitative determination as to whether the distribution of patient scores for a VUS is similar to other pathogenic variants or other benign variants can be made. For example, supposing that variants 680 and 682 are VUS rather than known benign or known pathogenic. Given the distribution of patient scores for each of these variants 680, 682, they can be appropriately placed into the distribution of known benign and known pathogenic based on their similarity to the patient score distributions for the known benign and known pathogenic variants.

[0140] FIG. 7A illustrates an example 700 of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure. In FIG. 7A, patient scores for a sample of 20 benign and 20 pathogenic variants are plotted in vertical stacks along the y axis, and the variant scores resulting from the Bayesian NF1 clinical variant model are plotted along the x axis. FIG. 7A shows that the clinical variant model correctly classifies these variants. The right side of FIG. 7A shows the data for a sample of VUS in the same gene, demonstrating that the model performance ostensibly extends to these variants.

[0141] FIG. 7B illustrates an example 710 of a distribution of patient scores and variant scores for classified variants, in accordance with some embodiments of the present disclosure. In FIG. 7B, a sample of 40 VUS suggests that the patient score pattern generalizes to these unlabeled variants. The variant score reflects a pathogenic prediction for the variants on the right, a benign prediction for the variants on the left, and an indeterminate score for variants without sufficient observations. FIG. 7B also shows that benign-appearing variants are observed with much higher frequency than pathogenic-appearing variants, meaning these benign nominations should have a larger impact on the number of VUS reports.

[0142] FIG. 8A illustrates an example 800 of performance data for a patient score generator and a variant score generator, in accordance with some embodiments of the present disclosure.

[0143] FIG. 8A plots data generated by running the described CVM pipeline for over 2,300 disease- associated gene sets. For each gene-disease specific model, the performance in both steps of the pipelineis shown. Filtering to just those that achieve high performance on a validation set yields 920 models which include close to 2,000 genes. These models come from across every clinical area, from cardiology to cancer. As shown by FIG. 8A, the CVM machine learning pipeline can predict variant pathogenicity using previously underutilized clinical evidence. The patient score model estimates the probability that a patient is affected with the relevant genetic condition. The variant score model combines relevant patient observations to predict whether a variant is causal of disease. In some embodiments, there is a minimum number of patient observations needed for a variant to be eligible for a clinical variant model. For example, in some implementations, clinical variant models as described may require a minimum of three observations from at least two separate families. However, in practice, for many of the genes, more patients observations may be requested to ensure that the models reach the confidence thresholds needed to provide evidence to a variant classification framework such as Sherloc.

[0144] In experimental results, the described approach has demonstrated high performance (AUROC > 0.8) in distinguishing between pathogenic and benign variants for a large number of conditions (n=920) and genes (n=l,977) across clinical areas. This model has the potential to provide both pathogenic and benign evidence to a large number of VUS, and substantially reduce VUS reports where it is applied.

[0145] FIG. 8B illustrates an example of a clinical variant modeling and classification system, in accordance with some embodiments of the present disclosure.

[0146] The development and application of clinical variant models for each gene follow several general steps: (1) First, a patient score is generated to represent the probability that a patient is affected with the molecular condition of interest. This patient score is used for the next step. (2) Second, a variant score is calculated to represent the probability that a variant is pathogenic based on the distribution of patient scores for that variant. (3) Next, the performance of CVM is determined by using a holdout set of known phenotype -genotype relationship data points. (4) Models that perform well are then calibrated by measuring positive and negative predictive values (PPV and NPV) from the previous step and then integrated with appropriate weight into a variant classification framework such as Sherloc. (5) Optionally, a subset of the variant classifications is reviewed by a panel of clinical genomic experts to ensure CVMs are performing as expected. Each one of these general steps is explained in further detail below.

[0147] Step 1: Patient Score generation

[0148] CVMs leverage clinical data to predict the pathogenicity of variants for a given genetic condition in a stepwise manner. By leveraging details found in the clinician-reported data from the test requisition form (e.g., personal health history; family health history; age; sex; patient’s race, ethnicity, and ancestry; ICD-10 codes; and clinical area of the test ordered), a model is first trained to learn the clinical picture that distinguishes patients with a positive molecular diagnosis for the condition of interest from genotype -negative controls (i.e., patients without VUS, LP, or P variants in the condition of interest and who do not have a molecular diagnosis in another condition). For each patient, a Patient Score is generated (e.g., on a scale from 0 to 1), which is the probability that a given patient is affected with the genetic condition of interest based on the clinical information alone. Based on what is learned, this information can be applied to other patients who have VUS in the condition of interest, provided thosepatients do not have a current molecular diagnosis in another gene. This is done by scoring the other patients’ clinical profiles on how similar they look to positive or negative cases.

[0149] Step 2: Variant Score generation

[0150] Using a set of known pathogenic and benign variants for the condition, called labels, a second model learns the distribution of patient scores that are typical for pathogenic and benign variants. Based on what is learned, VUS can be scored based on how similarly their distribution of patient scores look to pathogenic and benign variants. A variant score is generated for each variant (e.g., on a scale from 0 to 1), which is the probability that a given variant is pathogenic. The higher the variant score, the higher the probability that the variant is pathogenic.

[0151] Step 3: Evaluation of CVM’s performance

[0152] All CVMs are carefully screened to ensure high performance in distinguishing pathogenic and benign variants (e.g., AUROC >0.8) by testing each model against a set of known pathogenic and benign variants that the model has not seen before (i.e., 20% holdout set).

[0153] Step 4: Model Calibration

[0154] Models passing the high-performance threshold are then calibrated. In some embodiments, calibrated models are then evaluated by clinical genomics experts before implementation. To date, clinical variant modeling has demonstrated high accuracy for over 600 genes and conditions.

[0155] Step 5: Clinical genomics expert review

[0156] Optionally, to gain further confidence in the prediction outputs of CVMs, a subset of the variants may be selected for thorough review by expert clinical genomic scientists. Variants with strong pathogenic (>99% PPV) and a sampling of (e.g., 163 / 1, 052) variants with strong benign (>95% NPV) CVM predictions were included. For each variant, all currently available non-CVM evidence was reviewed to evaluate for any concerning contradictory data. In this review, 93% (13 / 14) of variants predicted to be pathogenic and 95% (155 / 163) predicted to be benign by CVM were confirmed by the experts, while the remainder were kept as VUS. Of note, experts may choose the most challenging variants with CVM benign predictions to review (e.g., CVM benign predictions for variants with some level of pathogenic evidence or variants in the PMS2 pseudogene region). In addition, in some embodiments, four of the predicted pathogenic variants had at least one CEINVAR entry of likely pathogenic or pathogenic, while a fifth variant predicted to be pathogenic by CVM was recently reclassified from a VUS to likely pathogenic based on new family segregation data that was obtained after the CVM prediction was generated, but prior to the results being reviewed. Similarly, 15 of the reviewed predicted benign variants had at least one CEINVAR entry of likely benign or benign by another submitter.

[0157] Orthogonal validations.

[0158] To gain further confidence in the prediction outputs of the CVM, a concordance analysis was performed comparing CVMs to the evolutionary model of variant effect (EVE), a deep learning model that uses orthogonal data, namely sequence conservation, to predict the pathogenicity of variants in humans2. CVM predictions were highly concordant with EVE predictions (90.5%).

[0159] For additional confidence in the prediction outputs of the CVM for the Lynch syndrome genes (MLH1, MSH2, MSH6, PMS2, EPCAM), the CVM model predictions were compared to functional datasets (e.g., multiplex assays of variant effects or MAVEs) for MLH1, MSH2, and PMS2. The variants that had both CVM model predictions and MAVE predictions showed high concordance (>98%).

[0160] Another mechanism to assess orthogonal validation for CVM could be performed for TSC2, a gene associated with Tuberous Sclerosis. Specifically, exons 26 and 32 of TSC2 are absent from all known clinically relevant transcripts of the gene 4, note legacy exon nomenclature). Using only clinical evidence, the CVM for TSC2 identified genetic variants predicted to be pathogenic in each of the gene’s 42 exons, with the exception of exons 26 and 32-a remarkable concordance between the clinical model and independent molecular data.

[0161] FIG. 8C illustrates an example of a variant classification system including clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0162] To integrate the predictions from the CVMs into a variant classification framework such as Sherloc, two tiers of predictions may be established based on predictive performance thresholds, as measured in negative and positive predictive values (NPV and PPV). The benign tier was defined as strong benign evidence, which is enough evidence to classify the variant as likely benign without evidence to the contrary. The pathogenic tier was insufficient to reach a likely pathogenic classification on its own but could reach the definitive classification with the addition of the variant being absent or within the expected pathogenic range in gnomAD or another piece of pathogenic evidence. The predictive performance thresholds for those tiers were respectively defined as (1) strong benign >95% NPV, (2) strong pathogenic >99% PPV. The third and final tier corresponded to predictions that fell between a 99% PPV and below 95% NPV, which were deemed insufficiently certain to be assigned a weight within the Sherloc scoring system for the first release of CVMs.

[0163] Clinical variant modeling is incorporated into a variant classification framework, such as Sherloc. In one example, 2 tiers of predictive bins were used to integrate CVM predictions into Sherloc. Variants with CVM predictions with >95% NPV were awarded 3 benign points and variants with CVM predictions >99% PPV were awarded 3 pathogenic points. This clinical evidence is taken into account in the context of all variant classification evidence in Sherloc. Typical cutoffs for variant classifications are shown at the bottom right: 5 benign points for benign, 3 benign points for likely benign, 4 pathogenic points for likely pathogenic, and 5 pathogenic points for a pathogenic classification. Clinical genomics expert scientists can override these classification thresholds when necessary due to conflicting evidence and other factors.

[0164] In general, the number of points associated with a particular score correlates with the confidence or certainty of the prediction. The number of bins does not need to be two; any number of bins (including no bins or an infinite number of bins) can be used. The points that are mapped to the scores output by the model can be incorporated into another variant classification framework such as Sherloc. In this way, output of the clinical variant model as described can be applied to the Sherloc framework or to any other variant labeling or classification framework.

[0165] FIG. 9A illustrates a method for clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0166] The method 900 is performed by processing logic that includes hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method is performed by components of a computing system, including, in some embodiments, components or flows shown in FIG. 9A that may not be specifically shown in other figures and / or including, in some embodiments, components or flows shown in other figures that may not be specifically shown in FIG. 9A. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, at least one process can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0167] At step 902, a processing device extracts phenotype data from clinical records associated with a patient population and a genetic condition, where the phenotype data includes natural language text including at least one of an indication for testing, a family history description, or demographic data.

[0168] At step 904, a processing device extracts genotype data from genetic test results associated with the genetic condition, where the genotype data includes at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant.

[0169] At step 906, a processing device uses the phenotype data including the natural language text and the genotype data to configure a first model including a natural language processing (NLP)-based machine learning model, to generate and output patient scores for the patient population, where a patient score includes a machine-learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text.

[0170] In some implementations, configuring the natural language processing (NLP)-based machine learning model includes configuring a first model to receive the natural language text and generate and output a text score, where the text score includes a machine -learned likelihood that the patient is affected with the genetic condition given the received natural language text; and configuring a second model to generate and output the patient score using the text score and second model input including the phenotype data excluding the natural language text. In some implementations, the processing device uses curriculum learning to configure the NLP -based machine learning model to generate and output the text scores, and then either curriculum learning or another training approach is used to train a second model to generate and output the patient scores. The curriculum learning includes using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order. In some implementations, the processing device iteratively adjusts a value of at least one clinical feature weight of the NLP-based machine learning model until output of a loss function computed based on the patient score output by the NLP -based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, where the configured NLP-based machine learning model is capable of outputting patient scores that satisfy at leastone second performance criterion. In some implementations, the processing device uses positive examples and negative examples to configure the NLP -based machine learning model, where a positive example includes a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0171] At step 908, a processing device uses the patient scores and the genotype data to configure a Bayesian causal model to generate and output variant scores, where a variant score includes a machine- learned probability that a variant is pathogenic with respect to the genetic condition. In some implementations, the Bayesian causal model includes a probability distribution of patient scores and variant scores.

[0172] In some implementations, the processing device uses the configured NLP-based machine learning model and the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition. In some implementations, the processing device provides the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient. In some implementations, the processing device provides the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system In some implementations, the processing device uses variant scores output by the configured Bayesian causal model as an input to a variant classification framework.

[0173] FIG. 9B illustrates a method for clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0174] The method 920 is performed by processing logic that includes hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method is performed by components of a computing system, including, in some embodiments, components or flows shown in FIG. 9B that may not be specifically shown in other figures and / or including, in some embodiments, components or flows shown in other figures that may not be specifically shown in FIG. 9B. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, at least one process can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0175] At step 922, a processing device extracts phenotype data from clinical records associated with a patient population and a genetic condition, where the phenotype data includes natural language text including at least one of an indication for testing, a family history description, or demographic data;

[0176] At step 924, a processing device uses the phenotype data including the natural language text to generate and output patient scores for the patient population, where a patient score includes a likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text.

[0177] In some implementations, the processing device configures a natural language processing (NLP)-based machine learning model to generate and output the patient scores by (i) configuring a first model to generate and output text scores, where a text score includes a machine-learned likelihood that the patient is affected with the genetic condition given first model input including the natural language text; and (ii) configuring a second model to generate and output patient scores using second model input including the text score and phenotype data excluding the natural language text. In some implementations, the processing device uses curriculum learning to configure the NLP-based machine learning model to generate and output the patient scores, where the curriculum learning includes using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order. In some implementations, the processing device configures a natural language processing (NLP)-based machine learning model to generate and output the patient scores; and iteratively adjusts a value of at least one clinical feature weight of the NLP-based machine learning model until output of a loss function computed based on the patient scores output by the NLP-based machine learning model satisfies at least one first performance criterion, to produce the configured NLP-based machine learning model, where the configured NLP -based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion. In some implementations, the processing device uses positive examples and negative examples to configure the NLP-based machine learning model, where a positive example includes a positive association between a molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0178] At step 926, a processing device extracting genotype data from genetic test results associated with the patient population and the genetic condition, where the genotype data includes a molecular diagnosis including at least one variant of a gene. At step 928, a processing device using the patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, where a variant score includes a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition. In some implementations, the Bayesian causal model includes a probability distribution of patient scores and variant scores.

[0179] In some implementations, the processing device uses the configured NLP -based machine learning model and the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition. In some implementations, the processing device provides the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient. In some implementations, the processing device provides the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system In some implementations, the processing device uses variant scores output by the configured Bayesian causal model as an input to a variant classification framework.

[0180] FIG. 9C illustrates a method for clinical variant modeling, in accordance with some embodiments of the present disclosure.

[0181] The method 930 is performed by processing logic that includes hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method is performed by components of a computing system, including, in some embodiments, components or flows shown in FIG. 9C that may not be specifically shown in other figures and / or including, in some embodiments, components or flows shown in other figures that may not be specifically shown in FIG. 9C. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, at least one process can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0181] At step 932, a processing device extracts phenotype data from clinical records associated with a patient population and a genetic condition, where the phenotype data includes natural language text including at least one of an indication for testing, a family history description, or demographic data.

[0182] At step 934, a processing device uses the phenotype data including the natural language text to configure a natural language processing (NLP)-based machine learning model to generate and output patient scores for the patient population, where a patient score includes a machine-learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text.

[0183] In some implementations, configuring the natural language processing (NLP)-based machine learning model includes configuring a first model to receive the natural language text and generate and output a text score, where the text score includes a machine -learned likelihood that the patient is affected with the genetic condition given the received natural language text; and configuring a second model to generate and output the patient score using the text score and second model input including the phenotype data excluding the natural language text. In some implementations, the processing device uses curriculum learning to configure the NLP -based machine learning model to generate and output the patient scores, where the curriculum learning includes using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order. In some implementations, the processing device iteratively adjusts a value of at least one clinical feature weight of the NLP-based machine learning model until output of a loss function computed based on the patient score output by the NLP-based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, where the configured NLP-based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion. In some implementations, the processing device uses positive examples and negative examples to configure the NLP-based machine learning model, where a positive example includes a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0184] At step 936, a processing device extracts genotype data from genetic test results associated with the genetic condition, where the genotype data includes at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant.

[0185] At step 938, a processing device uses the patient scores and the genotype data to generate and output variant scores, where a variant score includes a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition. In some implementations, the processing device configures a Bayesian causal model to generate and output the variant scores. In some implementations, the Bayesian causal model includes a probability distribution of patient scores and variant scores.

[0186] In some implementations, the processing device uses the configured NLP-based machine learning model and the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition. In some implementations, the processing device provides the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient. In some implementations, the processing device provides the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system In some implementations, the processing device uses variant scores output by the configured Bayesian causal model as an input to a variant classification framework.

[0187] FIG. 10 illustrates an example computing system that includes clinical variant modeling in accordance with some embodiments of the present disclosure.

[0188] In the embodiment of FIG. 10, computing system 1000 includes a data selection subsystem 1002, a feature generation subsystem 1012, a training and calibration subsystem 1020, a model validation subsystem 1030, and one or more scoring models 1040. Data sources that may be accessed and used by components of computing system 1000 include data stores for model performance criteria 1032, one or more training data sets 1034, model validation criteria 1036, and one or more validation data sets 1038.

[0189] Any one or more of the components of computing system 1000 can correspond to similarly described components and / or processes shown in other figures and / or described herein. For example, the one or more scoring models 1040 can correspond to a single natural language processing (NLP)-based model (e.g., Ml or M2, described above) or a combination of NLP -based models (e.g., a modeling pipeline that includes Ml and M2), or a Bayesian causal model, or a combination of one or more NLPbased models and a Bayesian causal model. In other words, scoring model(s) 1040 can include a patient score generator, a variant score generator, or both a patient score generator and a variant score generator.

[0190] For example, scoring model(s) 1040 can include one or more machine learning models that are configured to determine probabilistic or statistical relationships between inputs and outputs using a machine learning algorithm. For example, given one or more inputs, a scoring model 1040 can output labels that can be used to classify the inputs into different categories or output scores that can be used to sort or rank the inputs into groups or ranked lists.

[0191] The data selection components 1004, 1006, 1008 can correspond to one or more of the data selection components, techniques or processes described herein. The feature generation subsystem 1012 can correspond to one or more of the feature generation components, techniques or processes describedherein. The model performance criteria 1032, training data set(s) 1034, model validation criteria 1036, and validation data set(s) 1038 can include data described herein as performance criteria, model validation data, training data, or validation data, as the case may be. Model validation subsystem 1030 can correspond to one or more of the model validation components, techniques or processes described herein.

[0192] Training and calibration subsystem 1020 includes a data set selection component 1022, a model training component 1024, and a model calibration component 1026. Model training component 1024 and / or model calibration component 1026 of training and calibration subsystem 1020 can train the one or more machine learning models of scoring model(s) 1040 by, for example, applying a supervised machine learning technique to training data that includes training examples of input data and reference (or ground-truth) labels. The predictive output of the one or more machine learning models is observed iteratively until a set of model performance criteria are satisfied. For example, differences between predictive output and expected output are quantified using a loss function. The model performance criteria are used to determine when the one or more machine learning models have converged so as to provide output that can be relied upon with a desired degree of certainty. The requisite level of certainty and the performance criteria are determined based on the requirements or design of a particular implementation of the one or more machine learning models.

[0193] Training data set(s) 1034 includes training data used to train the one or more scoring models 1040 in some implementations. Training data set(s) 1034 include, for example, sets of input features and corresponding reference (or ground-truth) labels. Training data set(s) 1034 can include or be derived from one or more databases of historical population data and / or genetic testing data, in some implementations. Examples of training data set(s) 1034 include clinical data sets, variant data sets, and / or ranked or ordered subsets used for, e.g., curriculum learning, as described above.

[0194] Data set selection component 1022 selects an appropriate data set for training or calibration of a particular scoring model 1040. For example, when a curriculum learning approach is used to train an M2 scoring model, data set selection component 1022 may select training examples that have the top k text scores, in rank order, where k is a positive integer. Alternatively or in addition, data set selection component 1022 may sort or order training examples used to train an Ml scoring model based on the amount of unstructured text contained in one or more unstructured text fields of a patient clinical record. For instance, data set selection component 1022 may group training examples according to the presence or absence of text in one or more unstructured text fields and then order the groups so that, for example, groups that have more text in the unstructured text fields are input to the Ml model before groups that have less text or no text in the unstructured text fields.

[0195] When a curriculum learning approach is used to train a scoring model, model training component 1024 can determine a schedule for applying the scoring model to ranked or ordered data sets prepared by data set selection component 1022, and model calibration component 1026 can provide feedback to model training component 1024, which model training component 1024 can use to alter the scheduling or pacing of the training process.

[0196] While not specifically shown in FIG. 10, computing system 1000 can include one or more user systems, which may be the same device as computing system 1000 or a different device. A usersystem includes at least one computing device, such as a personal computing device, a server, a mobile computing device, or a smart appliance. The user system includes at least one software application, including a user interface, installed on or accessible by a network to a computing device. For example, a user interface can include a graphical display screen that displays controls and graphical elements for operating and / or manipulating output of one or more scoring models 1040 and / or controlling one or more data selection, feature generation, training, calibration, or validation processes.

[0197] A user interface can be used to input data, initiate user interface events, and view or otherwise perceive output that includes patient predictions, variant pathogenicity predictions and / or other data produced by the clinical variant modeling system. Examples of user interfaces include web browsers, command line interfaces, and mobile app front ends. A user interface can include application programming interfaces (APIs). A user interface can include a front end portion of an application system that is used by a clinician or a scientist or a lab technician. For example, output of the clinical variant modeling system can be transmitted to and displayed by a user interface of a computing device used by a clinician. Alternatively or in addition, another version of a user interface can include a front end portion of an application software system that is used by variant scientists and / or other individuals working in the field of genetic testing. As such, output of the clinical variant modeling system 1050 can be transmitted to and displayed by a user interface of a computing device used by any of these and / or other individuals.

[0198] An application system can include any type of application software system that provides or enables the generation, display, or manipulation of output produced by a clinical variant modeling system. Examples of application systems include but are not limited to variant classification systems, DNA (deoxyribonucleic acid) analysis software, genetic testing software, medical testing software, healthcare management software, or any combination of any of the foregoing.

[0199] While not specifically shown, data storage system can include data stores and / or data services that store data received, used, manipulated, and produced by an application system and / or clinical variant modeling system, such as training data, validation data, machine learning model parameters and coefficients, performance criteria, validation criteria, machine learning model output, etc. In some embodiments, a data storage system includes multiple different types of data storage and / or a distributed data service. As used herein, data service may refer to a physical, geographic grouping of machines, a logical grouping of machines, or a single machine. For example, a data service may be a data center, a cluster, a group of clusters, or a machine.

[0200] The data storage system resides on at least one persistent and / or volatile storage device that can reside within the same local network as at least one other device of computing system 1000 and / or in a network that is remote relative to at least one other device of computing system 1000. Thus, although depicted as being included in computing system 1000, portions of data storage system 1080 can be part of computing system 1000 or accessed by computing system 1000 over a network.

[0201] While not specifically shown, it should be understood that any component of computing system 1000 can include one or more interfaces embodied as computer programming code stored in computer memory that when executed causes a computing device to enable bidirectional communication with any other component of computing system 1000 using a communicative coupling mechanism.Examples of communicative coupling mechanisms include network interfaces, inter-process communication (IPC) interfaces and application program interfaces (APIs).

[0202] Each component of computing system 1000 is implemented using at least one computing device that may be communicatively coupled to one or more electronic communications networks. Any component of computing system 1000 can be bidirectionally communicatively coupled by a network to any other component of computing system 1000.

[0203] A typical user of computing system 1000 can be an administrator or end user of an application system and / or clinical variant modeling system.

[0204] The features and functionality of the components of computing system 1000 are implemented using computer software, hardware, or software and hardware, and can include combinations of automated functionality, data structures, and digital data, which are represented schematically in the figures. Components may be shown in the figures as separate elements for ease of discussion but, except as otherwise described, the illustrations are not meant to imply that separation of these elements is required. The illustrated systems, services, and data stores (or their functionality) can be divided over any number of physical systems, including a single physical computer system, and can communicate with each other in any appropriate manner.

[0205] While not specifically shown, a network can be implemented on any medium or mechanism that provides for the exchange of data, signals, and / or instructions between the various components of computing system 1000. Examples of a network include, without limitation, a Local Area Network (LAN), a Wide Area Network (WAN), an Ethernet network or the Internet, or at least one terrestrial, satellite, optical, or wireless link, or a combination of any number of different networks and / or communication links.

[0206] FIG. 11 is a block diagram of an example computer system in which aspects of the present disclosure can operate. FIG. 11 illustrates an example machine of a computer system 1100 within which a set of instructions, for causing the machine to perform any of the methodologies discussed herein, can be executed. In some embodiments, the computer system 1100 can correspond to a component of a networked computer system (e.g., the computing system 1000 of FIG. 10) that includes, is coupled to, or utilizes a machine to execute an operating system to perform operations described above corresponding to aspects of the computing system 1000 of FIG. 10.

[0207] The machine is connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, and / or the Internet. The machine can operate in the capacity of a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0208] The machine is a personal computer (PC), a smart phone, a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term “machine” shall also be taken to includeany collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any of the methodologies discussed herein.

[0209] The example computer system 1100 includes a processing device 1102, a main memory 1104 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a memory 1105 (e.g., flash memory, static random access memory (SRAM), etc.), an input / output system 1110, and a data storage system 1140, which communicate with each other via a bus 1130.

[0210] Processing device 1102 represents at least one general -purpose processing device such as a microprocessor, a central processing unit, or the like. More particularly, the processing device can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 1102 can also be at least one special-purpose processing device such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 1102 is configured to execute instructions 1112 for performing the operations and steps discussed herein.

[0211] Instructions 1112 include portions of a clinical variant modeling system 1150, when those portions of the clinical variant modeling system are being executed by processing device 1102. Thus, the clinical variant modeling system 1150 is shown in dashed lines as part of instructions 1112 to illustrate that, at times, portions of the clinical variant modeling system are executed by processing device 1102. For example, when at least some portion of the clinical variant modeling system 1150 is embodied in instructions to cause processing device 1102 to perform the method(s) described above, some of those instructions can be read into processing device 1102 (e.g., into an internal cache or other memory) from main memory 1104 and / or data storage system 1140. However, it is not required that all of the clinical variant modeling system 1150 be included in instructions 1112 at the same time and portions of the clinical variant modeling system 1150 are stored in at least one other component of computer system 1100 at other times, e.g., when at least one portion of the clinical variant modeling system are not being executed by processing device 1102.

[0212] The computer system 1100 further includes a network interface device 1108 to communicate over the network 1120. Network interface device 1108 provides a two-way data communication coupling to a network. For example, network interface device 1108 can be an integrated-services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, network interface device 1108 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, network interface device 1108 can send and receive electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0213] The network link can provide data communication through at least one network to other data devices. For example, a network link can provide a connection to the world-wide packet datacommunication network commonly referred to as the “Internet,” for example through a local network to a host computer or to data equipment operated by an Internet Service Provider (ISP). Local networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data to and from computer system 1100.

[0214] Computer system 1100 can send messages and receive data, including program code, through the network(s) and network interface device 1108. In the Internet example, a server can transmit a requested code for an application program through the Internet and network interface device 1108. The received code can be executed by processing device 1102 as it is received, and / or stored in data storage system 1140, or other non-volatile storage for later execution.

[0215] The input / output system 1110 includes an output device, such as a display, for example a liquid crystal display (LCD) or a touchscreen display, for displaying information to a computer user, or a speaker, a haptic device, or another form of output device. The input / output system 1110 can include an input device, for example, alphanumeric keys and other keys configured for communicating information and command selections to processing device 1102. An input device can, alternatively or in addition, include a cursor control, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processing device 1102 and for controlling cursor movement on a display. An input device can, alternatively or in addition, include a microphone, a sensor, or an array of sensors, for communicating sensed information to processing device 1102. Sensed information can include voice commands, audio signals, geographic location information, and / or digital imagery, for example.

[0216] The data storage system 1140 includes a machine -readable storage medium 1142 (also known as a computer-readable medium) on which is stored at least one set of instructions 1144 or software embodying any of the methodologies or functions described herein. The instructions 1144 can also reside, completely or at least partially, within the main memory 1104 and / or within the processing device 1102 during execution thereof by the computer system 1100, the main memory 1104 and the processing device 1102 also constituting machine -readable storage media.

[0217] In one embodiment, the instructions 1112, 1114, 1144 include instructions to implement functionality corresponding to a clinical variant modeling system (e.g., any one or more components of the clinical variant modeling approaches and techniques described herein).

[0218] Dashed lines are used in FIG. 11 to indicate that it is not required that the clinical variant modeling system 1150 be embodied entirely in instructions 1112, 1114, and 1144 at the same time. In one example, portions of the clinical variant modeling system are embodied in instructions 1144, which are read into main memory 1104 as instructions 1114, and portions of instructions 1114 are read into processing device 1102 as instructions 1112 for execution. In another example, some portions of the clinical variant modeling system are embodied in instructions 1144 while other portions are embodied in instructions 1114 and still other portions are embodied in instructions 1112.

[0219] While the machine-readable storage medium 1142 is shown in an example embodiment to be a single medium, the term “machine-readable storage medium” should be taken to include a single medium or multiple media that store the at least one set of instructions. The term “machine -readablestorage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any of the methodologies of the present disclosure. The term “machine-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0220] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to convey the substance of their work most effectively to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, optical, or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0221] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure can refer to the action and processes of a computer system, or similar electronic computing device, which manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage systems.

[0222] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed for the intended purposes, or it can include a general -purpose computer selectively activated or reconfigured by a computer program stored in the computer. For example, a computer system or other data processing system, such as the computing system 1100, can carry out the above-described technologies in response to its processor executing a computer program (e.g., a sequence of instructions) contained in a memory or other non-transitory machine-readable storage medium. Such a computer program can be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0223] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the method. The structure for a variety of these systems will appear as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated hat a variety of programming languages can be used to implement the teachings of the disclosure as described herein.

[0224] The present disclosure can be provided as a computer program product, or software, which can include a machine-readable medium having stored thereon instructions, which can be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine -readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as a read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory components, etc.

[0225] Illustrative aspects of the technologies disclosed herein are provided below. An embodiment of the technologies may include any of the aspects described herein, or any combination of any of the aspects described herein, or any combination of any portions of the aspects described herein.

[0226] In some aspects, the techniques described herein relate to a method including: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data includes natural language text including at least one of an indication for testing, a family history description, or demographic data; extracting genotype data from genetic test results associated with the genetic condition, wherein the genotype data includes at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; using the phenotype data including the natural language text and the genotype data, configuring a natural language processing (NLP)-based machine learning model to generate and output patient scores for the patient population, wherein a patient score includes a machine -learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; and using the patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein a variant score includes a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

[0227] In some aspects, the techniques described herein relate to a method, further including: using the configured NLP-based machine learning model and the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition.

[0228] In some aspects, the techniques described herein relate to a method, further including: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

[0229] In some aspects, the techniques described herein relate to a method, further including: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

[0230] In some aspects, the techniques described herein relate to a method, further including: using variant scores output by the configured Bayesian causal model as an input to a classification framework.

[0231] In some aspects, the techniques described herein relate to a method, wherein configuring the natural language processing (NLP)-based machine learning model further includes: configuring a first model to receive the natural language text and generate and output a text score, wherein the text scoreincludes a machine-learned likelihood that the patient is affected with the genetic condition given the received natural language text; and configuring a second model to generate and output the patient score using the text score and second model input including the phenotype data excluding the natural language text.

[0232] In some aspects, the techniques described herein relate to a method, further including: using curriculum learning to configure the NLP -based machine learning model to generate and output the patient scores, wherein the curriculum learning includes using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

[0233] In some aspects, the techniques described herein relate to a method, further including: iteratively adjusting a value of at least one clinical feature weight of the NLP -based machine learning model until the patient score output by the NLP-based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, wherein the configured NLP -based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

[0234] In some aspects, the techniques described herein relate to a method, further including: using positive examples and negative examples to configure the NLP-based machine learning model, wherein a positive example includes a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0235] In some aspects, the techniques described herein relate to a method, wherein the Bayesian causal model includes a probability distribution of patient scores and variant scores.

[0236] In some aspects, the techniques described herein relate to a system including: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory includes at least one instruction that, when executed by the at least one processor, cause the at least one processor to perform at least one operation including: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data includes natural language text including at least one of an indication for testing, a family history description, or demographic data; using the phenotype data including the natural language text, generating and outputting patient scores for the patient population, wherein a patient score includes a likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; extracting genotype data from genetic test results associated with the patient population and the genetic condition, wherein the genotype data includes a molecular diagnosis including at least one variant of a gene; and using the patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein a variant score includes a machine -learned likelihood that a variant is pathogenic with respect to the genetic condition.

[0237] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: using the configured Bayesian causal model to generate and output aprediction as to whether a variant associated with an unseen clinical record is benign, of uncertain significance, or pathogenic with respect to the genetic condition.

[0238] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

[0239] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

[0240] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: using variant scores output by the configured Bayesian causal model as an input to a classification framework.

[0241] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: configuring a natural language processing (NLP)-based machine learning model to generate and output the patient scores by (i) configuring a first model to generate and output text scores, wherein a text score includes a machine-learned likelihood that the patient is affected with the genetic condition given first model input including the natural language text; and (ii) configuring a second model to generate and output patient scores using second model input including the text score and phenotype data excluding the natural language text.

[0242] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: using curriculum learning to configure the NLP -based machine learning model to generate and output the patient scores, wherein the curriculum learning includes using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

[0243] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: configuring a natural language processing (NLP)-based machine learning model to generate and output the patient scores; and iteratively adjusting a value of at least one clinical feature weight of the NLP-based machine learning model until the patient scores output by the NLP -based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, wherein the configured NLP -based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

[0244] In some aspects, the techniques described herein relate to a system, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further including: using positive examples and negative examples to configure theNLP-based machine learning model, wherein a positive example includes a positive association between a molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0245] In some aspects, the techniques described herein relate to a system, wherein the Bayesian causal model includes a probability distribution of patient scores and variant scores.

[0246] In some aspects, the techniques described herein relate to at least one non-transitory machine- readable storage medium including at least one instruction that, when executed by at least one processor, causes the at least one processor to perform at least one operation including: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data includes natural language text including at least one of an indication fortesting, a family history description, or a demographic description; using the phenotype data including the natural language text, configuring a natural language processing (NLP)-based machine learning model to generate and output patient scores for the patient population, wherein a patient score includes a machine-learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; extracting genotype data from genetic test results associated with the patient population and the genetic condition, wherein the genotype data includes a molecular diagnosis including at least one variant of a gene; and using the patient scores and the genotype data to generate and output variant scores, wherein a variant score includes a likelihood that a variant is pathogenic with respect to the genetic condition.

[0247] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further including: using the configured NLP -based machine learning model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition.

[0248] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further including: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

[0249] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further including: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

[0250] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further including: using the variant scores as input to a classification framework.

[0251] In some aspects, the techniques described herein relate to an at least one non-transitory machine-readable storage medium, wherein configuring the NLP-based machine learning model further includes: configuring a first model to receive the natural language text and generate and output a text score, wherein the text score includes a machine -learned likelihood that the patient is affected with the genetic condition given the received natural language text; and configuring a second model to generate and output the patient score using second model input including the text score and second phenotype data not including the natural language text.

[0252] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein configuring the NLP -based machine learning model further includes: using curriculum learning to configure the NLP-based machine learning model to generate and output the patient scores, wherein the curriculum learning includes using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

[0253] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further including: iteratively adjusting a value of at least one clinical feature weight of the NLP-based machine learning model until the patient score output by the NLP -based machine learning model satisfies at least one first performance criterion, to produce the configured NLP-based machine learning model, wherein the configured NLPbased machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

[0254] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further including: using positive examples and negative examples to configure the NLP-based machine learning model, wherein a positive example includes a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0255] In some aspects, the techniques described herein relate to an at least one non-transitory machine -readable storage medium, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further including: using a Bayesian causal model to generate and output variant scores, wherein the Bayesian causal model includes a probability distribution of patient scores and variant scores.

[0256] In some aspects, the techniques described herein relate to a method including extracting phenotype data from clinical records associated with a patient population and a genetic condition, where the phenotype data includes natural language text including at least one of an indication for testing, a family history description, or demographic data; extracting genotype data from genetic test results associated with the genetic condition, where the genotype data includes at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; and using the phenotype dataincluding the natural language text and the genotype data, configuring a machine learning model to generate and output patient scores for the patient population, where a patient score includes a machine- learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text.

[0257] In some aspects, the techniques described herein relate to a method including: extracting genotype data from genetic test results associated with a genetic condition, where the genotype data includes at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; and using patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, where a variant score includes a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

[0258] In some aspects the techniques described herein related to any one or more aspects, steps, components, elements, processes, or limitations that are at least one of described in the enclosed description and / or shown in the accompanying drawings.

[0259] Clause 1. A method comprising: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication for testing, a family history description, or demographic data; extracting genotype data from genetic test results associated with the genetic condition, wherein the genotype data comprises at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; using the phenotype data including the natural language text and the genotype data, configuring a natural language processing (NLP)-based machine learning model to generate and output patient scores for the patient population, wherein a patient score comprises a machine-learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; and using the patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein a variant score comprises a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

[0260] Clause 2. The method of clause 1, further comprising: using the configured NLP -based machine learning model and the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition.

[0261] Clause 3. The method of clause 2, further comprising: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

[0262] Clause 4. The method of clause 2, further comprising: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

[0263] Clause 5. The method of clause 2, further comprising: using variant scores output by the configured Bayesian causal model as an input to a classification framework.

[0264] Clause 6. The method of any of clauses 1-5, wherein configuring the natural language processing (NLP)-based machine learning model further comprises: configuring a first model to receive the natural language text and generate and output a text score, wherein the text score comprises amachine-learned likelihood that the patient is affected with the genetic condition given the received natural language text; and configuring a second model to generate and output the patient score using the text score and second model input comprising the phenotype data excluding the natural language text.

[0265] Clause 7. The method of clause 6, further comprising: using curriculum learning to configure the NLP -based machine learning model to generate and output the patient scores, wherein the curriculum learning comprises using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

[0266] Clause 8. The method of any of clauses 1-7, further comprising: iteratively adjusting a value of at least one clinical feature weight of the NLP-based machine learning model until the patient score output by the NLP-based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, wherein the configured NLP-based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

[0267] Clause 9. The method of any of clauses 1-8, further comprising: using positive examples and negative examples to configure the NLP -based machine learning model, wherein a positive example comprises a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0268] Clause 10. The method of any of clauses 1-9, wherein the Bayesian causal model comprises a probability distribution of patient scores and variant scores.

[0269] Clause I L A system comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises at least one instruction that, when executed by the at least one processor, cause the at least one processor to perform at least one operation comprising: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication fortesting, a family history description, or demographic data; using the phenotype data including the natural language text, generating and outputting patient scores for the patient population, wherein a patient score comprises a likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; extracting genotype data from genetic test results associated with the patient population and the genetic condition, wherein the genotype data comprises a molecular diagnosis comprising at least one variant of a gene; and using the patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein a variant score comprises a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

[0270] Clause 12. The system of clause 11, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: using the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign, of uncertain significance, or pathogenic with respect to the genetic condition.

[0271] Clause 13. The system of clause 12, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

[0272] Clause 14. The system of clause 12, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

[0273] Clause 15. The system of any of clauses 11-14, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: using variant scores output by the configured Bayesian causal model as an input to a classification framework.

[0274] Clause 16. The system of any of clauses 11-15, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: configuring a natural language processing (NLP)-based machine learning model to generate and output the patient scores by (i) configuring a first model to generate and output text scores, wherein a text score comprises a machine-learned likelihood that the patient is affected with the genetic condition given first model input comprising the natural language text; and (ii) configuring a second model to generate and output patient scores using second model input comprising the text score and phenotype data excluding the natural language text.

[0275] Clause 17. The system of clause 16, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: using curriculum learning to configure the NLP-based machine learning model to generate and output the patient scores, wherein the curriculum learning comprises using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

[0276] Clause 18. The system of any of clauses 11-17, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: configuring a natural language processing (NLP)-based machine learning model to generate and output the patient scores; and iteratively adjusting a value of at least one clinical feature weight of the NLP-based machine learning model until the patient scores output by the NLP -based machine learning model satisfies at least one first performance criterion, to produce the configured NLPbased machine learning model, wherein the configured NLP -based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

[0277] Clause 19. The system of any of clauses 11-18, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising:: using positive examples and negative examples to configure the NLP -based machine learning model, wherein a positive example comprises a positive association between a moleculardiagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0278] Clause 20. The system of any of clauses 11-19, wherein the Bayesian causal model comprises a probability distribution of patient scores and variant scores.

[0279] Clause 21. At least one non-transitory machine-readable storage medium comprising at least one instruction that, when executed by at least one processor, causes the at least one processor to perform at least one operation comprising: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication for testing, a family history description, or a demographic description; using the phenotype data including the natural language text, configuring a natural language processing (NLP)-based machine learning model to generate and output patient scores for the patient population, wherein a patient score comprises a machine -learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; extracting genotype data from genetic test results associated with the patient population and the genetic condition, wherein the genotype data comprises a molecular diagnosis comprising at least one variant of a gene; and using the patient scores and the genotype data to generate and output variant scores, wherein a variant score comprises a likelihood that a variant is pathogenic with respect to the genetic condition.

[0280] Clause 22. The at least one non-transitory machine-readable storage medium of clause 21, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using the configured NLP -based machine learning model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition.

[0281] Clause 23. The at least one non-transitory machine-readable storage medium of clause 22, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

[0282] Clause 24. The at least one non-transitory machine-readable storage medium of clause 22, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

[0283] Clause 25. The at least one non-transitory machine-readable storage medium of any of clauses 21-24, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using the variant scores as input to a classification framework.

[0284] Clause 26. The at least one non-transitory machine-readable storage medium of any of clauses 21-25, wherein configuring the NLP -based machine learning model further comprises: configuring a first model to receive the natural language text and generate and output a text score, wherein the text score comprises a machine-learned likelihood that the patient is affected with the genetic condition given thereceived natural language text; and configuring a second model to generate and output the patient score using second model input comprising the text score and second phenotype data not comprising the natural language text.

[0285] Clause 27. The at least one non-transitory machine-readable storage medium of clause 26, wherein configuring the NLP -based machine learning model further comprises: using curriculum learning to configure the NLP -based machine learning model to generate and output the patient scores, wherein the curriculum learning comprises using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

[0286] Clause 28. The at least one non-transitory machine-readable storage medium of any of clauses 21-27, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: iteratively adjusting a value of at least one clinical feature weight of the NLP-based machine learning model until the patient score output by the NLP-based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, wherein the configured NLP -based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

[0287] Clause 29. The at least one non-transitory machine -readable storage medium of any of clauses 21-28, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using positive examples and negative examples to configure the NLP -based machine learning model, wherein a positive example comprises a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

[0288] Clause 30. The at least one non-transitory machine -readable storage medium of any of clauses 21-29, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using a Bayesian causal model to generate and output variant scores, wherein the Bayesian causal model comprises a probability distribution of patient scores and variant scores.

[0289] Clause 31. A method comprising extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication for testing, a family history description, or demographic data; extracting genotype data from genetic test results associated with the genetic condition, wherein the genotype data comprises at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; and using the phenotype data including the natural language text and the genotype data, configuring a machine learning model to generate and output patient scores for the patient population, wherein a patient score comprises a machine-learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text.

[0290] Clause 32. A method comprising extracting genotype data from genetic test results associated with a genetic condition, wherein the genotype data comprises at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; and using patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein avariant score comprises a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

[0291] Clause 33. The method of clause 31 or 32, including any of the preceding clauses.

[0292] In the foregoing specification, embodiments of the disclosure have been described with reference to specific example embodiments thereof. It will be evident that various modifications can be made thereto without departing from the broader spirit and scope of embodiments of the disclosure as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

CLAIMSWhat is claimed is:

1. A method comprising: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication for testing, a family history description, or demographic data; extracting genotype data from genetic test results associated with the genetic condition, wherein the genotype data comprises at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; using the phenotype data including the natural language text and the genotype data, configuring a natural language processing (NLP)-based machine learning model to generate and output patient scores for the patient population, wherein a patient score comprises a machine-learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; and using the patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein a variant score comprises a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

2. The method of claim 1, further comprising: using the configured NLP -based machine learning model and the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition.

3. The method of claim 2, further comprising: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

4. The method of claim 2, further comprising: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

5. The method of claim 2, further comprising: using variant scores output by the configured Bayesian causal model as an input to a variant classification framework.

6. The method of claim 1, wherein configuring the natural language processing (NLP)-based machine learning model further comprises:configuring a first model to receive the natural language text and generate and output a text score, wherein the text score comprises a machine-learned likelihood that the patient is affected with the genetic condition given the received natural language text; and configuring a second model to generate and output the patient score using the text score and second model input comprising the phenotype data excluding the natural language text.

7. The method of claim 6, further comprising: using curriculum learning to configure the NLP -based machine learning model to generate and output the patient scores, wherein the curriculum learning comprises using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

8. The method of claim 1, further comprising: iteratively adjusting a value of at least one clinical feature weight of the NLP -based machine learning model until output of a loss function computed based on the patient score output by the NLP -based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, wherein the configured NLP -based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

9. The method of claim 1, further comprising: using positive examples and negative examples to configure the NLP -based machine learning model, wherein a positive example comprises a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

10. The method of claim 1, wherein the Bayesian causal model comprises a probability distribution of patient scores and variant scores.

11. A system comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises at least one instruction that, when executed by the at least one processor, cause the at least one processor to perform at least one operation comprising: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication for testing, a family history description, or demographic data; using the phenotype data including the natural language text, generating, and outputting patient scores for the patient population, wherein a patient score comprises a likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text;extracting genotype data from genetic test results associated with the patient population and the genetic condition, wherein the genotype data comprises a molecular diagnosis comprising at least one variant of a gene; and using the patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein a variant score comprises a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

12. The system of claim 11, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: using the configured Bayesian causal model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign, of uncertain significance, or pathogenic with respect to the genetic condition.

13. The system of claim 12, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

14. The system of claim 12, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

15. The system of claim 11, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: using variant scores output by the configured Bayesian causal model as an input to a variant classification framework.

16. The system of claim 11, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: configuring a natural language processing (NLP)-based machine learning model to generate and output the patient scores by (i) configuring a first model to generate and output text scores, wherein a text score comprises a machine -learned likelihood that the patient is affected with the genetic condition given first model input comprising the natural language text; and (ii) configuring a second model to generate and output patient scores using second model input comprising the text score and phenotype data excluding the natural language text.

17. The system of claim 16, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising:using curriculum learning to configure the NLP-based machine learning model to generate and output the patient scores, wherein the curriculum learning comprises using the text scores to determine an order for the second model input and applying the second model to the second model input in accordance with the order.

18. The system of claim 11, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: configuring a natural language processing (NLP)-based machine learning model to generate and output the patient scores; and iteratively adjusting a value of at least one clinical feature weight of the NLP-based machine learning model until output of a loss function computed based on the patient scores output by the NLP-based machine learning model satisfies at least one first performance criterion, to produce the configured NLP -based machine learning model, wherein the configured NLP-based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

19. The system of claim 18, wherein the at least one instruction, when executed by the at least one processor, causes the at least one processor to perform at least one operation further comprising: using positive examples and negative examples to configure the NLP-based machine learning model, wherein a positive example comprises a positive association between a molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

20. The system of claim 11, wherein the Bayesian causal model comprises a probability distribution of patient scores and variant scores.

21. At least one non-transitory machine-readable storage medium comprising at least one instruction that, when executed by at least one processor, causes the at least one processor to perform at least one operation comprising: extracting phenotype data from clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication for testing, a family history description, or a demographic description; using the phenotype data including the natural language text, configuring a natural language processing (NLP)-based machine learning model to generate and output patient scores for the patient population, wherein a patient score comprises a machine -learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text; extracting genotype data from genetic test results associated with the patient population and the genetic condition, wherein the genotype data comprises a molecular diagnosis comprising at least one variant of a gene; andusing the patient scores and the genotype data to generate and output variant scores, wherein a variant score comprises a likelihood that a variant is pathogenic with respect to the genetic condition.

22. The at least one non-transitory machine-readable storage medium of claim 21, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using the configured NLP -based machine learning model to generate and output a prediction as to whether a variant associated with an unseen clinical record is benign or pathogenic with respect to the genetic condition.

23. The at least one non-transitory machine-readable storage medium of claim 22, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction as to whether the variant is benign or pathogenic to a clinician for use in formulating, by the clinician, a diagnosis of a patient.

24. The at least one non-transitory machine-readable storage medium of claim 22, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: providing the prediction to at least one system, process, model, or component of a variant classification system or a clinical data system.

25. The at least one non-transitory machine-readable storage medium of claim 21, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using the variant scores as input to a variant classification framework.

26. The at least one non-transitory machine-readable storage medium of claim 21, wherein configuring the NLP-based machine learning model further comprises: configuring a first model to receive the natural language text and generate and output a text score, wherein the text score comprises a machine-learned likelihood that the patient is affected with the genetic condition given the received natural language text; and configuring a second model to generate and output the patient score using second model input comprising the text score and second phenotype data not comprising the natural language text.

27. The at least one non-transitory machine-readable storage medium of claim 26, wherein configuring the NLP-based machine learning model further comprises: using curriculum learning to configure the NLP -based machine learning model to generate and output the patient scores, wherein the curriculum learning comprises using the text scores to determine anorder for the second model input and applying the second model to the second model input in accordance with the order.

28. The at least one non-transitory machine-readable storage medium of claim 21, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: iteratively adjusting a value of at least one clinical feature weight of the NLP -based machine learning model until output of a loss function computed based on the patient score output by the NLP -based machine learning model satisfies at least one first performance criterion, to produce the configured NLP-based machine learning model, wherein the configured NLP -based machine learning model is capable of outputting patient scores that satisfy at least one second performance criterion.

29. The at least one non-transitory machine-readable storage medium of claim 21, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using positive examples and negative examples to configure the NLP -based machine learning model, wherein a positive example comprises a positive association between the molecular diagnosis and the genetic condition, and a negative example includes a negative association between the molecular diagnosis and the genetic condition.

30. The at least one non-transitory machine-readable storage medium of claim 21, wherein the at least one instruction, when executed by at least one processor, causes the at least one processor to perform at least one operation further comprising: using a Bayesian causal model to generate and output variant scores, wherein the Bayesian causal model comprises a probability distribution of patient scores and variant scores.

31. A method comprising : extracting phenotype data from a set of clinical records associated with a patient population and a genetic condition, wherein the phenotype data comprises natural language text comprising at least one of an indication for testing, a family history description, or demographic data; extracting genotype data from genetic test results associated with the genetic condition, wherein the genotype data comprises at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; and using the phenotype data including the natural language text and the genotype data, configuring a machine learning model to generate and output patient scores for the patient population, wherein a patient score comprises a machine -learned likelihood that a patient is affected with the genetic condition given the phenotype data including the natural language text.

32. A method comprising: extracting genotype data from genetic test results associated with a genetic condition, wherein the genotype data comprises at least one variant of a gene and at least one molecular diagnosis associated with the at least one variant; and using patient scores and the genotype data, configuring a Bayesian causal model to generate and output variant scores, wherein a variant score comprises a machine-learned likelihood that a variant is pathogenic with respect to the genetic condition.

Images