Probabilistic variant interpretation

Abstract

Examples may create input data for a causal machine learning model using pathogenicity evidence data associated with a genetic variant and a health condition. Examples may apply a causal machine learning model to the input data to produce a trained causal model. A graphical representation of the trained causal model may include nodes connected via acyclic directed edges. A first node of the nodes may represent a pathogenicity evidence variable related to the health condition. At least one second node may represent a cause of the pathogenicity evidence variable. At least one third node may represent an effect of the pathogenicity evidence variable. An acyclic directed edge may represent a relationship between two of the nodes. Examples may output predictive data sampled from the trained causal model.

Description

PROBABILISTIC VARIANT INTERPRETATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to United States Provisional Patent Application No. 63 / 579,939 filed August 31, 2023 and United States Provisional Patent Application No. 63 / 562,696 filed March 7, 2024, each of which is incorporated herein by this reference in its entirety.TECHNICAL FIELD

[0002] A technical field to which this application relates is genetic testing. Another technical field to which this application relates is machine learning-based systems for genetic variant classification or variant interpretation.BACKGROUND

[0003] Genetic variants are differences in DNA sequences between individuals in a population. There are many different types of variants, including but not limited to structural variations, single-nucleotide polymorphisms, insertion and deletion variations, copy number variations, and translocations and inversions.

[0004] Genetic sequencing technology continues to evolve rapidly. High-throughput sequencing technologies increasingly enable genetic testing spanning genotyping, single genes, gene panels, exomes, genomes, transcnptomes and epigenetic assays for genetic diseases. The increased complexity of analysis and interpretation of clinical genetic testing and the increased volume of testing have been accompanied by new challenges in the interpretation of genetic variants. For example, clinical molecular laboratories are increasingly detecting novel variants in the course of testing patient specimens for a rapidly increasing number of genes associated with diseases. While some phenotypes are associated with a single gene, many are associated with multiple genes.

[0005] The continued expansion of genetic sequencing into more areas of clinical medicine, catalyzed by reduced costs and broadened clinical guidelines, has resulted in an explosion in the number of rare variants observed through testing for hereditary disease. Despite a commensurate increase in the amount of available data with potential use for interpreting these variants, a high percentage (e.g., approximately half) of identified variants in hereditary testing are classified as Variants of Uncertain Significance (VUS).BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The disclosure will be understood more folly from the detailed description given below and from the accompanying drawings of various embodiments of the disclosure. The drawings are for explanation and understanding only and should not be taken to limit the disclosure to the specific embodiments shown.

[0007] FIG. 1 A illustrates an example of a machine learning-based process for variant interpretation, in accordance with some embodiments of the present disclosure.

[0008] FIG. IB illustrates an example of a causal model, in accordance with some embodiments of the present disclosure.

[0009] FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G illustrate examples of probabilistic models for variant interpretation, in accordance with some embodiments of the present disclosure.

[0010] FIG. 3 illustrates an example of a component-based process for creating a causal model for variant interpretation using components of a modeling system, in accordance with some embodiments of the present disclosure.

[0011] FIG. 4 illustrates an example process for training and validating a probabilistic model for variant interpretation, in accordance with some embodiments of the present disclosure.

[0012] FIG. 5A illustrates an example of a variant interpretation system, in accordance with some embodiments of the present disclosure

[0013] FIG. 5B illustrates an example of a component-based process for a variant interpretation system, in accordance with some embodiments of the present disclosure.

[0014] FIG. 6 illustrates an example of an evidence -based variant interpretation platform, in accordance with some embodiments of the present disclosure.

[0015] FIG. 7 is a table listing examples of models that may be included in an evidence-based variant interpretation platform, in accordance with some embodiments of the present disclosure.

[0016] FIG. 8A, FIG. 8B, FIG. 8C, and FIG. 8D illustrates an example of experimental results for a probabilistic variant interpretation model, in accordance with some embodiments of the present disclosure.

[0017] FIG. 9A, FIG. 9B, FIG. 9C, FIG. 9D, FIG. 9E, FIG. 9F, and FIG. 9G illustrate methods, systems, apparatus, and / or non-transitory computer readable media configured for creating probabilistic models and using probabilistic models for variant interpretation, in accordance with some embodiments of the present disclosure.

[0018] FIG. 10 illustrates an example computing system that includes a probabilistic variant interpretation model in accordance with some embodiments of the present disclosure.

[0019] FIG. 11 is a block diagram of an example computer system in which aspects of the present disclosure can operate.DETAILED DESCRIPTION

[0020] In some areas, variant classification or variant interpretation interchangeably may refer to a process of interpreting information about genetic variants based on evidence supporting or rejecting a causal relationship with a given disease. In some areas, variant classification may refer to the process of classifying a variant based on a combination of evidence while variant interpretation may refer to the use of variant classification information to make diagnostic decisions. In some areas, variant interpretation may refer to the process of assessing variant information in order to apply a classification and inform one or more clinicians to make a diagnostic decision(s). The amount and significance of any given piece of evidence can vary depending on the variant, the gene, the disease, and / or relationships of the evidence toother pieces of evidence. Currently, variant interpretations do not themselves provide diagnoses, but can be used by clinicians to make diagnostic decisions.

[0021] The primary purpose of clinical genetic testing is to identify genetic variants in individuals and determine, for each variant, whether it is pathogenic (has disease-causing potential) or benign (may not have disease-causing potential). Currently, that question cannot be definitively answered for many variants, either due to insufficient data or inadequate tools to evaluate the available data. Certain guidelines attempt to estimate the likelihood that a variant is pathogenic or benign using a rules-based system such as the American College of Medical Genetics and Genomics (ACMG) guidelines and the Sherloc framework (see. e.g., Nykamp K, Anderson M, Powers M, et al. Sherloc: a comprehensive refinement of the ACMG- AMP variant classification criteria. Genet Med. 2017; 19(10): 1105-1117). Given a set of data and information as input, these other systems use heuristics to convert data into qualitative evidence bins, which are then combined to output qualitative classifications. For example, these other systems may assign each piece of data to a qualitative category such as Very Strong, Strong, Moderate or Supporting evidence based on expert intuition, which introduces biases, subjectivity and potential for inconsistencies. Furthermore, in such systems, data is lost in the qualitative categorization process because the qualitative evidence bins are crude. For example, if a piece of data is more compelling than “Moderate” but not quite at the “Strong” threshold, either evidence bin may be selected for the data, depending upon the judgment of the domain expert or historical examples, thus reducing precision. The other systems then classify a variant as pathogenic (P), likely pathogenic (LP), variant of uncertain significance (VUS), likely benign (LB), or benign (B) based on the particular combination of evidence available for that variant. For example, a variant with one strong and two moderate evidence may be identified as likely pathogenic. As with the data binning, the other systems perform these categorical assignments based on intuition and as such, introduce subjectivity. Moreover, interaction evidence may be more complex than simply summing the evidence. For example, there may be synergistic interactions between the different types of evidence or partially overlapping or redundant evidence that should not simply be summed. Although the ACMG guidelines target 90%~99% certainty as for the two “likely” categories (likely pathogenic and likely benign) and a >99% certainty for pathogenic and benign, this certainty-based thresholding is aspirational in nature, and the guidelines do not produce quantitative classifications. Because the other systems are qualitative by nature they do not measure accuracy or precision of the resulting variant classifications or variant interpretations. Consequently, these systems likely introduce subjectivity and inconsistencies, producing sub-optimal results for patients and clinicians managing those patients. Additionally, these other systems restrict the clinical utility of genetic test results by producing only a minimal set of outcomes that can be used as medical management decision-making boundaries. Current management recommendations set a single boundary where LP and P variants may be deemed medically actionable, while VUS, LB and B are deemed not medically actionable; this single boundary is used regardless of gene, disease, and the medical intervention.

[0022] The described approaches, which may be implemented using methods, systems, apparatus, non-transitory computer readable media, etc., are designed to achieve a transition from the existing, sub-optimal qualitative systems to a quantitative variant interpretation platform that can consume all available relevant data for any given variant, any gene, in real time, and produce an accurate and precise quantitative computation of the probability of pathogenicity (PoP). The described approaches include methods for evaluating data associated with a variant to quantify the informativeness of that data as evidence for the variant being pathogenic or benign. Examples use a machine learning model to calculate the PoP of a variant, given the complex constellation of phenotypes observed in a collection of patients observed with this variant. Furthermore, the described approaches include methods for evaluating the PoP of a variant, given a set of evidence (which may include quantitatively measured and / or qualitatively binned evidence). For example, the set of evidence may include model prediction scores (i.e., quantitative evidence) and / or criteria (i.e., qualitatively binned evidence) based on in vitro functional assay results, population allele frequency data, changes in mRNA splicing and / or predictions of nonsense-mediated decay for a given variant.

[0023] Conventional machine learning models are correlative, not causal, in that those models do not necessarily consider the causal relationships between variables. This aspect of conventional models has statistical ramifications because the models cannot leverage known statistical relationships between variables. In a correlative model, it may be hard to know how to interpret the model output when two predictive variables are correlated with an outcome like pathogenicity. For instance, if the two variables represent two independent biological mechanisms (causes) of pathogenicity (e.g., protein abundance and protein stability), then just one of those variables may be sufficient to predict pathogenicity while the other is inconsequential. On the other hand, if the two variables are causally downstream (effects) of pathogenicity, then a mismatch between their respective predictions (e.g., high allele frequency in a healthy population cohort, versus strong correlation with the disease phenotype in an affected cohort) should increase the uncertainty in the pathogenicity prediction. A correlative model may not be able to make these kinds of distinctions between upstream and downstream variables. Furthermore, correlative modeling approaches may produce models that are biologically nonsensical and fail to generalize to broader datasets. In contrast, a causal probabilistic graphical model relies on domain knowledge to define causal relationships between variables.

[0024] In a probabilistic model, as an example, if three variables are connected sequentially in a graph, e.g., A predicts B and B predicts C, then, given an estimate for the value of A, a probabilistic model can be used to predict likely values for C. Conventional models take point values as estimates and generate point values as output. Probabilistic graphical models, on the other hand, can keep track of entire probability distributions, not just point values. Whereas in a conventional model, the input for A might be a point value like “0.5,” in a probabilistic graphical model, the input for A could be a distribution, such as a bell curve centered at 0.5.

[0025] The ability to consider the distribution of possible values of evidence and predictions can be very useful in common situations, such as when measurements have some degree of uncertainty, and / or when the modeling task has true randomness in the process. For example, having a BRCA1 mutation does not guarantee that a person will develop cancer, but it does increase the person’s cancer risk. For an incompletely penetrant disease, seeing a particular variant lead to disease in 50% of individuals couldindicate that the variant is causal, but it is by no means a guarantee that the variant is causal. Probabilistic modeling can both measure the likelihood that a variant is the root cause of a disease and account for uncertainty in the prediction that the variant is the root cause of a disease.

[0026] In some examples, the described approaches include a scalable model that utilizes natural language processing (NLP) alone or in combination with other evidence modeling techniques, and Bayesian inference to predict variant pathogenicity (e.g., from clinical phenotype data reported by ordering providers, such as in test requisition forms). In some examples, an NLP classifier is used to learn features of the indication and family history fields that are predictive of a molecular diagnosis of a given condition. A classifier combines these features and demographic information for a given patient to determine a patient score, e.g., the probability that a given patient is affected with the condition. Patient scores are computed in this manner for a population of patients. A generative, hierarchical Bayesian inference model is fit to these patient scores and labeled variant pathogenicity data, if available. Sampling the posterior predictive distribution from the inference model yields a variant score, e.g., the probability that a given variant is pathogenic for the associated condition. In this way, the described approaches can account for both uncertainty in the affected status of each patient observation and uncertainty due to the number of observations for each variant.

[0027] In some examples, the described approaches include a scalable model that utilizes evidence modeling techniques, a broad set of biological variables representing molecular and / or cellular causes of pathogenicity, a broad set of variables representing the observable effect or impact of pathogenicity in human individuals, and Bayesian inference to predict variant pathogenicity.

[0028] In some embodiments or examples, with probabilistic graphical modeling (PGM), a directed acyclic graph (DAG) is used to express conditional dependence structures between random variables. The nodes of a DAG represent conditionally independent variables. The edges, depicted as arrows, of the DAG imply dependence between corresponding nodes with the directionality of the arrows implying the direction of causality. In some examples, PGM implementations include generation of a DAG that represents causal relationships between a broad set of biological variables relevant to assessing pathogenicity, including the nodes, the edges, and the directionality of the edges.

[0029] Examples of the described DAG modeling approaches include a first node, representing pathogenicity, in the center with upstream (cause of pathogenicity) variables above and pointing to this center node, and / or downstream (effect of pathogenicity) variables below and pointing from this center node. Upstream nodes above the center node includes variables that represent data pertaining to biological properties that cause pathogenicity, such as variables pertaining to the impact of the variant on protein function, variables pertaining to the impact of the variant on protein stability, variables pertaining to the impact of the variant on mRNA processing such as mRNA splicing, variables pertaining to the impact of the variant on nonsense mediated decay, variables pertaining to the impact of the variant on mRNA and / or protein expression, variables pertaining to the biological criticality of the disrupted DNA and / or RNA nucleotide and / or protein amino acid residue, such as those ascertained from evolutionary conservation data, etc.

[0030] Downstream nodes below the center node include variables that represent data pertaining to properties that are the effect of pathogenicity, such as variables pertaining to the impact of pathogenicity on individual patient phenotype, impact of pathogenicity on the general population allele frequency, impact of pathogenicity on a select cohort allele frequency, impact of pathogenicity on familial genotypephenotype co-segregation patterns, etc. For any upstream or downstream node, there may be additional nodes and edges that represent one or more other relevant variables. For example, a first node pertaining to the impact of the variant on mRNA processing may be joined to a second node that represents data from in vitro mRNA splicing assay, and / or a third node that represents data from an in silico mRNA splicing prediction algorithm, and / or a fourth node that represents positional information relative to consensus splice sequence.

[0031] The relationships represented visually by the DAG are encoded in the model by specifying the probability of the available observations in terms of the variables and parameters in the model. For example, the variable “pathogenicity” is only partially observed. It is represented in terms of all of its parent variables with a function, for example, either a linear model or a deep learning model may be valid. Observations of the variable (e.g., known labels for existing variants) and observations of all upstream variables are used to fit the model by maximizing the likelihood of all those parameters given the data. The parameters learned from the fitting process can be used to sample the distribution for the variable 'pathogenicity,' or any other variable, when the variable is unobserved.

[0032] In some examples, an evidence -based variant modeling platform includes many different evidence models, each or any of which can be used alone or in combination for variant interpretation. For example, models can be grouped according to the categories of evidence criteria used by the Sherloc framework (e.g., assay data, in silico data, population data, etc.), and each grouping can include multiple alternative types of models for predicting pathogenicity given the associated type of evidence. For each grouping of models, the best-performing model based on various performance metrics can be selected for a particular gene. The selected model can then be used in existing variant interpretation systems such as the ACMG guidelines and the Sherloc framework, or alternatively can be used as a node in PGM. Alternatively or in addition, the Bayesian approaches described herein can be used to develop any one or more of the models in any of the groupings on the platform, which can also be used in existing variant interpretation systems or as a node in a PGM.

[0033] The described modeling strategies have been applied to a proprietary genotype / phenotype database, composed of data collected with permission from thousands (e.g., one thousand, ten thousand, one hundred thousand, five hundred thousand) or millions (e.g., 1 million, 2 million, 5 million, 10 million, 20 million, 50 million, 100 million) of patients referred for clinical genetic testing for hereditary conditions and thousands (e.g., one thousand, ten thousand, one hundred thousand, five hundred thousand) or millions (e.g., 1 million, 2 million, 5 million, 10 million, 20 million, 50 million, 100 million) classified genetic variants. In one embodiment, the described approaches yielded high-performance (AUROC > 0.8, as evaluated on a holdout set of labeled variants in each gene) models for over 1100 genes associated with a broad range of clinical areas (e.g., oncology, cardiology, neurology, metabolic). In this embodiment,over 20,000 VUS across these genes and conditions received high confidence probabilistic predictions (PoP greater than 0.95 or less than 0.05) that could result in new evidence for interpretation. .

[0034] In some examples, combined machine learning-based models are generated to calculate the PoP from various individual machine learning -based models considered together. Each of the machine learning-based models is constructed and trained using large datasets containing thousands to millions of data records and automated and / or semi-automated processes. The outputs of several machine learningbased models (e.g., in silico model for a particular gene + assay model for that particular gene + population model) can be used as individual nodes in a PGM. Thus, the PGM itself is a composition of multiple different machine learning-based models, each configured and trained using large datasets and automated and / or semi-automated processes as mentioned above, and the relationships between these models are machine -learned using automated and / or semi-automated tools. The described approaches demonstrate the utility of a Bayesian model to effectively integrate predictive data produced by many different models, with significant potential to reduce VUS, accelerate genetic diagnoses, and improve treatment personalization for patients and providers.

[0035] The disclosure will be understood more folly from the detailed description given below, which references the accompanying drawings. The detailed description of the drawings is for explanation and understanding, and should not be taken to limit the disclosure to the specific embodiments described.

[0036] In the drawings and the following description, references may be made to components that have the same name but different reference numbers in different figures. The use of different reference numbers in different figures indicates that the components having the same name can represent the same embodiment or different embodiments of the same component. For example, components with the same name but different reference numbers in different figures can have the same or similar functionality such that a description of one of those components with respect to one drawing can apply to other components with the same name in other drawings, in some embodiments.

[0037] Also, in the drawings and the following description, components shown and described in connection with some embodiments can be used with or incorporated into other embodiments. For example, a component illustrated in a certain drawing is not limited to use in connection with the embodiment to which the drawing pertains, but can be used with or incorporated into other embodiments, including embodiments shown in other drawings.

[0038] FIG. 1 A illustrates an example of a machine learning-based process for variant interpretation, in accordance with some embodiments of the present disclosure. In the example of FIG. 1A, phenotype data, genotype data, and domain knowledge are used to construct a causal machine learning model that models relationships between different types of pathogenicity evidence for a given combination of genetic variant and health condition using a Bayesian inference approach. Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference uses prior knowledge in the form of a prior distribution in order to estimate posterior probabilities. A prior probability distribution of an uncertain quantity, often simply called the prior, is its assumed probability distribution before some evidence is taken into account. The posterior probability is a type of conditional probability that resultsfrom updating the prior probability with information summarized by a likelihood function via an application of Bayes’ theorem. A likelihood function (often simply called the likelihood) measures how well a statistical model explains observed data by calculating the probability of seeing that data under different parameter values of the model. The likelihood function is constructed from the joint probability distribution of the random variable that (presumably) generated the observations.

[0039] In FIG. 1A, person one and person two represent two examples of subjects in a population N, although any population N can include any number M of subjects (where N and M are positive integers, and the value of M can be different for different populations). A population N includes M human subjects who have participated in genetic testing, where N represents one or more defining characteristics of the population such as demographic criteria and M represents the number of subjects in that population. For example, M may be ten human subjects, one hundred human subjects, one thousand human subjects, ten thousand human subjects, one hundred thousand human subjects, one million human subjects, two million human subjects, five million human subjects, ten million human subjects, one hundred million human subjects, or a billion human subjects. Each subject in the population N has associated phenotype data 110 and genotype data 112. The phenotype data 110 includes observable characteristics or traits of a subject, such as morphology, developmental processes, biochemical and physiological properties, behaviors, and results of behaviors. For a given subject, the phenotype data 110 can include demographic information, family history of diseases and other health conditions, and indications. Indications include signs, symptoms, or medical conditions that suggest a treatment, test, or procedure may be advisable or necessary. Indications can also refer to the reason a medication or other treatment might be used. For example, aspirin is indicated for adults with risk factors for cardiovascular disease, such as hypertension or diabetes. The phenotype data 110 is collected and used with the subjects’ permission. For example, the phenotype data may be collected via test requisition forms filled out by a subject or a clinician.

[0040] The genotype data 112 includes information about the genetic makeup of each individual subject in the population N. For instance, a subject may submit a DNA (deoxyribonucleic acid) sample for genetic testing using DNA sequencing, and the results of the genetic testing provide information about the subject’s genetic makeup. A DNA sample includes one or more genes 102. Each gene includes one or more regions 104, e.g., contiguous groups or sequences of nucleotides or proteins (e.g., A G A C G C T, where A refers to adenine, G refers to guanine, C refers to cytosine, T refers to thymine). Each nucleotide in a region has one or more positions at which genetic variants may be located. For instance, in FIG. 1 A, person one of population N has cytosine at position 106 of region 104 of their copy of gene 102, while person two of population N has thymine at position 106 of region 104 of their copy of gene 102. In the example of FIG. 1A, thymine is considered a genomic variant 108. A source of genotype data 112 can be a publicly available population database, such as gnomAD, a private population database, or any database with population data.

[0041] For each population in the one to N human populations (where each such population contains M subjects), the genotype data 112 can include gene-level data corresponding to each gene 102, regionlevel data corresponding to one or more regions 104 of a gene, position-level data corresponding to one or more positions 106, and variant-level data corresponding to one or more genomic variants 108. Gene-leveldata refers to any property, attribute, or metric that is determined for a defined sequence of DNA identified as a gene. For example, the gene-level data includes information that is specific to a particular gene, such as gene length. Region-level data refers to any property, attribute, or metric that is determined for a region of a gene, i.e., a defined portion of a gene that is typically largerthan a position but smaller than the entire gene. The region-level data includes information that is specific to a particular region of the gene. For example, region-level data can include a property of a region that includes a variant. A region can refer to a portion of a gene that includes a variant and one or more adjacent or neighboring nucleotides. Position-level data refers to any property, attribute, or metric that is determined for a position within a gene, i.e., a single nucleotide or amino acid position in the gene. The position-level data includes information that is specific to a particular position of the gene regardless of whether a variant is present at that position. Whereas typically, a gene can be thousands of nucleotides long, a variant typically occurs at a single position among those thousands of nucleotides. The variant-level data includes information about a particular variant located at a particular position of the gene. A variant can include one nucleotide or more than one nucleotide. When a variant includes more than one nucleotide, the position of the variant refers to a region, such that the terms position and region may be synonymous in that context.

[0042] Domain knowledge 114 can include learned information, such as quantitative and / or qualitative compilations of clinical expertise and / or other evidence of one or more characteristics of one or more genetic variants (including, but not limited to, pathogenicity). The domain knowledge 114 is stored in one or more datasets and / or encoded in one or more machine learning models. For example, domain knowledge 114 can include statistical correlations between different types or categories of evidence of pathogenicity of a variant with respect to a given health condition. One example of domain knowledge is a population frequency model that models correlations between allele frequencies and variant pathogenicity. Another example of domain knowledge is historical variant interpretation information, for example, historical output of a variant classification framework, where the framework may score different pieces of pathogenicity evidence differently for different populations, genes, variants, and / or health conditions, and the history of these scores can be used to develop probabilistic relationships between evidence types and pathogenicity. An additional example of domain knowledge can include cause and effect relationships between different types or categories of evidence of pathogenicity.

[0043] For instance, domain knowledge 114 can be used by causal model builder 118 to construct, train, and / or validate a causal model (e.g., a probabilistic graphical model such as a directed acyclic graph that represents a hierarchical Bayesian network). Domain knowledge 114 can include one or more outputs, or a combination of outputs, generated by an evidence modeling platform (EMP), such as scores, distributions, or labels, examples of which are described herein. Additional examples of domain knowledge 114 include one or more, or a combination of any of the following: CVM (patient clinical data evaluated through an EMP), PFM (population frequency data evaluated through an EMP), FIM (evolutionary conservation, protein structure and protein stability data evaluated through an EMP), DMA (published in vitro experimental study data evaluated through an EMP), DML (internally performed in vitro experimental study data evaluated through EMP), splice site loss predictions obtained via an open source model such as PANGOLIN, splice site gain predictions (obtained via, e.g., PANGOLIN), output ofa vanant classification framework (e.g., prediction of nonsense mediated decay, human-curated data based on the application of the Sherloc framework).

[0044] Data pre-processing 116 includes one or more components that select data to be used to build the causal model from among the sources of phenotype data 110, genotype data 112, and domain knowledge 114. For example, given a variant and health condition, data pre-processing 116 may filter one or more of the phenotype data 110, genotype data 112, and domain knowledge 114 to exclude less predictive data and include more highly predictive data. If available, data pre-processing 116 may join known pathogenicity labels with associated variant information. Data pre-processing 116 may transform one or more of the phenotype data 110, genotype data 112, and domain knowledge 114 into a format that is suitable or required for input to the causal model. Data pre-processing 116 is optional in some examples. For instance, the phenotype data 110, genotype data 112, and / or domain knowledge 114 may be obtained from sources in which they are already pre-formatted such that no pre-processing is needed The output of data pre-processing 116 can include training and / or validation data sets and / or one or more structured representations of phenotype data 110, genotype data 112, and / or domain knowledge 114, such as feature sets, embeddings, tensors, rules, weights, parameters, etc.

[0045] Causal model builder 118 applies the domain knowledge 114 and / or output of data preprocessing 116 to construct a graphical representation of the causal model or selects a model structure for the causal model from a library of available model architectures. The graphical representation of the causal model includes nodes and edges, where each node represents a type or category of evidence and each edge represents a type of relationship between two nodes. In a causal model, or probabilistic graphical model (PGM), the edges may be acyclic directed edges. Causal model builder 118 uses the domain knowledge 114 and / or output of data pre-processing 116 to create nodes that map to evidentiary categories, create and weight edges between the nodes, and determine the overall topography of the resulting graphical mode.

[0046] Training / validation 120 applies the causal model constructed or selected by causal model builder 118 to training and validation data sets, which are prepared using inputs including portions of one or more of the phenotype data 110, genotype data 112, and domain knowledge 114. While not specifically shown in FIG. 1A, the training and validation processes, including respective data sets, are shown and described in more detail in subsequent figures. Once the causal model has met or exceeded applicable performance criteria through training and validation 120, the trained and validated causal model is made available for responding to queries, e.g., sampling, via model serving / inferencing component 122. Model serving / inferencing component 122 provides access to the causal model via, e.g., one or more application programming interfaces (APIs), user interfaces, batch processes, query mechanisms, etc. Via model serving / inferencing component 122, variant interpretation data 124 can be sampled and output for use by, e.g., an engineer, researcher or clinician to, for example, assist with a diagnosis, tune the causal model, update the domain knowledge 114, etc. Variant interpretation data 124 can include one or more values representing one or more probabilities of pathogenicity (PoP), a portion or the entirety of the causal model, the updated domain knowledge 114, a graphical representation of the causal model, etc. Other examples of variant interpretation data 124 include a probability that a patient will exhibit a particularphenotype (beyond the probability that the variant is pathogenic, the risk of disease for a given patient, a risk of a particular phenotypic manifestation) and a predicted functional / cellular assay output before the assay is performed, which could be useful for prioritizing experiments. An example use of variant interpretation data 124 is to simulate the impact of missing data, e.g., the predicted effects on the model outputs (such as PoP) if more data (e.g., more patient information, more family history information, functional assay results, etc.) were available. Another example use of variant interpretation data 124 is to identify and highlight specific pieces of information that are missing but needed to make a proper classification and thus inform patient care (for example, a specific type of blood test is needed for a proper classification).

[0047] The components and processes shown in FIG. 1A are described in more detail with reference to subsequent figures. The examples shown in FIG. 1A, and the accompanying description above, are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0048] FIG. IB illustrates an example of a causal model, in accordance with some embodiments of the present disclosure. The example of FIG. IB is illustrative and non-limiting.

[0049] In the example of FIG. IB, a causal model 150 is expressed in graphical form as a probabilistic graphical model (PGM), or more specifically as a directed acyclic graph (DAG), or even more specifically, as a Bayesian network represented as a DAG. A PGM is a modeling framework that uses graphs to represent conditional dependencies between random variables. A DAG is a specific type of graph that can be used within a PGM. In a DAG, each edge expresses a direction of causality by way of an arrow, where the head of the arrow indicates the direction of causality, the tail of the arrow connects to a node that represents a causal variable, and the arrow head connects to the node that is affected by the causal variable. Also in a DAG, there are no cycles, meaning that no path through the graph can return to the same node (no loops). A Bayesian network is a type of PGM that can be used for causal inference and probabilistic reasoning, in which the graph structure encodes the factorization of the joint probability distribution of the variables. As described with reference to the examples shown in FIG. IB, FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G, PGMs can encode complex distributions over multiple different variables.

[0050] The causal model 150 is a probabilistic graphical modeling (PGM) represented as a directed acyclic graph, which is used to express conditional dependence structures between random variables. The nodes of the causal model 150 (e.g., node 1, node 2, node 3, node 4) represent independent variables. The edges (e.g., E12, E13, E14, E24, E34) represent dependencies between connected nodes, where the directionality of the arrows on the edges represents the direction of causality from one node to another node. For example, in the causal model 150, the edge E12 represents a conditional dependency between the independent variable represented by node 1 and the independent variable represented by node 2. Node 2 is conditionally independent of Node 3 given the observation of node 1. The edge E13 represents a conditional dependency between the independent variable represented by node 1 and the independent variable represented by node 3. The edge E14 represents a conditional dependency between the independent variable of node 1 and the independent variable of node 4. The edge E24 represents aconditional dependency between the independent variables of node 2 and node 4, which also implicitly encodes the relationship between node 1 and node 2. The edge E34 represents a conditional dependency between the independent variables of node 3 and node 4, which also implicitly encodes the relationship between node 1 and node 3.

[0051] As applied to the problem of variant classification or variant interpretation, each node of the causal model 150 represents a type of evidence, where the type of evidence may be related to pathogenicity of one or more variants with respect to one or more health conditions and / or some other characteristic of one or more variants or health conditions. Nodes are based on biological properties or any other relevant information, and the number of nodes in the causal model 150 is determined using domain knowledge as to the types of evidentiary variables that are useful, available, reliable, etc.

[0052] Each edge computationally represents the relationship between the types of evidence represented by the nodes connected by the edge. For instance, different types of evidence variables can alternatively or in combination impact other evidence variables. Each node can represent output of a particular type of evidence model, and each relationship between nodes can be further represented by another machine learning model, a mathematical model, function, a heuristic, a linear model with weights, a deep learning model, or any other mathematical or logical function.

[0053] In some examples, each node of the causal model 150 represents a type of evidence modeled by an evidence modeling platform such as platform 600 described with reference to FIG. 6. The configuration and topology of the causal model created by causal model builder 118 can vary depending upon the available types of evidence, reliability of each type of evidence (e.g., weight values assigned by an EMP or variant classification framework), and / or other factors.

[0054] In some examples of the described PGM implementations, a DAG is generated using domain knowledge so that it accurately represents assumed causal relationships between biological variables relevant to assessing variant pathogenicity, including the nodes, the edges, and the directionality of the edges. For instance, as applied to variant interpretation, a DAG is described as having a single node, representing pathogenicity, in the center (e.g., node 4 of causal model 150) with upstream variables above and pointing to this center node (e g., node 2 and node 3), or downstream variables below and pointing from this center node (e.g., node 5 and node 6).

[0055] The upstream or parent nodes above the center node includes variables that represent data pertaining to biological properties that cause pathogenicity, such as variables pertaining to the impact of the variant on protein function, variables pertaining to the impact of the variant on protein stability, variables pertaining to the impact of the variant on mRNA processing such as mRNA splicing, variables pertaining to the impact of the variant on nonsense mediated decay, variables pertaining to the impact of the variant on mRNA and / or protein expression, and variables pertaining to the biological criticality of the disrupted DNA and / or RNA nucleotide and / or protein amino acid residue, such as those ascertained from evolutionary conservation data.

[0056] Downstream or child nodes below the center node include variables that represent data pertaining to biological properties that are the effect of pathogenicity: variables pertaining to the impact of pathogenicity on individual patient phenotype, impact of pathogenicity on the general population allelefrequency, impact of pathogenicity on a select cohort allele frequency, and impact of pathogenicity on familial genotype -phenotype co-segregation patterns. For nodes both above and below the central pathogenicity node, there may be additional nodes and edges that represent other relevant variables (e.g., node 1). For example, a node pertaining to the impact of a variant on mRNA processing may be joined to: a node that represents data from in vitro mRNA splicing assay, a node that represents data from an in silico mRNA splicing prediction algorithm, and a node that represents positional information relative to consensus splice sequence.

[0057] The relationships represented visually by the DAG are encoded in the model by specifying the likelihood, e g., the probability, of the available observations in terms of the variables and parameters in the model. For example, a variable such as 'pathogenicity' may be only partially observed, and therefore is represented in terms of all of its parent variables with a function; for example, a linear model in the context of a GLM (generalized linear model), or a deep learning model both may be valid. Observations of the variable (e.g., known labels for existing variants) and observations of all upstream variables are used to fit the model by maximizing the likelihood of all those parameters given the data. Then the learned parameters can be used to sample the distribution for the variable 'pathogenicity' when it is unobserved.

[0058] As shown by way of FIG. IB, FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G, the structure of the causal model 150 is flexible, adaptable, and able to be rebuilt and redesigned as new evidence is obtained, as new information about the supporting models is obtained (e.g., changes in weightings of the evidence models provided by an EMP), etc. In some examples, the causal model 150 is a unified model that has the same topology, structure, variables, parameters, etc., for all genetic variants, which may or may not correspond to the example shown in FIG. IB, or any of the examples shown in FIG. IB, FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G. In other examples, e g., as shown by FIG. 2C and FIG. 2D, the causal model 150 can be modified or adapted for different variants such that different configurations of the causal model 150 are constructed, trained, maintained, and served for different variants, groups of variants, genes, or groups of genes.

[0059] A new variable can be added to causal model 150 using domain knowledge to determine an appropriate representation of the new variable (e.g., what type of model best represents the variable, such as a distribution or a point value), the relative positioning of the variable with respect to other variables represented by nodes in the graph (e.g. independence or conditional independence, with respect to other variables), the relationships between the causal variables and the variable of interest and the downstream variables and the variable of interest (e.g., by determining the causal functions by which different variables are connected), encoding the representation and relationships into the causal model 150 using a modeling tool (e.g., software) and retraining the model. Variables and / or relationships between variables, and / or other aspects of the model, can be updated in a similar way.

[0060] The components and processes shown in FIG. IB are described in more detail with reference to subsequent figures. The examples shown in FIG. IB, and the accompanying description above, are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0061] FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G illustrate examples of probabilistic models for variant interpretation, in accordance with some embodiments of the present disclosure. In the illustrated examples, the probabilistic models are described using graphical representations, e.g., directed acyclic graphs including nodes and acyclic directed edges between the nodes.

[0062] Any one or more of the examples shown in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G can be pre -constructed and stored, for example, in a model library. These examples and / or variations thereof are constructed using domain knowledge (e.g., domain knowledge 114 of FIG. 1A). For example, domain knowledge is used to identify evidence categories for inclusion in a probabilistic model and to exclude evidence categories that are not relevant to variant classification or variant interpretation. Evidence categories included in the probabilistic model are represented by nodes of a directed acyclic graph (DAG). Domain knowledge is also used to designate causal relationships between the categories of evidence (e.g., A causes B, B causes C). These causal relationships are represented as acyclic directed edges between the nodes of the DAG.

[0063] Through a training process, weights can be assigned to the nodes and / or edges, where the weight values reflect the varying degree of availability or reliability of different types of evidence. For example, a node represents a variable that corresponds to an evidence category. The variable can be independent or conditionally independent, for example. Also, the variable can be expressed as a distribution or a point value. Domain knowledge and / or other methods allowing to learn and estimate parameters can be used to parameterize the variable. For instance, the plausible range for the prior probability of pathogenicity for a variant in a given gene can be determined based on domain knowledge, such as the observed pathogenicity rate in that gene. However, if an input’s distribution is not well understood via domain knowledge, other tools can be used to estimate the distribution for the input.

[0064] Additionally or alternatively, domain knowledge and / or other methods or tools can be used to determine how different variables (e.g., evidence categories) relate to each other, and these relationships can be encoded in the link function of the model. For instance, knowledge that two variables are linearly anticorrelated is an example of domain knowledge that can be encoded into the model.

[0065] FIG. 2A illustrates a probabilistic graphical model 200 with a central node 202, an upstream node 204, and a downstream node 206. The central node 202 represents the pathogenicity variable 202. For example, sampling the pathogenicity node 202 provides a probability that a variant is pathogenic with respect to a disease (or health condition), and this probability is influenced by the upstream node 204. The upstream node 204 represents a causal variable, e.g., a variable that, given the functional impact of a variant, has a likelihood of causing a disease. In the example of FIG. 2A, the node 204 represents the altered protein function variable. The relationship between the altered protein function variable and the pathogenicity variable is represented by directed edge 203. The direction of the edge 203 indicates the direction of causality. As such, the edge 203 indicates that the altered protein function variable is causal of pathogenicity.

[0066] The downstream node 206 represents an effect variable, e.g., a variable that, given how common the variant is in the general population, has a likelihood that the variant was pathogenic. In theexample of FIG. 2A, the node 206 represents the observations in population databases. The relationship between the observations represented by node 206 and the variant pathogenicity vanable is represented by directed edge 205. The direction of the edge 205 indicates the direction of causality. As such, the edge 205 indicates that the observations in population databases represented by node 206 are an effect that results from variant pathogenicity.

[0067] FIG. 2B illustrates another example of a probabilistic graphical model 210 with a central node 212, an upstream node 214, and two downstream nodes 216, 218. The central node 212 represents the variant pathogenicity variable. The upstream node 214 represents a causal variable, e.g., EMP model output from an MSE (molecular stability engine, which predicts on the structure and stability of a protein with a given genetic variant). The relationship between the upstream node 214 and the central node 212 is represented by directed edge 213. The direction ofthe edge 213 indicates the direction of causality. As such, the edge 213 indicates that the MSE (molecular stability engine) variable is causal of pathogenicity. In some examples, the MSE is a model that is supported by an EMP and the node 214 represents the output ofthe MSE, e.g., a distribution representing molecular stability.

[0068] The downstream nodes 216, 218 each represent an effect variable, e.g., a variable that, given how common the variant is in the general population, has a likelihood that the variant was pathogenic. In the example of FIG. 2B, the node 216 represents an effect variable, population frequency model, e.g., a model of correlations between population allele frequency and pathogenicity. The node 218 represents an effect variable, e.g., an EMP model based on evolutionary conservation data. The relationship between nodes 212 and 216 is represented by directed edge 215, and the relationship between nodes 212 and 218 is represented by directed edge 217. The direction ofthe edges 215, 217 indicates the direction of causality. As such, the edges 215, 217 indicate that the variables represented by nodes 216, 218 are effects that result from variant pathogenicity.

[0069] FIG. 2C illustrates another example of a probabilistic graphical model 220 with a central node 224, an upstream node 234, and downstream nodes 226, 228, 230. In the example of FIG. 2C, a variantlevel model containing variant-level variables represented by nodes 224, 226, 228, 230 is extended to include gene-level variables at nodes 234, 236. The directed edges 225, 227, 229, 235, 237 represent causal relationships between the nodes connected by the edges, respectively.

[0070] In the example of FIG. 2C, node 228 represents an allele number (AN) in a population database such as gnomAD, which reflects the number of individual chromosomes sampled (e.g., representative of sampling size). Node 230 represents an allele count (AC) in a population database such as gnomAD, which reflects the number of times a particular variant was detected. Together with allele number (node 228), these variables are used to calculate allele frequency (node 226) and confidence in the estimate via understanding of sample size. Node 234 represents the label proportions (Ip) variable, which is the proportion of known pathogenic and known benign labels in a dataset such as gnomAD. The label proportions are used to determine the prior probabilities. Node 236 represents the beta distributions (P_i) variables, which are parameters to the AF distribution defined at the gene level (i.e ., specific to each gene).

[0071] FIG. 2D illustrates another example of a probabilistic graphical model 240 similar to the example of FIG. 2C, including both variant-level variables 244 and gene-level variables 242, except that the model 240 imposes a constraint 246 on the label proportion variable 249 and the Beta hyperparameters 248. The constraint 246 can enable the model to be used for genes that have variables that satisfy the constraint 246. As such, the model 240 can be used for a group of genes that all satisfy the constraint 246 (e.g., rather than having to build a separate model for each gene in the group).

[0072] FIG. 2E illustrates another example of a probabilistic graphical model 250 with a central node 252, upstream nodes 254, 256, 262, and downstream nodes 264, 266, 270. In the example of FIG. 2E, each of the upstream nodes 254, 256, 262 and downstream nodes 264, 266, 270 represents a probabilistic graphical model specific to the variable represented by the respective node 254, 256, 262, 264, 266, 270. The variable-specific sub-models are each probabilistic graphical models in the illustrated example but could be other types of models.

[0073] In the example of FIG. 2E, nodes 254, 256, and 262 are each representative of a variable that represents a biological concept related to a cause of pathogenicity (e.g., disruption of protein effects, mRNA processing, or cell / tissue function by the variant could cause pathogenicity). Nodes 264, 266, 270 each are representative of a variable that represents an observable effect of pathogenicity (e.g., patients affected with disease, impact on allele frequency in the population, impact on how well the reference allele is conserved across evolution). Node 252 represents predicted pathogenicity of a variant.

[0074] FIG. 2F illustrates another example of a probabilistic graphical model 280 similar to model 250 but extended to include a second central node 288 in addition to central node 294. The second central node 288 represents drug responsiveness of the variant in a somatic setting to anti -cancer drug Vemurafenib. Extending the PGM to include multiple central nodes improves the utility of, for example, somatic / tumor testing for drug responsiveness by enabling the leveraging of data / knowledge from germline testing. Currently germline variant interpretation (assessing pathogenicity of a variant) and somatic (tumor) variant interpretation (diagnosis, prognosis and disease management information such as expected drug -responsiveness) have very little overlap. That is, those evaluations are done independently and often times, data relevant for one variant is not viewed as relevant for the other variant, and therefore not used at all. Creating a connected DAG as shown and described enables data relevant for one variant (e.g., co-segregation of the variant with disease in a germlme setting) to be used to inform the other variant (e.g., drug responsiveness in tumor).

[0075] Nodes 284, 286, 290, 294 are upstream of node 288, and node 292 is downstream of node 288. In the example of FIG. 2F, the model for the protein effects variable is extended by node 288 because a node of the sub-model 286 has a causal relationship with the variable represented by the node 288.

[0076] FIG. 2G illustrates another example of a probabilistic graphical model 281. The model 281 includes a central pathogenicity node 291, a number of upstream (causal) nodes 283, a number of downstream (effect) nodes 285, upstream acyclic directed edges 289, and downstream acyclic directed edges 287. Each of the upstream nodes 283 is connected to the central pathogenicity node 291 by one of the acyclic directed edges 289 and each of the downstream nodes 285 is connected to the centralpathogenicity node 291 by one of the acyclic directed edges 287. For each of the acyclic directed edges, the direction of the arrow indicates the direction of causality.

[0077] In the example of FIG. 2G, each of the upstream nodes 283 and each of the downstream nodes 285 corresponds to a different machine learning model that produces predictive output relating to a particular type or category of pathogenicity evidence (e.g., a distribution or a point value). For example, each of the upstream nodes 283 and each of the downstream nodes 285 may correspond to a machine learning model of an evidence modeling platform such as platform 600 described with reference to FIG. 6. In FIG. 2G, CVM refers to a clinical variant model (containing patient clinical data modeled and evaluated through an EMP), PFM refers to a population frequency model (population frequency data modeled and / or evaluated through EMP), FIM refers to a model of evolutionary conservation, protein structure and protein stability data evaluated through an EMP, DMA refers to a model of published in vitro experimental study data modeled and / or evaluated through an EMP, DML refers to a model of internally performed in vitro experimental study data modeled and / or evaluated through EMP, PANGOLIN splice loss refers to a model of splice site loss predictions obtained from an external source such as a PANGOLIN model, and splice gain refers to a model of splice site gain predictions obtained from an external source such as a PANGOLIN model.

[0078] Collectively, the examples shown in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G illustrate a number of different configurations of a causal model for variant interpretation. In any of the models, any of the nodes can be sampled to obtain predictive information related to the sampled node.

[0079] The examples shown in FIG. 2A, FIG. 2B, FIG. 2C, FIG. 2D, FIG. 2E, FIG.2F, and FIG. 2G and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0080] FIG. 3 illustrates an example component-based process for creating a probabilistic model for variant interpretation, in accordance with some embodiments of the present disclosure. FIG. 3 illustrates examples of communications between various components of a computing system involved in executing a component-based process 300, including machine learning techniques that are used by the various components of the computing system to create a causal model for variant interpretation. The componentbased process 300 is performed by processing logic embodied in the various components of the computing system, which include hardware (e g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the componentbased process 300 is performed by one or more of the components of a computing system, including, in some embodiments, components or flows shown in FIG. 3 that may not be specifically shown in other figures and / or including, in some embodiments, components or flows shown in other figures that may not be specifically shown in FIG. 3. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and someprocesses can be performed in parallel. Additionally, at least one process can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0081] The component-based process 300 prepares one or more data sets, e.g., training data set 338, validation data set 342, for input to a causal model 328 via model training component 326 and model validation subsystem 330, respectively, using one or more machine learning techniques. Portions of the process 300 are performed by components of a modeling computing system (e.g., modeling system 1050 described with reference to FIG. 10), including a data pre-processing subsystem 308, a modeling subsystem 320, a model validation subsystem 330, and a model serving / inferencing subsystem 332. Data sources from which data is received during various portions of the process 300 include unlabeled data 302, labeled data 304, domain knowledge 306, feature manifest 318, model performance criteria 336, training data set 338, model validation criteria 340, and validation data set 342.

[0082] Unlabeled data 302 includes unlabeled phenotype and / or genotype data. For example, unlabeled data 302 includes population data extracted from gnomAD or a similar database and / or features extracted from test requisition forms using natural language processing, such as natural language descriptions of family history, indications, etc.

[0083] The unlabeled data 302 does not include associated pathogenicity labels or scores. Pathogenicity labels or scores (e.g., benign or pathogenic) can be obtained from labeled data 304. Labeled data 304 is a reference database that links variants with associated ground-truth pathogenicity labels or scores, such as ClinVar or an internally developed database curated by genetic scientists and / or other genetic experts. Examples of labeled data 304 include gene-specific labels, pooled labels from a collection of genes with similar biological properties (e.g., gene families, genes in the same pathway), labels obtained from a proprietary database (e.g., compiled based on historical data), labels generated from a proprietary database, manipulated (e.g., simulated recalculation of labels with data added or removed), labels obtained from (e.g., external) data sources such as ClinVar, with various sort or filter parameters (e.g., all of ClinVar, trusted ClinVar submitter data only, certain classification bin only, etc.), and labels for a subset or subsets of variant types (e.g., missense, splice site, etc.). Other examples of unlabeled data 302 and / or labeled data 304 include outputs from other models, empirical measurements, predicted variant effects, and public datasets.

[0084] The labeled data 304 can be joined or merged with corresponding unlabeled data 302 (e.g., using a common identifier or key value) to form training data sets 338, or separate training data sets can be formed for each of unlabeled data 302 and labeled data 304. In other words, a model training component 326 of modeling subsystem 320 can train causal model 328 on a first training data set 338 that contains only unlabeled data 302 and separately train causal model 328 on a second training data set that contains only labeled data 304, or model training component 326 can combine the unlabeled data 302 and labeled data 302 and train the causal model 328 on the combined training data set. Additional details and examples relating to training and validation are provided with reference to the modeling subsystem 320 and model validation subsystem 330.

[0085] Data pre-processing subsystem 308 includes a fetching component 310, a natural language processing (NLP) component 312, a filtering component 314, and a feature generation component 316.For example, data pre-processing subsystem 308 converts variant identifiers to a common format, identifies and fetches the right data when available, filters fetched data to a given scope (e.g., missense variants), and creates appropriate data structures (e.g., annotated tensors). In some embodiments, the data input to process 300 is already in a format that can be used directly by modeling subsystem 320. In these embodiments, the data pre-processing subsystem 308 may be omitted.

[0086] The fetching component 310 executes one or more processes by which raw data is retrieved from one or more data sources (e g., unlabeled data 302, labeled data 304, domain knowledge 306), using, for example, a query mechanism or search engine. The fetching component 310 evaluates unlabeled data 302 and labeled data 304 and selects portions of unlabeled data 302 and labeled data, 304 to be used to generate data sets that can be input to create training and / or validation data sets for causal model 328.

[0087] The NLP component 312 is used for natural language processing of raw input data, as needed. The NLP component 312 may be employed when the fetched data (e g., unlabeled data 302) is in the form of unstructured content such as natural language text. When the fetched data does not contain unstructured content, NLP component 312 may be omitted. The NLP component 312 applies one or more NLP techniques to unstructured data such as natural language text extracted from documents such as test requisition forms. For example, NLP component 312 uses entity recognition techniques to locate and identify, e.g., family history and / or indication information within a document and then extract the identified information.

[0088] Filtering component 314 filters the input data (e.g., unlabeled data 302, labeled data 304, domain knowledges 306) to create one or more data sets for processing by, e.g., feature generation component 314, modeling subsystem 320, and / or model validation subsystem 330. Filtering component 314 can filter raw data selectively to obtain data sets that are to be used to construct, train, or validate causal model 328. Filtering component 314 applies one or more filters to data 302, 304. The filters include criteria for determining whether certain of the data 302, 304 is to be included or excluded from the model training and / or validation processes. For example, data associated with variants that do not have a known association with any disease may be filtered by filtering component 314 and thereby excluded from training data set 338 and / or validation data set 342. As another example, raw data obtained from an external source such as gnomAD data may be filtered by filtering component 314 to extract only the data that is relevant to a specific variable, such as allele frequency. In some embodiments, the input data may already be filtered and thus filtering component 314 may be omitted.

[0089] Feature generation component 316 converts input data to a form that can be input to causal model 328. For example, feature generation component 316 uses a feature manifest 318 to convert the output resulting from the application of fetching component 310, NLP component 312, and / or filtering component 314 to the data 302, 304 to a format that can be input to and read and processed by causal model 328, such as tensors, vectors, embeddings, etc. Feature manifest 318 includes specifications for the types and configurations of features that can be input to causal model 328. In some embodiments, the input data may already be in the appropriate format and thus feature generation component 316 may be omitted.

[0090] Modeling subsystem 320 constructs and trains causal model 328. In some embodiments, modeling subsystem 320 receives as input the feature sets engineered and output by feature generation component 316. In other embodiments, modeling subsystem 320 receives as input formatted data from another system or component. Modeling subsystem 320 includes a data set creation component 322, a construction component 324, and a model training component 326.

[0091] Data set creation component 322 creates data sets for model training and validation. Data set creation component 322 divides the feature sets, e.g., as engineered and output by feature generation component 316, into training and validation (e.g., holdout) data sets, e.g., training data set 338 and validation data set 342. For example, in some embodiments, data set creation component 3 22 creates gene- or variant-specific training and validation data sets.

[0092] Model construction component 324 uses domain knowledge 306 to select or construct causal model 328. For example, model construction component 324 can query a library of causal models (e g., model library 334) to identify a causal model from the library that corresponds to the domain knowledge 306. As another example, model construction component 324 uses domain knowledge 306 to create a graphical representation of the causal model 328, including identifying nodes, determining an arrangement or topology of nodes, and creating edges (e.g., acyclic directed edges) between nodes. For instance, model construction component 324 can use domain knowledge 306 to identify types of pathogenicity evidence that are key determinants of pathogenicity, to determine whether the evidence is upstream (e.g., causal) with respect to pathogenicity or downstream (e.g., an effect) of pathogenicity, and to establish relationships between the nodes. Examples of causal models that can be constructed by model construction component 324 are described with reference to FIGS. 2A-2F.

[0093] Model training component 326 executes a model training process that applies a causal model 328 selected or constructed by model construction component 324 to one or more of the data sets created by the data set creation component 322. For example, model training component 326 iteratively applies the causal model to training data set 338 and adjusts one or more model parameters and / or feature weights until a comparison of the predicted model output generated by the causal model 328 by sampling the pathogenicity node of the causal model and the expected model output evidenced by the ground-truth labels obtained via labeled data 304 satisfies (e.g., meets or exceeds) model performance criteria 336. When the model performance criteria 336 are satisfied, modeling subsystem 320 ends the model training process and produces a trained causal model 328.

[0094] Model validation subsystem 330 applies a model validation process to the trained causal model 328 produced by modeling subsystem 320. Model validation subsystem 330 applies the trained causal model 328 to validation data set 342 to determine whether model validation criteria 340 are satisfied (e.g., met or exceeded). If the trained causal model 328 is successfully validated by model validation subsystem 330, the validated causal model 328 is provided to model serving / inferencing subsystem 332 for inferencing, e.g., to generate pathogenicity predictions, probability of pathogenicity, or estimates for novel (i.e., previously unseen) variants. Via model serving / inferencing subsystem 332, the pathogenicity node or any other node of the causal model 328 can be sampled to obtain predictive data. Alternatively or in addition, the predictive data output by the validated causal model 328 can be stored for future use (e.g.,for access or lookup by one or more downstream processes, systems, or services). In some embodiments, the predictive data can be used to update domain knowledge 306.

[0095] The examples shown in FIG. 3 and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0096] FIG. 4 illustrates an example process for training and validating a probabilistic model for variant interpretation, in accordance with some embodiments of the present disclosure.

[0097] In FIG. 4, a model training and / or validation process 400 applies causal machine learning model 404 to an input data set 402 using machine learning. The input data set 402 can be a training data set 338 or a validation data set 342 as described with reference to FIG. 3.

[0098] In response to the input data set 402, the causal machine learning model 404 generates model output 406, e.g., by sampling a pathogenicity node of the causal machine learning model 404. Based on an evaluation of the model output 406 at sub-process 412 using model performance or validation criteria, one or more model parameters 410, node weights and / or edge weights 408 of the causal machine learning model 404 may be adjusted at sub-process 414 and another iteration of the process 400 may be initiated. On each iteration, a decision sub-process 414 determines, based on the evaluation performed by subprocess 412, whether to continue iterating the process 400 or proceed with model serving / inferencing at sub-process 416.

[0099] In some examples, sub-process 412 includes evaluating model performance using a segregation performance technique (e.g., area under receiver operating curve or AUROC) to determine how well variants that are known to be pathogenic or known to be benign are able to be separated (e g., predicted accurately by the model). In some examples, sub-process 412 uses calibration metrics to determine the accuracy of the PoP calculations output by the model at different prediction ranges. In some examples, sub-process 412 valuates model performance using one or more of a test set reclassification rate, a VUS reclassification rate, a concordance with other classifications (e.g., Sherloc classifications), model accuracy, or error analysis to identify stages in the model at which error(s) occur and the reasons for the error(s). In some examples, based on the output of sub-process 412, sub-process 414 adjusts the input data by adding or removing input features or changing the method used to label the input data so as to revise the set of data considered to be trusted or ground-truth labeled data. In some examples, subprocess 414 adjusts one or more model parameters 410 based on the output of sub-process 412.

[0100] Other examples of criteria that can be used to evaluate the causal machine learning model 404 include how well the model can handle missing data, how easy it is to explain how the model reached a conclusion for a particular variant, the ease of changing or updating by simulating input data (for example, adding a splice prediction for a particular variant to see how it would change the prediction), how well the model incorporates domain knowledge such as scientific or clinical expertise, how well parameters of the model correspond to scientific concepts, whether the model can represent differences between pooled groups of variants, the fraction of variants the model is capable of serving with actionable predictions, the difference between the model prediction of a label (e.g., pathogenic / benign) in the control set and arandom set of VUS variants, concordance with a variant classification or variant interpretation framework, and the ability of the model to update in real time and make on demand predictions.

[0101] The examples shown in FIG. 4 and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0102] FIG. 5 A illustrates an example of a variant interpretation or variant classification system, in accordance with some embodiments of the present disclosure.

[0103] In FIG. 5A, a system 500 includes a data layer 502, a model layer 512, an application layer 514, and an infrastructure layer 516. Each or any of the layers 502, 512, 514, 516 can be implemented as one or more software-based components over one or more computing devices.

[0104] Data layer 502 includes a number of different datasets 536, 538, 540, 542. The datasets 536, 538, 540, 542 can include, for example, model training data and / or model validation data, phenotype data, genotype data, domain knowledge, model output, etc. The datasets 536, 538, 540, 542 each are supported by one or more data ingestion components including online data ingestion component 528, offline data ingestion component 530, data ingestion utilities component 532, and label methods 534. The data ingestion components 528, 530 and data ingestion utilities component 532 are configured according to the requirements of respective data sets and / or system requirements. For example, the system 500 connects to sources of updates to the datasets 536, 538, 540, 542 via one or more components of the data layer 502. As an example, the system 500 is capable of connecting to one or more online data stores and / or offline data stores.

[0105] In some examples, data ingestion components 528, 530, 532 enable the model layer 512 to systematically access various types of data used in creating, training, validating, and using the models. Online data ingestion component 528 can be used to ingest data into data sets when the data is updated frequently, such as new patient phenotype data, which is updated as new patients are referred for testing. Online data ingestion component 528 is capable of connecting the system 500 to updated data (e.g., online data store such as an online database). Offline data ingesting component 530 can be used to ingest static datatype that can be stored in e.g., offline or nearline data stores (e.g., gnomAD population allele frequency data or Pangolin splice effect predictions).

[0106] Data ingestion utilities component 532 can include one or more tools that can be used to schedule or orchestrate data ingestion processes; for example, the timing, frequency, and methods for selecting, invoking or calling one or both of the online and / or offline data ingestion components 528, 530. Label methods 534 includes methods (e.g., computer programs) that can apply labels to items in data sets, e.g., based on rules or classification algorithms. Label methods 534 can include one or more tools that can be used to ingest label data from various data sources (e.g., proprietary variant databases, external data sources such as ClinVar, etc.). Label methods 534 can include one or more processes for selectively assigning labels to data, e.g., determining which labels to apply to different types of data. For example, a label method 534 may determine not to use label data that has not been verified or determine to use only label data from certain specific, trusted sources.

[0107] The system 500 is flexible in that the labels that can be obtained and used by label methods 534 can be adapted to the requirements of a particular design or implementation. In some examples, the labels Pathogenic (P) and Benign (B) are used (e.g., for binary classification via supervised machine learning). In another implementation, the labels Pathogenic, Variant of Uncertain Significance (VUS) and Benign are used (e.g., in multi-class machine learning-based classification). In some embodiments, P and LP (Likely Pathogenic) data are grouped in the Pathogenic label and / or LB (Likely Benign) and B data are grouped together in the Benign label.

[0108] Model layer 512 includes a model abstractions layer 510 and a model implementations layer 504. The model abstractions layer 510 is configured to delegate model interactions to the underlying models of the model implementations layer 504. Model abstractions layer 510 is capable of determining the types of interactions that are available for a given model type. For example, in a response to a request (e g., a sampling request), a Bayesian model may deliver a distribution whereas another model type may only deliver a point estimate. Model abstractions layer 510 can include a set of common methods that can be used across or by any of the models in the model implementations layer 504 as well as logic for unique interactions.

[0109] The model implementations layer 504 includes a number N of different models 506, 508, where N is a positive integer (not limited to 2, although two models are shown for illustration). For example, model 1 506 can be a Bayesian inference model while model 2 508 can be a deep learning model or regression model. As another example, model 1 506 can be a first configuration of a Bayesian inference model while model 2 508 can be a second configuration of a Bayesian inference model different from the model 1 506. The models 506, 508 can have the same architecture while trained on different training data, or different model architectures. For example, the model implementations layer 504 can include any one or more of the models described with reference to any of FIG. IB, 2A-2F or FIG. 6. The models included in model implementations layer 504 are models that have been trained and validated, for example using the platform 600 described with reference to FIG. 6.

[0110] The application layer 514 includes interaction tools (API, UI) and tools for querying and monitoring the models in the model implementations layer 504. For example, application layer 514 includes a model serving component 518, one or more application programming interfaces (APIs) 520, an interactive component 522, one or more cloud platform components 524, and one or more monitoring components 526. The model serving component 518 enables access to the models in the model implementations layer 504, e.g., for obtaining samples or predictions from the models. The APIs 520, interactive component 522, cloud platform components 524, and monitoring components 526 support the model serving component 518 depending upon the type of interaction being received by the application layer 514 (e.g., the type of connection, query, communication, etc.).

[0111] In some examples, model serving component 518 provides model serving functions that enable online access and querying capabilities. For example, model serving component 518 can enable a clinical reporting software application to access pathogenicity prediction output by one or more models of the model layer 512, which can be imported into the clinical reporting software, e.g., to provide an interpretation for a variant observed during genetic testing.

[0112] It should be noted that the model serving component 518 is capable of serving predictive output of the one or more models of the model layer 512, and that the predictive output of such models (including the causal models described herein) can include pathogenicity predictions and / or other types of predictions. For example, the causal models can be sampled at a molecular cellular function node and the predictive model output served at that node can be used to determine whether a molecular cellular function is disrupted by a variant. As another example, model output can be segregated at or grouped for a particular family or group of individuals who are genetically related, which could be useful to a patient to determine recommended next steps. In another example, model output can be sampled at a molecular function node to obtain predictive data about molecular function that could be useful for designing drugs.

[0113] In an example of a causal model constructed so that a pathogenicity node is connected to both a first parent node and a second parent node such that the molecular function (first parent node) and transcript / protein expression nodes (second parent node) are causal inputs to the pathogenicity node, then, if it is known that a variant is pathogenic and also known that the molecular function is not disrupted, then in response to input of these parameters, the model output likely can be used to infer that the variant is pathogenic via a means other than molecular function, which, based on the model architecture, may indicate that the likely cause of pathogenicity is transcript / protein expression and not molecular function. In this way, sampling data from other nodes of the causal model can identify potential alternative sources of causality. These and other examples demonstrate that the potential applications for the described causal models are not limited to the clinical context but also can be useful for, e.g., pharmaceutical research, drug discovery selection, and / or other applications.

[0114] The topology and architecture of the causal model (e.g., the DAG) as described does not have any known limits, because its output is reflective of the level of uncertainty in the model design. Causal models as described herein incorporate uncertainty and can also inform evidence, data sources or experimentation that may be useful to pursue to help address the uncertainty.

[0115] The one or more APIs 520 provide one or more application programming interfaces, which can be used to enable access by a calling program to model input and / or output data by the model layer 512. For example, data can be obtained from the model layer 512 via one or more APIs 520 and used for data science experiments and analyses.

[0116] The interactive component 522 can provide, for example, a web interface through which scientists and / or clinicians, for example, can interact with one or more models of the model layer 512. For example, the interactive component 522 can generate and present simulations in response to requests received via the interactive component 522 (e.g., from a user device), where the simulations can simulate impact of new data on model predictions (e.g., "what would happen to the prediction if we had another patient observation?").

[0117] The monitoring components 526 provide monitoring services that can monitor the performance of one or more models of the model layer 512 and detect changes in the model performance over time, e.g., for quality control purposes.

[0118] The cloud platform components 524 can provide access control and security services, such as protected cloud access to model outputs, e.g., for research purposes or clinical reporting applications.

[0119] The infrastructure layer 516 supports the other layers 502, 512, 514 with tools and utilities for computations (e.g., compute component 544), data storing (e.g., storage 546), event logging (e.g., logging 548), communications (e.g., docker 550, CI / CD 552), and security (e.g., security 554).

[0120] For example, the infrastructure layer 516 can include the backend computational components and associated distribution tools to support the application layer 514. As another example, the infrastructure layer 516 can support an evidence modeling platform (EMP) in addition to the described causal model-based variant classification or variant interpretation system. For instance, the infrastructure layer 516 can but is not required to facilitate the supply of information from the EMP for input to the causal modeling system. The examples shown in FIG. 5A, and the accompanying description above, are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0121] FIG. 5B illustrates an example of a component-based process for variant classification or variant interpretation, in accordance with some embodiments of the present disclosure.

[0122] In FIG. 5B, a computing system includes a number of datasets 562, a data preparation subsystem 564, a feature store 572, a label store 574, model experiments 576, a model training component 578, a model registry 582, a model serving component 584, online predictions 586, a batch prediction / inferencing component 588, and a batch prediction store 590.

[0123] The datasets 562 include evidentiary data that may be relevant to variant classification or interpretation, such as various types of pathogenicity evidence. For a given variant, the evidentiary data may or may not include a label (e.g., a pathogenicity label such as pathogenic, benign or VUS).

[0124] The datasets 562 can include data stores and / or searchable databases that store raw data and / or output of machine learning-based models, such as output of one or more machine learning models. For example, the one or more machine learning models may be contained in an evidence modeling platform. The datasets 562 can include internally generated data and / or data obtained from one or more external sources. For instance, the datasets 562 can include one or more of: a CVM dataset (containing patient clinical data modeled and evaluated through an EMP), a PFM dataset (population frequency data modeled and / or evaluated through EMP), a FIM dataset (evolutionary conservation, protein structure and protein stability data evaluated through an EMP) a DMA dataset (published in vitro experimental study data modeled and / or evaluated through EMP), a DML (internally performed in vitro experimental study data modeled and / or evaluated through EMP), splice site loss predictions obtained from an external source such as a PANGOLIN model, splice site gain predictions obtained from an external source such as a PANGOLIN model, allele frequency (AF) data, or predictions of nonsense mediated decay, human- curated data based on an application of a variant classification framework (e.g., Sherloc framework).

[0125] The data preparation subsystem 564 includes a feature ingestion and pre-processing component 566, a label pipeline 568, and a data versioning component 570. The feature ingestion and preprocessing component 566 ingests and pre-processes data obtained from one or more of the datasets 562 individually or in combination, which may or may not include labels. For example, data can be fetched and preprocessed as described with reference to FIG. 1A, FIG. 3 and / or FIG. 5A. The feature ingestion and pre-processing component 566 can create combined or linked sets of features extracted from differenttypes of evidence obtained from multiple different datasets 562. The features extracted from the datasets 562 can be annotated with a common identifier, such as a variant identifier. Alternatively or in addition, the features extracted from the datasets 562 can be joined, for example, using the common identifier, to create a combined set of features that can be input to a machine learning model (e.g., a Bayesian causal model as described herein). The feature ingestion and pre-processing component 566 outputs feature sets to feature store 572 for use by model training component 578.

[0126] The label pipeline 568 determines whether labeled data exists for a given type of evidentiary data for a variant, and the extent to which the labels, if available, are reliable (e.g., obtained from a trusted source or independently verified). The label pipeline 568 may assign a VUS label to variants that do not otherwise have a label. The label pipeline 568 aligns the label data with the respective variant identifiers and outputs the label data to label store 574 for use by model training component 578.

[0127] The data versioning component 570 applies timestamp data to the feature sets output by the feature ingestion and pre-processing component 566 and the label data output by the label pipeline 568 in order to synchronize and manage different versions of the feature sets and corresponding label data, if any. The label data and the feature sets are stored separately in the example of FIG. 5B (e.g., the feature sets are stored in feature store 572 while the label data is stored in label store 574). In this example, the separate storage of these data can facilitate unsupervised training of one or more portions of a machine learning model while also enabling supervised training of other portions of the machine learning model, by model training component 578. This example is illustrative only and the label data and the feature sets may be stored together without departing from the scope of this disclosure.

[0128] Once prepared, the labels and feature sets are used fortraining / inference. The model training component 578 obtains training data from one or more of feature store 572 or label store 574 and applies the machine learning model to the training data. In the context of the Bayesian causal models described herein, inference and / or training may be used to refer to the process of fitting the Bayesian model to the training data before the Bayesian model is used for prediction. As noted above, various combinations of labeled and / or unlabeled training data may be used to develop the Bayesian model. In this context, the model training component 578 treats labels as just another variable that may or may not be observed for a variant.

[0129] Fitting the Bayesian model to the training data includes the model training component 578 iteratively (e.g., in a closed loop) applying model experiments 576 to the Bayesian model to test how the model has responded to the training data (e.g., updates to weight values, parameter values, etc.), until applicable performance criteria are met at decision block 580. The model experiments 576 and applicable performance criteria can include, for example, any one or more of the experiments described with reference to FIG. 4. The decision block 580 represents functionality that is included in the model training component 578 but is shown separately in FIG. 5B for purposes of illustration. As new training data becomes available, the Bayesian model is fit to the new data, thereby creating a new version of the model. In this way, the Bayesian model naturally accommodates sparse or missing data and adapts as more complete data becomes available.

[0130] Because the model training component 578 can fit the Bayesian model to training sets of variant data independently of labels, any or all of the variant-related data, with or without labels, can be used for inference (e.g., even if a label has not been observed). Also, the feature sets can be arbitrary in the sense that some features may be observed while other features may be unobserved, for a given variant, in a given version of a feature set. The Bayesian model accommodates such arbitrary feature sets in that the observed features can be used for inference notwithstanding the unobserved features, with the caveat that the unobserved features have not been observed.

[0131] As an example, if a request or query identifies a variant and only one feature, the Bayesian model can still provide some information about the distribution of that feature, with a level of uncertainty that reflects the fact that only one feature is known. As another example, even if the Bayesian model does not have a label coupled with the feature set for a variant, the model can still provide information about the feature distribution for the known feature As noted above, any node of the graphical representation of the Bayesian model can be sampled and the output from the sampled node can provide predictive information with respect to the corresponding variable.

[0132] The model training component 578 continues applying training data and model experiments 576 to the model until decision block 580 determines that the applicable performance criteria have been met. In response to the decision block 580 determining that the applicable performance criteria are met, the trained model is registered and stored in model registry 582 and made available to batch prediction / inferencing component 588. The model registry 582 can store one or more different versions of Bayesian models. Batch prediction / inferencing component 588 uses the trained model to generate batch predictions and stores the batch predictions in batch prediction store 590. Batch prediction store 590 can be queried for predictive data produced by the trained model. A batch prediction is a prediction provided by the trained model over a large number of variants that have known labels.

[0133] The model registry 582 makes the trained model accessible to model serving component 584, e.g., via an API. The model serving component 584 samples or queries the model registry 582 in response to request or queries received via, e.g., one or more user devices over a network. In response to the requests or queries, the model serving component 584 samples the trained Bayesian model and uses the sampled model output to produce online predictions 586, e.g., for presentation via one or more user devices. An example of a scenario involving an online prediction 586 is a clinical application in which a person’s test results identify a new variant that has not been previously observed. In this scenario, the model serving component 584 can query the Bayesian model in real time to obtain a prediction and return the prediction in response to the query.

[0134] The examples shown in FIG. 5B and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0135] FIG. 6 illustrates an example of an evidence -based variant classification or variant interpretation platform, in accordance with some embodiments of the present disclosure.

[0136] In FIG. 6, an evidence -based modeling platform 600 provides a processing pipeline 602. The pipeline 602 is used to generate, evaluate, and integrate various sources of evidence for clinical variantclassification or interpretation using different sources of input data. A wide range of machine learning (ML) models / algorithms are used in conjunction with domain knowledge (e.g., clinical expertise) to generate and output interpretations of gene variants. Different models are queried to obtain information about different attributes of a gene variant that may or may not contribute to its pathogenicity. For example, attributes such as allele frequency, functional data, and stability of a resulting protein structure can be modeled using different modeling approaches, and the pipeline 602 can select and obtain evidence from among these different models. The platform 600 provides a framework for evaluating and using diverse evidence to resolve variants otherwise classified as having uncertain significance (VUS) with respect to a health condition. For example, the configuration of the pipeline 602 can enable the platform 600 to improve variant classification or interpretation for a wide range of hereditary conditions, leading to a reduction in VUS, particularly in ancestry groups that are underrepresented in genomic studies and databases.

[0137] In one study, the performance of the platform 600 was measured against established approaches to variant classification, and its impact was evaluated for resolving VUSs across various clinical areas and diverse ancestry groups. A hypothesis was that utilizing the platform 600 for evaluating and integrating novel evidence at scale can accelerate the rate of VUS resolution, especially for individuals from ancestry groups under-represented in genomic studies, therefore leading to increased equity in variant classification or interpretation. De-identified patient data were approved for analysis under Western Independent Review Board protocol number, without the need for additional individual informed consent. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines for cohort studies. Next-generation sequencing data and clinical and demographic information were obtained for individuals referred to a laboratory, where clinician-ordered germline testing of single genes or multi -gene panels was performed. Individuals who requested data deletion or opted out of data use were excluded from the study.

[0138] Variants were classified as Benign, Likely Benign, VUS, Likely Pathogenic, or Pathogenic using the Sherloc framework, a validated system based on guidelines from the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Sherloc classifies variants using a semi -quantitative point-based rubric to collate the separate contributions of heterogeneous evidence types into a classification decision.

[0139] In the study, 27,477 / 496,209 novel variants were used to validate the platform 600, as they received evidence from at least one model and had obtained a non-VUS classification independently of that evidence. Across 8,517 unique pathogenic variants and 14,960 unique benign variants, the combination of models provided in the pipeline 602 achieved 98% Negative Predictive Value and 90% Positive Predictive Value. Overall, 22.3% of individuals participating in the study had at least one variant classified based on the input from one or more of the models of the platform 600 (e.g., 21.6% from VUS to B / LB, 1.4% to P / LP). Impact varied by clinical area, being the greatest among individuals tested for immunology (60.5%) and neurology (48.5%). Impact also varied by ancestry, with the greatest impact seen among individuals of Asian (33.7%) and Black (28.7%) ancestries compared to White individuals (18.5%, two-sided t-test p<le- 10). Results from this study demonstrate that a systematic approach tointegrating machine learning models into variant classification can yield highly accurate interpretations, enabling a significant proportion of individuals with a VUS to receive a more definitive result. The results reflect the fact that the models in the platform 600 are largely informed by biologically-grounded hypotheses, not biased by the ancestral origin of evidence, leading to more equity in variant classification or interpretation.

[0140] The platform 600 provides an analysis infrastructure for evaluating and weighting diverse evidence for use in variant interpretation at scale. Based on machine learning from labeled examples (e.g., known pathogenic and benign variants), the platform 600 can be used for both gene-specific and genomewide analyses, and is designed to incorporate novel evidence through iterative updates overtime. A function of the platform 600 is to evaluate whether a type of evidence meets a quality threshold for inclusion in a variant interpretation. In one embodiment, if the evidence meets or exceeds the quality threshold, the platform 600 assigns a standardized weight to its value for use in an ACMG- based clinical variant classification or interpretation. In contrast to other approaches that may combine heterogeneous data into a meta-model, the platform 600 is comprised of a plurality of different evidence models that each independently evaluate a distinct category of evidence and determine whether that evidence can reliably inform variant classification or interpretation.

[0141] In the illustrated example of the platform 600, the platform 600 provides several functions: First, the platform 600 is capable of building and training evidence models (but this capability is not required for all models in the platform 600). The models developed using the platform 600 include models that are generated by supervised machine-learning and trained using gene-specific labeled data (i.e., known benign and pathogenic variants) that can be based on a diverse range of evidence types. Examples of the models that can be included in the platform 600 include but are not limited to: CVM (patient clinical data evaluated through EMP), PFM (population frequency data evaluated through EMP), DMA (published in vitro experimental study data evaluated through EMP), DML (internally performed in vitro experimental study data evaluated through EMP), Splice site loss predictions (from PANGOLIN), Splice site gain predictions (from PANGOLIN), Sherloc EV0016 (prediction of nonsense mediated decay, human-curated data based on the application of Sherloc), and / or FIM (evolutionary conservation, protein structure and protein stability data evaluated through an EMP).

[0142] Second, the platform 600 evaluates models (this capability is applied to all models included in the platform 600). Because it is a flexible platform, the platform 600 can evaluate both models that have been built using the platform 600 as well as a multitude of external, unsupervised and / or genome-wide algorithms and models. For example, the platform 600 evaluates whether a particular dataset or model can reliably distinguish benign from pathogenic variants. To perform such evaluations, the platform 600 uses a set of held out labeled data (e.g., a portion of training data that has not been used to train the model) to test how well a given model separates known benign and pathogenic variants. The platform 600 includes evidence from models that meet or exceed a defined quality threshold in variant interpretation.

[0143] Third, the platform 600 can be used to predict the pathogenicity of novel variants and / or previously classified variants. The platform uses model evidence that meets or exceeds the quality threshold to predict the pathogenicity of variants with otherwise uncertain clinical significance .

[0144] Fourth, the platform 600 standardizes pathogenicity predictions across the various evidencebased models. The platform 600 calibrates (e.g., adjusts weight values) predictions for each variant using empirical data, such that the strength (or confidence) of the prediction can be expressed in attribute points (e.g., 2P, IB). This score is determined by how cleanly the model separates known benign and pathogenic variants. In other words, evidence is weighted according to its model’s predictive performance (i.e., PPV and NPV) for known variants. This evidence -based weighting allows the platform 600 to standardize diverse data and models (e.g., population allele frequency, evolutionary conservation, physicochemical molecular properties, etc.) to a common framework that can be used for variant classification or interpretation. The platform 600 evaluates and standardizes diverse models and datatypes. In one embodiment, the output produced by the platform 600 (e.g., qualified, weighted evidence) can be input to an ACMG-based clinical variant interpretation process that can result in a pathogenicity / benignity score. In another embodiment, the output produced by the platform 600 (e.g., qualified, weighted evidence) can be input to a probabilistic or causal (e.g., Bayesian) clinical variant interpretation process.

[0145] The platform 600 is a flexible platform designed to address the specific challenges of evidence-driven variant interpretation in the era of big data. By using an adaptable framework based on supervised learning from labeled data, the platform 600 can estimate the predictive value of many different types of data, and thereby bring disparate forms of evidence into a common quantitative system for use in variant interpretation.

[0146] Prior to model-building or evaluation, labels can be calculated at the genomic, pre-mRNA, mRNA or protein level, depending on the nature of the evidence. Similarly, when building models, the platform 600 can use gene-specific (most common) or genome-wide data, as biologically appropriate. Because gene products have unique features and properties, many models do not generalize well across all genes, leading to inaccurate predictions. Rather, models often perform better (in terms of PPV and NPV) when trained on gene-specific data. For example, models of protein tertiary structure may be better predictors of pathogenicity for proteins that have a structural function in cells (e.g., ion channels) than for cytoplasmic enzymes, where only a small portion of the protein (e.g., the active site) may be required for function. Gene-specific model training and evaluation can enable the platform 600 to determine which types of data produce the best predictive models for each gene. The platform also enables training across genes when necessary, for example, when an insufficient number of labels are available for a given gene.

[0147] In contrast, other biological phenomena are not specific to individual genes (e.g., RNA splicing, allele frequency) and are thus modeled using genome-wide data. Importantly, model evaluation and standardization steps (using labels) can be applied both to data generated by the platform 600, and also to external algorithms and models such as evolutionary models of variant effect (EVEs).

[0148] Moreover, model evaluation and standardization can use different kinds of labeled data beyond known pathogenic and benign variants, providing further flexibility. For example, the platform 600 uses empirically measured splicing outcomes to assess the accuracy of in silico SpliceAI predictions. The flexible design of the platform 600 enables it to be applied to new evidence and new tools as they emerge.

[0149] Rather than being static, the platform 600 is designed to use iterations of machine learning to update its predictive models when new information becomes available. Models included in the platform600 are thus updated periodically and continue to be adjusted based on novel empirical evidence. This enables models included in the platform 600 to leverage up-to-date, clinically validated information.

[0150] The platform 600 includes a circularity filter 604. The circularity filter 604 filters model input data for circularity before use in future iterations. For example, prior predictions or variant classifications / interpretations that are generated based on evidence provided by the platform 600 may be filtered out by circularity filter 604 so that they are not used as training data in future iterations.

[0151] In contrast to some other tools, the platform 600 is capable of making predictions that are rooted in biological processes and the platform 600 maintains a transparent connection to those primary data. Independent evidence types are modeled, evaluated, and standardized separately rather than being mixed. Pathogenicity predictions stemming from each piece of evidence are each passed to a separate, ACMG-based variant interpretation process. Because of this, variant classification scientists using predictions based on the platform 600 can identify types of data that are key determinants of output scores, and evaluate the evidence provided by the platform 600 in the context of other available information. This contrasts with ensemble or meta-predictors, which typically combine different evidence types, thereby obscuring the information about which types of data drive their output scores and creating a risk of double dipping during variant interpretation.

[0152] Moreover, because many models of the platform 600 are grounded in fundamental biology (e.g., physicochemical properties), they are less susceptible to biases related to unequal representation of ancestry groups in genomic databases, leading to improved equity in variant interpretation.

[0153] Referring to FIG. 6, the pipeline 602 includes software -based components for model training, model evaluation, prediction using trained models, and standardization of model output.

[0154] The model training component uses data for variants with known pathogenicity to train the machine learning algorithms used by the models. For each model, the model training component applies an applicable machine learning algorithm to one or more training data sets until the output of the model meets or exceeds one or more performance criteria (e.g., the model converges in that the difference between the model output and the expected output is consistently below a threshold level of error tolerance). As part of the model training process, the model training component includes feature generation processes and label generation processes.

[0155] For feature generation, the model training component may convert input data from an assay, dataset, or method into a format that the model can accept as input (e.g., a tensor, an embedding, etc.), and / or the model may join data from different sources with a common unique identifier (ID), such as a variant ID. For label generation, the model training component may obtain variant pathogenicity label data (e.g., benign, pathogenic, etc.) from one or more data sources such as ClinVar databases, EVEs and / or others. The model training component may time-stamp the label data and associate it with the applicable variant ID. The label data may be stored separately from the model feature data. For example, some models may be trained only on unlabeled feature data, other models may be trained only on labeled data (in which case the feature data and label data may be joined by the variant ID), and / or still other models may be trained on both unlabeled and labeled data (e.g., a training data set could include data forvariants for which pathogenicity has not been determined as well as data for variants with known pathogenicity labels).

[0156] The model evaluation component evaluates the quality of the output of the models trained by the model training component and determines based on the evaluation whether to include a model in the platform 600. For instance, the model evaluation component applies a holdout data set (e.g., a portion of a training data set that was not used to train the model, which contains data for variants with known pathogenicity values) to a trained model and compares one or more model performance metrics (e.g., AUROC, area under receiver operating curve, or others) to a threshold value of the metric. The performance metrics can measure, for example, the ratio of true positives to false positives in the model output. If the ratio is below a threshold value (e.g., there are many more false positives than true positives in the model output), the model may be rejected. If the ratio meets or exceeds the performance threshold, the model may be accepted for inclusion in the platform 600.

[0157] The prediction component of the pipeline 602 uses only the highest performing models of the models evaluated by the model evaluation component for prediction. For example, when a novel variant is encountered (e g., a previously unlabeled variant), the prediction component applies one or more of the models that have been accepted into the platform 600 by the model evaluation component to the data associated with the novel variant, and generates the predictive output using those models, where the predictive output may include a predicted pathogenicity, e.g., a predicted effect of the novel variant on the likelihood of occurrence of a particular health condition. The prediction component may select a model to be used for the prediction based on the data that is available for the novel variant. For example, the prediction component may select one model if the available data includes data for one biological property and a different model if the available data includes data for a different biological property.

[0158] In some examples, the prediction component employs multiple different models of the platform and, for each model, bins the model output using predictive value thresholds which determine the relative weight given to the output of the respective model.

[0159] The standardization component converts the model output produced by the prediction component to standardized scores that can be used for variant interpretation. The standardized scores reflect the weight or credibility of the associated type of evidence in terms of predicting pathogenicity for a vanant. These standardized scores are output by the platform 600 and can be stored in, for example, an evidence compendium, which can be queried by a clinician, for example, for clinical variant interpretation. As variants are newly interpreted using model output provided by the platform 600, this new data can be used to iteratively update one or more of the models in the platform 600. For example, new data can be used to improve previously approved models, or to improve previously rejected models so that they can be approved for use within the platform, or to create new models for the platform 600. The circularity filter 604 can be used to determine whether or not to update the platform 600 with a given set of new data.

[0160] The platform 600 is adaptable to a wide range of training and validation data. For example, artificial intelligence (Al)-based in silico splicing predictions can be validated using empirical splicingdata. For example, if gene-specific EVE is not possible due to insufficient gene-level label data, a genome-wide gwEVE that relies on genome wide (gene -agnostic) label data can be used as an alternative.

[0161] In some studies, information on patient age, gender, and genetic ancestry was provided by ordering clinicians. For genetic ancestry, clinicians selected from preset genetically similar groups (referred to as racial / ethnic groups on test order forms): Ashkenazi Jewish, Asian, Black, French Canadian, Hispanic, Native American, Pacific Islander, Sephardic Jewish, White (non-Jewish and nonFrench Canadian for the purposes of this study), or Other. When Other was selected, a free-text response could be added. If free-text responses matched a preset group, individuals were included in that group. Individuals with more than one reported genetic ancestry were grouped as Multiple.

[0162] In some studies, clinical areas included cardiology, oncology (hereditary cancer), immunology, inherited metabolic disorders, and neurology. If genes were included in tests associated with two clinical areas (e g., NF1 in both hereditary cancer and neurology), both clinical areas were considered. Since some genes are involved in multiple hereditary diseases, patient attributes were analyzed based on the clinical area of the ordered panel (i.e., test clinical area) and variant attributes based on primary gene-disease relationships (i.e., gene clinical area).

[0163] To test the performance of the models in the platform 600, the concordance between the classification of a variant reached without evidence provided by the platform 600 was compared to predictions output by the platform 600 for that variant. Novel variants observed internally between May 1, 2022 and May 1, 2023 (N=496,204 unique variants) were selected fortesting. The analysis was restricted to the subset of variants which had received evidence from at least one model of the platform 600 and predictions for variants reaching classification without evidence provided by platform 600 (8,517 unique pathogenic variants and 14,960 unique benign variants) were compared with evidence provided by platform 600 to estimate a benign prospective performance (NPV) and pathogenic prospective performance (PPV).

[0164] Additionally, the accuracy of predictions output by the platform 600 were evaluated prospectively against externally validated variants. More specifically, for variants classified as (likely) pathogenic or benign using predictions produced by platform 600, the number and frequency of occurrence of those same variants being later classified as pathogenic or benign by external labs using independent new data was evaluated. In some evaluations, the analysis was limited to novel ClinVar variants reaching classifications between May 1, 2022 and May 1, 2023. Only variants reaching classification and submitted by third parties were considered.

[0165] The impact of the machine learning models integrated on variant classification was also assessed using an internal proprietary system. To do so, 10 subsets of 5,000 patients seen internally between May 1, 2022 and May 1, 2023 were sampled. Across those patients, all of the observed variants that received some type of evidence from the modeling platform were removed from the compendium of evidence forthose variants.

[0166] The calculated impact rate corresponds to the proportion of patients with at least one variant that would have been a VUS without the platform 600-produced predictions that subsequently reached a more definitive classification (e.g., Likely Pathogenic, Pathogenic, Likely Benign and Benign). Thisimpact was computed across the main clinical areas (Oncology, Metabolic, Immunology, Neurology and Cardiology) as well as across patient ethnicity (e.g., using self-reported data in test requisition forms, for these groups: White, Hispanic, Black / African-American, Asian, Ashkenazi Jewish).

[0167] Predictive modeling can help standardize and quantify the interpretation of assay data. By learning from known Pathogenic (P) or Benign variants (B) (effectively positive and negative controls), the predictive value of each data type can be determined and mapped to a single standard of “probability of pathogenicity,” which can be used in variant classification or interpretation. For example, when the input data (e g., high-throughput assay data) does not correlate well with known P / B variants, the model cannot be expected to be useful for variant classification or interpretation. Therefore, a performance evaluation step is used to ensure high quality predictions. This may limit the number of genes for which the platform 600 is able to make predictions, but ensures greater confidence in the results.

[0168] The output of the platform 600 can be just one component of a variant classification or variant interpretation system. The results of the predictive models in the platform 600 can be evaluated by, e.g., a clinical genomic scientist, a variant classification or variant interpretation system, alongside all other data that are relevant for variant classification or interpretation, such as patient phenotype and segregation data, to reach a final variant classification or otherwise interpret a variant.

[0169] In the platform 600, a diversity of models are utilized to address challenges specific to different genes and gene variants. The platform 600 implements a complex interplay of various models that are continuously updated, along with the addition of newly validated models, in the context of clinical expertise. Different models are used to look at various attributes of a gene variant that may or may not contribute to its pathogenicity, such as allele frequency, functional data, and stability of the resulting protein structure. Examples of such models that may be included in the platform 600 are described in FIG. 7. Each such model, leveraging one diverse datatype, enables key insights into the molecular mechanisms of disease to be uncovered, even while using data that may be biased from highly represented cohorts in the genetic testing landscape.

[0170] The platform 600 can generate pathogenicity scores for variants that can be used in many different variant classification or variant interpretation systems. As described above, the platform 600 includes a suite of evidence models that use different data sources and different modeling techniques to generate predictions for different types of variants (i.e., missense, splicing, etc.). In order to use these models in a consistent manner in a variant classification or variant interpretation system, the models may be required to pass performance criteria, and, if a semi-quantitative interpretation system like Sherloc is used, the models are thresholded based on performance to provide discrete evidence (i.e., attribute points). Similarly, to use the outputs of these models as an input in a probabilistic variant classification model, the models may be required to pass performance criteria.

[0171] Within the Sherloc interpretation system, the performance of the model over sets of variants is used to determine how many points the variants can receive on a gene-based level. During the evidence assignment step, metric thresholds (e.g., variants with a PPV>=0.95 can receive 2 points) are used to assign points to variants. Filters are applied over functional elements grouping of the model to ensure a minimum quality over relevant functional elements.

[0172] In a first step, bins are discovered that meet the predictive value thresholds specified on training data using validation predictions. In some examples, predictions produced by the platform 600 are converted into discrete point values. To achieve this, the variants are assigned to performance bins using PPV and NPV. Different performance bins have different point value associations based on the particular model type and the performance threshold. The transcripts are then filtered by performance. To do so, performance metrics are computed over the validation data predictions and return a list of transcripts passing one or more performance filters (e.g., AUROC >=0.7, AUROC>=0.8, AUROC>=0.9, AUROC>=0.95, AUROC>=0.99). If a transcript does not pass the filter, the point values of all variants within that transcript are set to 0.

[0173] When releasing models for use in a variant interpretation system, the best performing model for a given transcript is selected from among competing models (e.g., models from the same category) using the points assigned to each variant within that transcript.

[0174] In some examples a SpliceAI validation was performed. SpliceAI is a deep residual neural network that predicts whether each position in a pre-mRNA transcript is a splice donor, splice acceptor, or neither, using as input only the genomic sequence of the pre-mRNA transcript. SpliceAI was validated using variants that are in regions of the genes potentially impacted by splicing. To assess SpliceAI ability to accurately predict a change in splicing, internal data from oncology RNA assays were used, selecting variants associated (true positive set) or not (true negative set) with a splicing change in at least one of our patients. In total, a set of 1236 variants was considered as positive controls and 81 variants as negative controls. A SpliceAI prediction was considered positive if all scores for the given variant were above or equal to 0.2. Overall, SpliceAI yielded 91.34% sensitivity and 79% specificity across tested variants and was integrated into the platform 600.

[0175] The accuracy of pathogenicity predictions generated using the platform 600 was evaluated using a set of known pathogenic and benign variants, finding high concordance with independent variant classification methods. A retrospective cohort study was used to evaluate the platform’s VUS reclassification performance against external, independent laboratories, again finding high concordance.

[0176] The overall performance of a given model may differ from gene to gene . While a model type may perform well for a specific gene or group of genes based on the features of the gene(s), it may not perform well for other genes. A model that is able to discriminate well between pathogenic and benign variants receives higher weights for its predictions in the pipeline 602 for variant classification or variant interpretation. In contrast, a model that shows poor discrimination between pathogenic and benign variants in a specific gene, will not be used in the pipeline 602. This gene specific validation of each model is useful when considering the integration of a prediction from a model into the platform 600.

[0177] The examples shown in FIG. 6 and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0178] FIG. 7 is a table listing examples of models that may be included in an evidence-based variant interpretation platform, in accordance with some embodiments of the present disclosure. Any one or moreof the models described in FIG. 7 may be incorporated into the platform 600, described with reference to FIG. 6, subject to the model evaluation and other processes described with reference to FIG. 6.

[0179] The examples shown in FIG. 7 and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0180] FIG. 8A, FIG. 8B, FIG. 8C, and FIG. 8D illustrates examples of experimental results for a probabilistic variant interpretation model, in accordance with some embodiments of the present disclosure.

[0181] FIG. 8A illustrates an example of a histogram, which shows a distribution of pathogenicity predictions produced by a Bayesian model as described herein across variants within a test set (not used for training). FIG. 8B illustrates an example of a histogram, which shows a distribution of pathogenicity predictions produced by the same Bayesian model, across variants within a VUS set (variants of uncertain significance). For both of the examples FIG. 8A and FIG. 8B, the Bayesian model was trained on MMR genes. FIG. 8A shows that for the test set, known pathogenic variants tend to receive high probability of pathogenicity while known benign variants tend to receive low probability of pathogenicity. These results show that the model tends to separate well known pathogenic variants from known benign variants. FIG. 8B shows that for the VUS set, variants tend to be split between low and high probabilities of pathogenicity, suggesting that some of the variants could be considered benign or pathogenic, respectively.

[0182] FIG. 8C illustrates examples of calibration plots for Bayesian models configured as described herein, e.g., performance of various implementations of the Bayesian models. In FIG. 8C, the x-axis is the predicted probabilities of pathogenicity generated by the various models, while the y-axis is the fraction of variants at those predicted probabilities of pathogenicity that are known to be pathogenic. A perfectly calibrated system would show that 20% of variants with a score of 0.2 are pathogenic, 80% of variants with a score of 0.8 are actually pathogenic, etc. Each line in the plot represents a different model implementation. The line labeled “pyro_02.csv” indicates results for a probabilistic graphical model-based implementation.

[0183] FIG. 8D illustrates a pairwise analysis for an example probabilistic graphical model-based implementation. FIG. 8D includes a series of heatmaps that depict the learned pairwise-interactions between various features used in the probabilistic graphical model-based implementation. The areas of the heat map that correspond to higher values on the x and y axes (typically up and / or to the right on the map) indicate a greater probability of pathogenicity. The areas of the heat map that correspond to lower values on the x and / or y axes (typically lower and / or to the left on the map) indicate a lower probability of pathogenicity. For example, the DMA vs. FIM heatmap shows that the model learned that the probability of pathogenicity is driven primarily by DMA (e.g., that a high DMA score indicates a high probability of pathogenicity, regardless of the FIM score). In another example, the DMA vs. CVM heatmap shows that the model learned that both DMA and CVM scores contribute close to equally towards the final probability of pathogenicity (e.g., variants with high CVM score could still get low probability of pathogenicity if DMA score is low, or vice versa).

[0184] The examples shown in FIG. 8A, FIG. 8B, FIG. 8C, and FIG. 8D and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0185] FIG. 9A, FIG. 9B, FIG. 9C, FIG. 9D, FIG. 9E, FIG. 9F, and FIG. 9G illustrate methods, systems, apparatus, and / or non-transitory computer readable media configured for creating probabilistic models and using probabilistic models for variant interpretation, in accordance with some embodiments of the present disclosure.

[0186] FIG. 9A is a flowchart of an example method 900. Portions of the method 900 may be embodied in one or more non-transitory computer readable media, executed by one or more processors, and / or implemented in one or more components of a computing system. Illustrative examples of portions of the method 900 are further described throughout this disclosure, for instance with reference to FIGS. 1A, IB, 2A-2F, 3, 4, 5A, and 5B.

[0187] At operation 902, a processing device creates input data for a causal machine learning model using pathogenicity evidence data associated with a genetic variant and a health condition. The pathogenicity evidence data is obtained via a plurality of evidence sources. In some examples, for a category of the pathogenicity evidence data, the plurality of evidence sources includes alternative types of models for predicting pathogenicity given the category of the pathogenicity evidence data, and the processing device is to, for the category of the pathogenicity evidence data, use at least one evaluation criterion to create the input data using a model selected from among the alternative types of models. In some examples, portions of operation 902 are performed by a data pre-processing component of a computing system.

[0188] At operation 904, the processing device applies the causal machine learning model to the input data to produce a trained causal model. A graphical representation of the trained causal model includes nodes connected via acyclic directed edges. A first node of the nodes represents a pathogenicity evidence variable of the pathogenicity evidence variables related to the health condition. At least one second node of the nodes represents a cause of the pathogenicity evidence variable. At least one third node of the nodes represents an effect of the pathogenicity evidence variable. An acyclic directed edge represents a relationship between two of the nodes.

[0189] In some examples, in the graphical representation of the trained causal model, the at least one second node is upstream of the first node and the at least one third node is downstream of the first node and the at least one second node. In some examples, in the graphical representation of the trained causal model, the at least one second node includes at least one variable that represents data pertaining to at least one biological property that causes pathogenicity of the genetic variant with respect to the health condition. In some examples, portions of operation 904 are performed by a model training component of a computing system. In some examples, in the graphical representation of the trained causal model, the at least one third node includes at least one variable that represents data pertaining to at least one biological property that is an effect of pathogenicity of the genetic variant with respect to the health condition. In some examples, the graphical representation of the trained causal model includes at least one of: a genelevel variable, a gene-level constraint, or a variant-level constraint.

[0190] In some examples, the at least one vanable includes at least one of: a vanable pertaining to an impact of the genetic variant on protein function, a variable pertaining to an impact of the genetic variant on protein stability, a variable pertaining to an impact of the genetic variant on mRNA processing, a variable pertaining to the impact of the genetic variant on nonsense mediated decay, a variable pertaining to an impact of the genetic variant on mRNA, a variable pertaining to an impact of the genetic variant on protein expression, a variable pertaining to a biological criticality of disrupted DNA, a variable pertaining to a biological criticality of a disrupted RNA nucleotide, or a variable pertaining to a biological criticality of a disrupted protein amino acid residue. In some examples, the at least one variable includes at least one of: a variable pertaining to an impact of pathogenicity on an individual patient phenotype, a variable pertaining to an impact of pathogenicity on a population allele frequency, a variable pertaining to an impact of pathogenicity on a select cohort allele frequency, or a variable pertaining to an impact of pathogenicity on familial genotype -phenotype co-segregation patterns.

[0191] At operation 906, the processing device, in response to a request, outputs predictive data sampled from the trained causal model. In some examples, the processing device uses the trained causal model to generate and output a prediction as to whether the genetic variant is benign or pathogenic with respect to the health condition. In some examples, the processing device provides the prediction as to whether the genetic variant is benign or pathogenic to a clinician for use in formulating a diagnosis of a patient. In some examples, the processing device stores the prediction for retrieval via at least one query.

[0192] In some examples, the processing device applies at least one performance criterion to the trained causal model, e.g., via a model evaluation component of a computing system. In some examples, the processing device is to sample data from any node of the trained causal model, and a node sampled by the prediction component corresponds to an unobserved variable with respect to the genetic variant and the health condition. In some examples, portions of operation 906 are performed by a prediction component of a computing system.

[0193] FIG. 9B is a flowchart of an example method 910. The method 910 may be used to create a training data set for a causal machine learning model. Portions of the method 910 may be embodied in one or more non-transitory computer readable media, executed by one or more processors, and / or implemented in one or more components of a computing system. Illustrative examples of portions of the method 900 are further described throughout this disclosure, for instance with reference to FIGS. 1A and 3.

[0194] At operation 912, a processing device applies a natural language processing (NLP) model to clinical phenotype data for a genetic variant. In some examples, the processing device uses the NLP model to extract the clinical phenotype data from at least one field of a test requisition form, where the at least one field includes at least one of an indication field or a family history field. In some examples, the processing device obtains data relating to a genetic variant via a plurality of sources, filters the data using at least one filtering criterion, converts the filtered data to a standardized format, and includes the data in the standardized format in the training data set. In some examples, the data includes at least one of: output of at least one machine learning model, empirical measurements, predicted variant effects, or public datasets. In some examples, the standardized format includes a data structure usable as input to the causalmachine learning model. In some examples, the processing device splits the data into at least one training data set to train the causal machine learning model and at least one holdout data set to validate the trained causal machine learning model.

[0195] At operation 914, the processing device, using the NLP model, identifies features of the clinical phenotype data that are predictive of a molecular diagnosis of a health condition with respect to the genetic variant. At operation 916, for each subject in a population of subjects, the processing device uses the identified features and demographic information to compute a first score including a probability of the subject being affected with the health condition. At operation 918, for the population of subjects, the processing device includes the first scores in the training data set. In some examples, the processing device includes labeled pathogenicity data in the training data set.

[0196] FIG. 9C is a flowchart of an example method 920. The method 920 may be used to train a causal machine learning model for variant interpretation. Portions of the method 920 may be embodied in one or more non-transitory computer readable media, executed by one or more processors, and / or implemented in one or more components of a computing system. Illustrative examples of portions of the method 900 are further described throughout this disclosure, for instance with reference to FIGS. 1A, IB, 2A-2F, 3, 4, 5 A, and 5B.

[0197] At operation 922, a processing device creates model input data using pathogenicity evidence data for a variant. At operation 924, the processing device applies the causal machine learning model to the model input data, where a graphical representation of the causal machine learning model includes nodes connected via acyclic directed edges, a first node represents a pathogenicity variable, at least one second node represents a cause of pathogenicity variable, at least one third node represents an effect of pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes. At operation 926, the processing device evaluates data sampled from the causal machine learning model iteratively until at least one performance metric exceeds at least one threshold performance criterion.

[0198] In some examples, the processing device is used to construct the causal machine learning model, for example by linking the at least one second node to the first node via at least one upstream directed acyclic edge; and linking the at least one third node to the first node via at least one downstream directed acyclic edge. In some examples, the processing device is used to construct the causal machine learning model using domain knowledge to at least one of: identify at least one cause of the pathogenicity variable and assign the at least one cause of the pathogenicity variable to the at least one second node; identify at least one effect of the pathogenicity variable and assign the at least one effect of the pathogenicity variable to the at least one third node; link the at least one second node to the first node via at least one upstream acyclic directed edge; or link the at least one third node to the first node via at least one downstream acyclic directed edge. In some examples, the method 920 includes constructing the causal machine learning model by fitting a hierarchical Bayesian inference model to the model input data, where the model input data includes phenotype data and variant pathogenicity data.

[0199] In some examples, the method 920 includes obtaining the model input data via a plurality of sources; and using domain knowledge to filter the model input data, where the domain knowledgeincludes types of relationships between variables, the variables include probability distributions, and the method further includes constructing the causal machine learning model by using the domain knowledge to assign weights to the probability distributions.

[0200] FIG. 9D is a component-based flowchart of an example of communications between components of a system or apparatus 930. Illustrative examples of portions of the system or apparatus are further described throughout this disclosure, for instance with reference to FIGS. 1A, IB, 2A-2F, 3, 4, 5A, 5B, 6, 10, and 11.

[0201] The system or apparatus 930 includes at least one processor 932 and at least one memory 934. The at least one memory 934 includes a causal machine learning model. A graphical representation of the causal machine learning model includes nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of the pathogenicity variable, at least one third node of the nodes represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes. In some examples, the at least one memory further includes sub-models and each sub-model is capable of providing output for a corresponding node of the graphical representation of the causal machine learning model. In some examples, the at least one memory further includes a model abstractions component capable of determining output types of the sub-models and specific interactions between the sub-models. In some examples, the system or apparatus 930 further includes at least one first device capable of receiving a request and providing output generated by the causal machine learning model in response to the request to at least one of a data store, a model, a network, a user interface, or at least one device.

[0202] FIG. 9E is a flowchart of an example method 940. The method 940 may be used to interpret a genetic variant. Portions of the method 940 may be embodied in one or more non-transitory computer readable media, executed by one or more processors, and / or implemented in one or more components of a computing system. Illustrative examples of portions of the system or apparatus are further described throughout this disclosure, for instance with reference to FIGS. 1A, IB, 2A-2F, 3, 4, 5A, 5B, 6, and 7.

[0203] At operation 942, a processing device samples a node of a causal machine learning model to obtain sampled data, where the node is associated with the genetic variant, and where a graphical representation of the causal machine learning model includes nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of pathogenicity variable, at least one third node of the nodes represents an effect of pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes. In some examples, sampling the node includes sampling a posterior predictive distribution associated with the node.

[0204] At operation 944, the processing device uses the sampled data to determine and output an interpretation for the genetic variant. In some examples, the interpretation includes a probability that the genetic variant is pathogenic for a health condition. In some examples, the processing device receives a request via a device; and provides the interpretation of the genetic variant to at least one of a data store, a network, a model, or the device.

[0205] FIG. 9F is a flowchart of an example method 950. Portions of the method 950 may be embodied in one or more non-transitory computer readable media, executed by one or more processors, and / or implemented in one or more components of a computing system. Illustrative examples of portions of the system or apparatus are further described throughout this disclosure, for instance with reference to FIGS. 1A, IB, 2A-2F, 3, 4, 5A, 5B, 6, and 7.

[0206] At operation 952, a processing device creates input data for a causal machine learning model using pathogenicity evidence data associated with a genetic variant and a health condition, where the pathogenicity evidence data is obtained via a plurality of evidence sources, and the plurality of evidence sources includes at least one of: patient clinical data, population frequency data, published in vitro experimental study data, internally performed in vitro experimental study data, splice site loss predictions, splice site gain predictions, or predictions of nonsense mediated decay curated based on an application of a variant classification or variant interpretation framework.

[0207] In some examples, the plurality of evidence sources includes: patient clinical data evaluated through an evidence modeling platform (EMP), population frequency data evaluated through the EMP, published in vitro experimental study data evaluated through the EMP, internally performed in vitro experimental study data evaluated through the EMP), splice site loss predictions, splice site gain predictions, and predictions of nonsense mediated decay curated based on the application of the variant classification or variant interpretation framework.

[0208] At operation 954, the processing device applies the causal machine learning model to the input data to produce a trained causal model, where a graphical representation of the trained causal model includes nodes connected via acyclic directed edges, a first node represents a pathogenicity variable related to the health condition, at least one second node represents a cause of pathogenicity variable, at least one third node represents an effect of pathogenicity variable, and an acyclic directed edge represents a relationship between two of the nodes. At operation 956, in response to a request, the processing device outputs predictive data sampled from the trained causal model.

[0209] FIG. 9G is a flowchart of an example method 960. Portions of the method 960 may be embodied in one or more non-transitory computer readable media, executed by one or more processors, and / or implemented in one or more components of a computing system. Illustrative examples of portions of the system or apparatus are further described throughout this disclosure, for instance with reference to FIGS. 1A, IB, 2A-2F, 3, 4, 5A, 5B, 6, and 7.

[0210] At operation 962, a processing device applies at least one evaluation criterion to output of one or more of a plurality of machine learning-based evidence models, the plurality of machine learning-based evidence models including at least one of: a first model of patient clinical data, a second model of population frequency data, a third model of published in vitro experimental study data, a fourth model of in vitro experimental study data, a fifth model of splice site loss predictions, a sixth model of splice site gain predictions, or a seventh model of predictions of nonsense mediated decay. In some examples, the plurality of machine learning-based evidence models includes the first model of patient clinical data, the second model of population frequency data, the third model of published in vitro experimental study data,the fourth model of in vitro experimental study data, the fifth model of splice site loss predictions, a sixth model of splice site gain predictions, and the seventh model of predictions of nonsense mediated decay.

[0211] At operation 964, the processing device determines one or more weight values for the output of the plurality of machine learning-based evidence models. At operation 966, the processing device applies the one or more weight values to the output of the plurality of machine learning -based evidence models. At operation 968, in response to a request, the processing device outputs predictive data based on the applied one or more weight values.

[0212] In some examples, the method 960 includes training at least one model of the plurality of machine learning-based evidence models using supervised machine learning. In some examples, the training further includes training the at least one model of the plurality of machine learning-based evidence models using gene-specific labeled data. In some examples, the method 960 includes at least one of: evaluating at least one first model of the plurality of machine learning-based evidence models individually or evaluating at least two second models of the plurality of machine learning-based evidence models in combination.

[0213] The examples shown in FIG. 9A, FIG. 9B, FIG. 9C, FIG. 9D, FIG. 9E, FIG. 9F, and FIG. 9G and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0214] FIG. 10 illustrates an example computing system 1000 that includes a modeling system in accordance with some embodiments of the present disclosure.

[0215] In the embodiment of FIG. 10, computing system 1000 includes one or more user systems1010, a network 1020, an application software system 1030, a modeling system 1050, and a data storage system 1080. Modeling system 1050 includes a model 1052, a model creator component 1054, and a data set 1060. Modeling system 1050 can include any one or more of the components described with reference to any of the preceding figures. For example, modeling system 1050 can include one or more components of the platform 600 described with reference to FIG. 6, the system 500 described with reference to FIG. 5A, the method 560 described with reference to FIG. 5B, portions of the process 400 described with reference to FIG. 4, the component-based process 300 described with reference to FIG. 3, any of the models described with reference to any of FIG. IB or 2A-2F, and / or any portions of the method 100 described with reference to FIG. 1A.

[0216] Model 1052 includes one or more models that are configured to determine probabilistic or statistical relationships between inputs and outputs using one or more predictive algorithms For example, given one or more inputs, model 1052 outputs labels that can be used to classify the inputs into different categories or scores that can be used to sort or rank the inputs into groups or ranked lists. Examples of model 1052 include one or more of the model(s) described above.

[0217] In some embodiments, model creator component 1054 creates or configures the one or more models of model 1052 by, for example, applying the model 1052 to one or more data sets 1060. One or more of the data sets 1060 can include training examples of input data and known or ground-truth labels or scores. In some implementations, the predictive output generated by the one or more models 1052 inresponse to the one or more data sets 1060 is observed iteratively until a set of model validation criteria are satisfied. For example, differences between predictive output and expected output are quantified using a loss function. The model validation criteria are used to determine when the one or more models have converged so as to provide output that can be relied upon with some degree of certainty. The requisite level of certainty and the validation criteria are determined based on the requirements or design of a particular implementation of the one or more models.

[0218] One or more data sets 1060 include data used to configure the one or more models 1052 in some implementations. A data set 1060 can include, for example, a set of model inputs such as labeled or unlabeled training data. In some embodiments, one or more of the data sets 1060 includes or is derived from a database of historical population data. One or more of the data sets 1060 include, for example, variant-level data and / or gene -level data.

[0219] User system 1010 includes at least one computing device, such as a personal computing device, a server, a mobile computing device, or a smart appliance. User system 1010 includes at least one software application, including a user interface 1012, installed on or accessible by a network to a computing device. For example, embodiments of user interface 1012 include a graphical display screen that displays controls and graphical elements for operating and / or manipulating one or more of model 1052, model creator component 1054, and data set 1060.

[0220] User interface 1012 can be used to input data, initiate user interface events, and view or otherwise perceive output that includes data produced by modeling system 1050. Examples of user interface 1012 include web browsers, command line interfaces, and mobile app front ends. User interface 1012 as used herein can include application programming interfaces (APIs).

[0221] Application software system 1030 is any type of application software system that provides or enables the generation, display, or manipulation of output produced by modeling system 1050. Examples of application software system 1030 include but are not limited to DNA (deoxyribonucleic acid) analysis software, genetic testing software, medical testing software, healthcare management software, or any combination of any of the foregoing.

[0222] Data storage system 1080 includes data stores and / or data services that store data received, used, manipulated, and produced by application software system 1030 and / or modeling system 1050, such as training data, model parameters, validation criteria, model output, etc. In some embodiments, data storage system 1080 includes multiple different types of data storage and / or a distributed data service. As used herein, data service may refer to a physical, geographic grouping of machines, a logical grouping of machines, or a single machine. For example, a data service may be a data center, a cluster, a group of clusters, or a machine.

[0223] Data storage system 1080 resides on at least one persistent and / or volatile storage device that can reside within the same local network as at least one other device of computing system 1000 and / or in a network that is remote relative to at least one other device of computing system 1000. Thus, although depicted as being included in computing system 1000, portions of data storage system 1080 can be part of computing system 1000 or accessed by computing system 1000 over a network, such as network 1020.

[0224] While not specifically shown, it should be understood that any of user system 1010, application software system 1030, modeling system 1050, and data storage system 1080 includes an interface embodied as computer programming code stored in computer memory that when executed causes a computing device to enable bidirectional communication with any other of user system 1010, application software system 1030, modeling system 1050, and data storage system 1080 using a communicative coupling mechanism. Examples of communicative coupling mechanisms include network interfaces, inter-process communication (IPC) interfaces and application program interfaces (APIs).

[0225] Each of user system 1010, application software system 1030, modeling system 1050, and data storage system 1080 is implemented using at least one computing device that is communicatively coupled to electronic communications network 1020. Any of user system 1010, application software system 1030, modeling system 1050, and data storage system 1080 can be bidirectionally communicatively coupled by network 1020. User system 1010 as well as other different user systems (not shown) can be bidirectionally communicatively coupled to application software system 1030 and / or modeling system 1050.

[0226] A typical user of user system 1010 can be an administrator or end user of application software system 1030 and / or modeling system 1050. User system 1010 is configured to communicate bidirectionally with application software system 1030 and / or modeling system 1050 over network 1020.

[0227] The features and functionality of user system 1010, application software system 1030, modeling system 1050, and data storage system 1080 are implemented using computer software, hardware, or software and hardware, and can include combinations of automated functionality, data structures, and digital data, which are represented schematically in the figures. User system 1010, application software system 1030, modeling system 1050, and data storage system 1080 are shown as separate elements in FIG . 4 for ease of discussion but, except as otherwise described, the illustration is not meant to imply that separation of these elements is required. The illustrated systems, services, and data stores (or their functionality) of each of user system 1010, application software system 1030, modeling system 1050, and data storage system 1080 can be divided over any number of physical systems, including a single physical computer system, and can communicate with each other in any appropriate manner.

[0228] Network 1020 can be implemented on any medium or mechanism that provides for the exchange of data, signals, and / or instructions between the various components of computing system 1000. Examples of network 1020 include, without limitation, a Local Area Network (LAN), a Wide Area Network (WAN), an Ethernet network or the Internet, or at least one terrestrial, satellite or wireless link, or a combination of any number of different networks and / or communication links.

[0229] The examples shown in FIG. 10 and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0230] For ease of discussion, in FIG. 11, aspects of the modeling system 1050 are represented as modeling system 1150.

[0231] FIG. 11 illustrates an example machine of a computer system 1100 within which a set of instructions, for causing the machine to perform any of the methodologies discussed herein, can beexecuted. In some embodiments, the computer system 1100 can correspond to a component of a networked computer system (e.g., the computing system 1000 of FIG. 10) that includes, is coupled to, or utilizes a machine to execute an operating system to perform operations described above corresponding to aspects of the modeling system 1050 of FIG. 10.

[0232] The machine is connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, and / or the Internet. The machine can operate in the capacity of a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0233] The machine is a personal computer (PC), a smart phone, a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instmctions to perform any of the methodologies discussed herein.

[0234] The example computer system 1100 includes a processing device 1102, a main memory 1104 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a memory 1105 (e.g., flash memory, static random access memory (SRAM), etc.), an input / output system 1110, and a data storage system 1140, which communicate with each other via a bus 1130.

[0235] Processing device 1102 represents at least one general-purpose processing device such as a microprocessor, a central processing unit, or the like. More particularly, the processing device can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 1102 can also be at least one special-purpose processing device such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 1102 is configured to execute instructions 1112 for performing the operations and steps discussed herein.

[0236] Instructions 1112 include portions of the modeling system 1150 when the processing device is executing those portions of the modeling system 1150. Thus, the modeling system is shown in dashed lines as part of instructions 1112 to illustrate that, at times, portions of the modeling system are executed by processing device 1102. For example, when at least some portion of the modeling system is embodied in instructions to cause processing device 1102 to perform the method(s) described above, some of those instmctions can be read into processing device 1102 (e.g., into an internal cache or other memory) from main memory 1104 and / or data storage system 1140. However, it is not required that all of the modeling system be included in instmctions 1112 at the same time and portions of the modeling system are stored in at least one other component of computer system 1100 at other times, e.g., when at least one portion of the modeling system are not being executed by processing device 1102.

[0237] The computer system 1100 further includes a network interface device 1108 to communicate over the network 1120. Network interface device 1108 provides a two-way data communication coupling to a network. For example, network interface device 1108 can be an integrated-services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, network interface device 1108 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, network interface device 1108 can send and receive electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0238] The network link can provide data communication through at least one network to other data devices. For example, a network link can provide a connection to the world-wide packet data communication network commonly referred to as the “Internet,” for example through a local network to a host computer or to data equipment operated by an Internet Service Provider (ISP). Local networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data to and from computer system 1100.

[0239] Computer system 1100 can send messages and receive data, including program code, through the network(s) and network interface device 1108. In the Internet example, a server can transmit a requested code for an application program through the Internet and network interface device 1108. The received code can be executed by processing device 1102 as it is received, and / or stored in data storage system 1140, or other non-volatile storage for later execution.

[0240] The input / output system 1110 includes an output device, such as a display, for example a liquid crystal display (LCD) or a touchscreen display, for displaying information to a computer user, or a speaker, a haptic device, or another form of output device. The input / output system 1110 can include an input device, for example, alphanumeric keys and other keys configured for communicating information and command selections to processing device 1102. An input device can, alternatively or in addition, include a cursor control, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processing device 1102 and for controlling cursor movement on a display. An input device can, alternatively or in addition, include a microphone, a sensor, or an array of sensors, for communicating sensed information to processing device 1102. Sensed information can include voice commands, audio signals, geographic location information, and / or digital imagery, for example.

[0241] The data storage system 1140 includes a machine-readable storage medium 1142 (also known as a computer-readable medium) on which is stored at least one set of instructions 1144 or software embodying any of the methodologies or functions described herein. The instructions 1144 can also reside, completely or at least partially, within the main memory 1104 and / or within the processing device 1102 during execution thereof by the computer system 1100, the main memory 1104 and the processing device 1102 also constituting machine-readable storage media.

[0242] In one embodiment, the instructions 1144 include instructions to implement functionality corresponding to a modeling system (e.g., the modeling system 1050 of FIG. 10).

[0243] Dashed lines are used in FIG. 11 to indicate that it is not required that the modeling system be embodied entirely in instructions 1112, 1114, and 1144 at the same time. In one example, portions of the modeling system are embodied in instructions 1144, which are read into main memory 1104 as instructions 1114, and portions of instructions 1114 are read into processing device 1102 as instructions 1112 for execution. In another example, some portions of the modeling system are embodied in instructions 1144 while other portions are embodied in instructions 1114 and still other portions are embodied in instructions 1112.

[0244] While the machine-readable storage medium 1142 is shown in an example embodiment to be a single medium, the term “machine-readable storage medium” should be taken to include a single medium or multiple media that store the at least one set of instructions. The term “machine-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any of the methodologies of the present disclosure. The term “machine-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0245] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to convey the substance of their work most effectively to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0246] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure can refer to the action and processes of a computer system, or similar electronic computing device, which manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage systems.

[0247] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed for the intended purposes, or it can include a general -purpose computer selectively activated or reconfigured by a computer program stored in the computer. For example, a computer system or other data processing system, such as the computing system 1000, can carry out the above-described technologies in response to its processor executing a computer program (e.g., a sequence of instructions) contained in a memory or other non-transitory machine-readable storage medium. Such a computer program can be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic oroptical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0248] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the method. The structure for a variety of these systems will appear as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the disclosure as described herein.

[0249] The present disclosure can be provided as a computer program product, or software, which can include a machine-readable medium having stored thereon instructions, which can be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as a read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory components, etc.

[0250] The examples shown in FIG. 11 and the accompanying description, above are provided for illustration purposes. This disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0251] Illustrative aspects of the technologies disclosed herein are provided below. An embodiment of the technologies may include any of the aspects described herein, or any combination of any of the aspects described herein, or any combination of any portions of the aspects described herein. An embodiment may include a system, a method (e.g., a computer-implemented method), an apparatus, or a non-transitory computer readable medium configured in accordance with one or more of the aspects described herein.

[0252] In some aspects, the techniques described herein relate to a system for modeling cause-and- effect relationships between pathogenicity evidence variables for a genetic variant and a health condition, including: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory includes: a data pre-processing component to create input data for a causal machine learning model using pathogenicity evidence data associated with the genetic variant and the health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources; a model training component to apply the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model includes nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable of the pathogenicity evidence variables related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and a prediction component to, in response to a request, output predictive data sampled from the trained causal model.

[0253] In some aspects, the techniques described herein relate to a system, wherein the prediction component is to use the trained causal model to generate and output a prediction as to whether the genetic variant is benign or pathogenic with respect to the health condition.

[0254] In some aspects, the techniques described herein relate to a system, wherein the prediction component is to provide the prediction as to whether the genetic variant is benign or pathogenic to a clinician for use in formulating a diagnosis of a patient.

[0255] In some aspects, the techniques described herein relate to a system, wherein the prediction component is to store the prediction for retrieval via at least one query.

[0256] In some aspects, the techniques described herein relate to a system, wherein, in the graphical representation of the trained causal model, the at least one second node is upstream of the first node and the at least one third node is downstream of the first node and the at least one second node.

[0257] In some aspects, the techniques described herein relate to a system, wherein, in the graphical representation of the trained causal model, the at least one second node includes at least one variable that represents data pertaining to at least one biological property that causes pathogenicity of the genetic variant with respect to the health condition.

[0258] In some aspects, the techniques described herein relate to a system, wherein the at least one variable includes at least one of: a variable pertaining to an impact of the genetic variant on protein function, a variable pertaining to an impact of the genetic variant on protein stability, a variable pertaining to an impact of the genetic variant on mRNA processing, a variable pertaining to the impact of the genetic variant on nonsense mediated decay, a variable pertaining to an impact of the genetic variant on mRNA, a variable pertaining to an impact of the genetic variant on protein expression, a variable pertaining to a biological criticality of disrupted DNA, a variable pertaining to a biological criticality of a disrupted RNA nucleotide, or a variable pertaining to a biological criticality of a disrupted protein amino acid residue.

[0259] In some aspects, the techniques described herein relate to a system, wherein, in the graphical representation of the trained causal model, the at least one third node includes at least one variable that represents data pertaining to at least one biological property that is an effect of pathogenicity of the genetic variant with respect to the health condition.

[0260] In some aspects, the techniques described herein relate to a system, wherein the at least one variable includes at least one of: a variable pertaining to an impact of pathogenicity on an individual patient phenotype, a variable pertaining to an impact of pathogenicity on a population allele frequency, a variable pertaining to an impact of pathogenicity on a select cohort allele frequency, or a variable pertaining to an impact of pathogenicity on familial genotype -phenotype co-segregation patterns.

[0261] In some aspects, the techniques described herein relate to a system, wherein the at least one memory further includes a model evaluation component to apply at least one performance criterion to the trained causal model.

[0262] In some aspects, the techniques described herein relate to a system, wherein, for a category of the pathogenicity evidence data, the plurality of evidence sources includes alternative types of models for predicting pathogenicity given the category of the pathogenicity evidence data, and the data preprocessing component is to, for the category of the pathogenicity evidence data, use at least oneevaluation criterion to create the input data using a model selected from among the alternative types of models.

[0263] In some aspects, the techniques described herein relate to a system, wherein the graphical representation of the trained causal model includes at least one of: a gene-level variable, a gene-level constraint, or a variant-level constraint.

[0264] In some aspects, the techniques described herein relate to a system, wherein the prediction component is to sample data from any node of the trained causal model, and a node sampled by the prediction component corresponds to an unobserved variable with respect to the genetic variant and the health condition.

[0265] In some aspects, the techniques described herein relate to a method for creating a training data set for a causal machine learning model, including: applying a natural language processing (NLP) model to clinical phenotype data for a genetic variant; using the NLP model, identifying features of the clinical phenotype data that are predictive of a molecular diagnosis of a health condition with respect to the genetic variant; for each subject in a population of subjects, using the identified features and demographic information to compute a first score including a probability of the subject being affected with the health condition; and for the population of subjects, including the first scores in the training data set.

[0266] In some aspects, the techniques described herein relate to a method, further including: including labeled pathogenicity data in the training data set.

[0267] In some aspects, the techniques described herein relate to a method, further including: using the NLP model to extract the clinical phenotype data from at least one field of a test requisition form, wherein the at least one field includes at least one of an indication field or a family history field.

[0268] In some aspects, the techniques described herein relate to a method, further including: obtaining data relating to a genetic variant via a plurality of sources; filtering the data using at least one filtering criterion; converting the filtered data to a standardized format; and including the data in the standardized format in the training data set.

[0269] In some aspects, the techniques described herein relate to a method, wherein the data includes at least one of: output of at least one machine learning model, empirical measurements, predicted variant effects, or public datasets.

[0270] In some aspects, the techniques described herein relate to a method, wherein the standardized format includes a data structure usable as input to the causal machine learning model.

[0271] In some aspects, the techniques described herein relate to a method, further including: splitting the data into at least one training data set to train the causal machine learning model and at least one holdout data set to validate the trained causal machine learning model.

[0272] In some aspects, the techniques described herein relate to a method for training a causal machine learning model for variant interpretation, including: creating model input data using pathogenicity evidence data for a variant; applying the causal machine learning model to the model input data, wherein a graphical representation of the causal machine learning model includes nodes connected via acyclic directed edges, a first node represents a pathogenicity variable, at least one second node represents a cause of the pathogenicity variable, at least one third node represents an effect of thepathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes; and evaluating data sampled from the causal machine learning model iteratively until at least one performance metric exceeds at least one threshold performance criterion.

[0273] In some aspects, the techniques described herein relate to a method, further including constructing the causal machine learning model by: linking the at least one second node to the first node via at least one upstream directed acyclic edge; and linking the at least one third node to the first node via at least one downstream directed acyclic edge.

[0274] In some aspects, the techniques described herein relate to a method, further including constructing the causal machine learning model using domain knowledge to at least one of: identify at least one cause of the pathogenicity variable and assign the at least one cause of the pathogenicity variable to the at least one second node; identify at least one effect of the pathogenicity variable and assign the at least one effect of the pathogenicity variable to the at least one third node; link the at least one second node to the first node via at least one upstream acyclic directed edge; or link the at least one third node to the first node via at least one downstream acyclic directed edge.

[0275] In some aspects, the techniques described herein relate to a method, further including: obtaining the model input data via a plurality of sources; and using domain knowledge to filter the model input data, wherein the domain knowledge includes types of relationships between variables, the variables include probability distributions, and the method further includes constructing the causal machine learning model by using the domain knowledge to assign weights to the probability distributions.

[0276] In some aspects, the techniques described herein relate to a method, further including: constructing the causal machine learning model by fitting a hierarchical Bayesian inference model to the model input data, wherein the model input data includes phenotype data and variant pathogenicity data.

[0277] In some aspects, the techniques described herein relate to an apparatus for variant interpretation, including: at least one processor; and at least one memory, wherein the at least one memory includes a causal machine learning model, wherein a graphical representation of the causal machine learning model includes nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of the pathogenicity variable, at least one third node of the nodes represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes.

[0278] In some aspects, the techniques described herein relate to an apparatus, wherein the at least one memory further includes sub-models and each sub-model is capable of providing output for a corresponding node of the graphical representation of the causal machine learning model.

[0279] In some aspects, the techniques described herein relate to an apparatus, wherein the at least one memory further includes a model abstractions component capable of determining output types of the submodels and specific interactions between the sub-models.

[0280] In some aspects, the techniques described herein relate to an apparatus, wherein the apparatus further includes at least one first device capable of receiving a request and providing output generated by the causal machine learning model in response to the request to at least one of a data store, a network, a user interface, or at least one device.

[0281] In some aspects, the techniques described herein relate to a method for interpreting a genetic variant, including: sampling a node of a causal machine learning model to obtain sampled data, wherein the node is associated with the genetic variant, and wherein a graphical representation of the causal machine learning model includes nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of the pathogenicity variable, at least one third node of the nodes represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes; and using the sampled data to determine and output an interpretation for the genetic variant.

[0282] In some aspects, the techniques described herein relate to a method, wherein sampling the node further includes: sampling a posterior predictive distribution associated with the node.

[0283] In some aspects, the techniques described herein relate to a method, wherein the interpretation includes a probability that the genetic variant is pathogenic for a health condition.

[0284] In some aspects, the techniques described herein relate to a method, further including: receiving a request via a device; and providing the interpretation of the genetic variant to at least one of a data store, a network, or the device.

[0285] In some aspects, the techniques described herein relate to a system for modeling cause-and- effect relationships between pathogenicity evidence variables for a genetic variant and a health condition, including: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory includes: a data pre-processing component to create input data for a causal machine learning model using pathogenicity evidence data associated with the genetic variant and the health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources, wherein the plurality of evidence sources includes at least one of: patient clinical data, population frequency data, published in vitro experimental study data, internally performed in vitro experimental study data, splice site loss predictions, splice site gain predictions, or predictions of nonsense mediated decay curated based on an application of a variant classification or variant interpretation framework; a model training component to apply the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model includes nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable of the pathogenicity evidence variables related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and a prediction component to, in response to a request, output predictive data sampled from the trained causal model.

[0286] In some aspects, the techniques described herein relate to a system, wherein the plurality of evidence sources includes: patient clinical data evaluated through an evidence modeling platform (EMP), population frequency data evaluated through the EMP, published in vitro experimental study data evaluated through the EMP, internally performed in vitro experimental study data evaluated through the EMP), splice site loss predictions, splice site gain predictions, and predictions of nonsense mediated decay curated based on the application of the variant classification or variant interpretation framework.

[0287] In some aspects, the techniques described herein relate to a method including: creating input data for a causal machine learning model using pathogenicity evidence data associated with a genetic variant and a health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources, and the plurality of evidence sources includes at least one of: patient clinical data, population frequency data, published in vitro experimental study data, internally performed in vitro experimental study data, splice site loss predictions, splice site gain predictions, or predictions of nonsense mediated decay curated based on an application of a variant classification or variant interpretation framework; applying the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model includes nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and in response to a request, outputting predictive data sampled from the trained causal model.

[0288] In some aspects, the techniques described herein relate to a method, wherein the plurality of evidence sources includes: patient clinical data evaluated through an evidence modeling platform (EMP), population frequency data evaluated through the EMP, published in vitro experimental study data evaluated through the EMP, internally performed in vitro experimental study data evaluated through the EMP), splice site loss predictions, splice site gain predictions, and predictions of nonsense mediated decay curated based on the application of the variant classification or variant interpretation framework.

[0289] In some aspects, the techniques described herein relate to an apparatus including: at least one processor; and at least one memory, wherein the at least one memory includes: a plurality of machine learning-based evidence models including at least one of: a first model of patient clinical data, a second model of population frequency data, a third model of published in vitro experimental study data, a fourth model of in vitro experimental study data, a fifth model of splice site loss predictions, a sixth model of splice site gain predictions, or a seventh model of predictions of nonsense mediated decay; a model evaluation component to apply at least one evaluation criterion to output of one or more of the plurality of machine learning-based evidence models; a model calibration component to use output of the model evaluation component to determine and apply weight values to the output of one or more of the plurality of machine learning-based evidence models; and a prediction component to output the output of the model calibration component to at least one device, system, process, model, component, or application.

[0290] In some aspects, the techniques described herein relate to an apparatus, wherein the at least one memory further includes a model training component to train at least one model of the plurality of machine learning-based evidence models using supervised machine learning.

[0291] In some aspects, the techniques described herein relate to an apparatus, wherein the model training component further trains the at least one model of the plurality of machine learning-based evidence models using gene-specific labeled data.

[0292] In some aspects, the techniques described herein relate to an apparatus, wherein the plurality of machine learning-based evidence models includes the first model of patient clinical data, the secondmodel of population frequency data, the third model of published in vitro experimental study data, the fourth model of in vitro experimental study data, the fifth model of splice site loss predictions, a sixth model of splice site gain predictions, and the seventh model of predictions of nonsense mediated decay.

[0293] In some aspects, the techniques described herein relate to an apparatus, wherein the model evaluation component is to at least one of: evaluate at least one first model of the plurality of machine learning-based evidence models individually or evaluate at least two second models of the plurality of machine learning-based evidence models in combination.

[0294] In some aspects, the techniques described herein relate to a method including: applying at least one evaluation criterion to output of one or more of a plurality of machine learning -based evidence models, the plurality of machine learning-based evidence models including at least one of: a first model of patient clinical data, a second model of population frequency data, a third model of published in vitro experimental study data, a fourth model of in vitro experimental study data, a fifth model of splice site loss predictions, a sixth model of splice site gain predictions, or a seventh model of predictions of nonsense mediated decay; determining one or more weight values for the output of the plurality of machine learning-based evidence models; applying the one or more weight values to the output of the plurality of machine learning-based evidence models; and in response to a request, outputting predictive data based on the applied one or more weight values.

[0295] In some aspects, the techniques described herein relate to a method, further including training at least one model of the plurality of machine learning-based evidence models using supervised machine learning.

[0296] In some aspects, the techniques described herein relate to a method, wherein the training further includes training the at least one model of the plurality of machine learning-based evidence models using gene-specific labeled data.

[0297] In some aspects, the techniques described herein relate to a method, wherein the plurality of machine learning-based evidence models includes the first model of patient clinical data, the second model of population frequency data, the third model of published in vitro experimental study data, the fourth model of in vitro experimental study data, the fifth model of splice site loss predictions, a sixth model of splice site gain predictions, and the seventh model of predictions of nonsense mediated decay.

[0298] In some aspects, the techniques described herein relate to a method, further including at least one of: evaluating at least one first model of the plurality of machine learning-based evidence models individually or evaluating at least two second models of the plurality of machine learning-based evidence models in combination.

[0299] In some aspects the techniques described herein related to any one or more aspects, steps, components, elements, processes, or limitations that are at least one of described in the enclosed description and / or shown in the accompanying drawings.

[0300] Clause 1. A system for modeling cause -and-effect relationships between pathogenicity evidence variables for a genetic variant and a health condition, comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises: a data pre-processing component to create input data for a causal machine learning model using pathogenicityevidence data associated with the genetic vanant and the health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources; a model training component to apply the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable of the pathogenicity evidence variables related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and a prediction component to, in response to a request, output predictive data sampled from the trained causal model.

[0301] Clause 2. The system of clause 1, wherein the prediction component is to use the trained causal model to generate and output a prediction as to whether the genetic variant is benign or pathogenic with respect to the health condition.

[0302] Clause 3. The system of clause 2, wherein the prediction component is to provide the prediction as to whether the genetic variant is benign or pathogenic to a clinician for use in formulating a diagnosis of a patient.

[0303] Clause 4. The system of clause 2 or clause 3, wherein the prediction component is to store the prediction for retrieval via at least one query.

[0304] Clause 5. The system of any of clauses 1-4, wherein, in the graphical representation of the trained causal model, the at least one second node is upstream of the first node and the at least one third node is downstream of the first node and the at least one second node.

[0305] Clause 6. The system of any of clauses 1-5, wherein, in the graphical representation of the trained causal model, the at least one second node comprises at least one variable that represents data pertaining to at least one biological property that causes pathogenicity of the genetic variant with respect to the health condition.

[0306] Clause 7. The system of clause 6, wherein the at least one variable comprises at least one of: a variable pertaining to an impact of the genetic variant on protein function, a variable pertaining to an impact of the genetic variant on protein stability, a variable pertaining to an impact of the genetic variant on mRNA processing, a variable pertaining to the impact of the genetic variant on nonsense mediated decay, a variable pertaining to an impact of the genetic variant on mRNA, a variable pertaining to an impact of the genetic variant on protein expression, a variable pertaining to a biological criticality of disrupted DNA, a variable pertaining to a biological criticality of a disrupted RNA nucleotide, or a variable pertaining to a biological criticality of a disrupted protein amino acid residue.

[0307] Clause 8. The system of any of clauses 1-7, wherein, in the graphical representation of the trained causal model, the at least one third node comprises at least one variable that represents data pertaining to at least one biological property that is an effect of pathogenicity of the genetic variant with respect to the health condition.

[0308] Clause 9. The system of clause 8, wherein the at least one variable comprises at least one of: a variable pertaining to an impact of pathogenicity on an individual patient phenotype, a variable pertainingto an impact of pathogenicity on a population allele frequency, a variable pertaining to an impact of pathogenicity on a select cohort allele frequency, or a variable pertaining to an impact of pathogenicity on familial genotype -phenotype co-segregation patterns.

[0309] Clause 10. The system of any of clauses 1-9, wherein the at least one memory further comprises a model evaluation component to apply at least one performance criterion to the trained causal model.

[0310] Clause 11. The system of any of clauses 1-10, wherein, for a category of the pathogenicity evidence data, the plurality of evidence sources comprises alternative types of models for predicting pathogenicity given the category of the pathogenicity evidence data, and the data pre-processing component is to, for the category of the pathogenicity evidence data, use at least one evaluation criterion to create the input data using a model selected from among the alternative types of models.

[0311] Clause 12 The system of any of clauses 1-11, wherein the graphical representation of the trained causal model comprises at least one of: a gene-level variable, a gene-level constraint, or a variantlevel constraint.

[0312] Clause 13. The system of any of clauses 1-12, wherein the prediction component is to sample data from any node of the trained causal model, and a node sampled by the prediction component corresponds to an unobserved variable with respect to the genetic variant and the health condition.

[0313] Clause 14. A method for creating a training data set for a causal machine learning model, comprising: applying a natural language processing (NLP) model to clinical phenotype data for a genetic variant; using the NLP model, identifying features of the clinical phenotype data that are predictive of a molecular diagnosis of a health condition with respect to the genetic variant; for each subject in a population of subjects, using the identified features and demographic information to compute a first score comprising a probability of the subject being affected with the health condition; and for the population of subjects, including the first scores in the training data set.

[0314] Clause 15. The method of clause 14, further comprising: including labeled pathogenicity data in the training data set.

[0315] Clause 16. The method of clause 14 or clause 15, further comprising: using the NLP model to extract the clinical phenotype data from at least one field of a test requisition form, wherein the at least one field comprises at least one of an indication field or a family history field.

[0316] Clause 17. The method of any of clauses 14-16, further comprising: obtaining data relating to a genetic variant via a plurality of sources; filtering the data using at least one filtering criterion; converting the filtered data to a standardized format; and including the data in the standardized format in the training data set.

[0317] Clause 18. The method of clause 17, wherein the data comprises at least one of: output of at least one machine learning model, empirical measurements, predicted variant effects, or public datasets.

[0318] Clause 19. The method of clause 17 or clause 18, wherein the standardized format comprises a data structure usable as input to the causal machine learning model.

[0319] Clause 20. The method of clause 17-19, further comprising: splitting the data into at least one training data set to train the causal machine learning model and at least one holdout data set to validate the trained causal machine learning model.

[0320] Clause 21. A method for training a causal machine learning model for variant interpretation, comprising: creating model input data using pathogenicity evidence data for a variant; applying the causal machine learning model to the model input data, wherein a graphical representation of the causal machine learning model comprises nodes connected via acyclic directed edges, a first node represents a pathogenicity variable, at least one second node represents a cause of the pathogenicity variable, at least one third node represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes; and evaluating data sampled from the causal machine learning model iteratively until at least one performance metric exceeds at least one threshold performance criterion.

[0321] Clause 22. The method of clause 21, further comprising constructing the causal machine learning model by: linking the at least one second node to the first node via at least one upstream directed acyclic edge; and linking the at least one third node to the first node via at least one downstream directed acyclic edge.

[0322] Clause 23. The method of clause 21 or clause 22, further comprising constructing the causal machine learning model using domain knowledge to at least one of: identify at least one cause of the pathogenicity variable and assign the at least one cause of the pathogenicity variable to the at least one second node; identify at least one effect of the pathogenicity variable and assign the at least one effect of the pathogenicity variable to the at least one third node; link the at least one second node to the first node via at least one upstream acyclic directed edge; or link the at least one third node to the first node via at least one downstream acyclic directed edge.

[0323] Clause 24. The method of clause 23, further comprising: obtaining the model input data via a plurality of sources; and using domain knowledge to filter the model input data, wherein the domain knowledge comprises types of relationships between variables, the variables comprise probability distributions, and the method further comprises constructing the causal machine learning model by using the domain knowledge to assign weights to the probability distributions.

[0324] Clause 25. The method of any of clauses 21-24, further comprising: constructing the causal machine learning model by fitting a hierarchical Bayesian inference model to the model input data, wherein the model input data comprises phenotype data and variant pathogenicity data.

[0325] Clause 26. An apparatus for variant interpretation, comprising: at least one processor; and at least one memory, wherein the at least one memory comprises a causal machine learning model, wherein a graphical representation of the causal machine learning model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of the pathogenicity variable, at least one third node of the nodes represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes.

[0326] Clause 27. The apparatus of clause 26, wherein the at least one memory further comprises submodels and each sub-model is capable of providing output for a corresponding node of the graphical representation of the causal machine learning model.

[0327] Clause 28. The apparatus of clause 27, wherein the at least one memory further comprises a model abstractions component capable of determining output types of the sub-models and specific interactions between the sub-models.

[0328] Clause 29. The apparatus of any of clauses 26-28, wherein the apparatus further comprises at least one first device capable of receiving a request and providing output generated by the causal machine learning model in response to the request to at least one of a data store, a network, a user interface, or at least one device.

[0329] Clause 30. A method for interpreting a genetic variant, comprising: sampling a node of a causal machine learning model to obtain sampled data, wherein the node is associated with the genetic variant, and wherein a graphical representation of the causal machine learning model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of the pathogenicity variable, at least one third node of the nodes represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes; and using the sampled data to determine and output an interpretation for the genetic variant.

[0330] Clause 31. The method of clause 30, wherein sampling the node further comprises: sampling a posterior predictive distribution associated with the node.

[0331] Clause 32. The method of clause 30 or clause 31, wherein the interpretation comprises a probability that the genetic variant is pathogenic for a health condition.

[0332] Clause 33. The method of any of clauses 30-32, further comprising: receiving a request via a device; and providing the interpretation of the genetic variant to at least one of a data store, a network, or the device.

[0333] Clause 34. A system for modeling cause-and-effect relationships between pathogenicity evidence variables for a genetic variant and a health condition, comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises: a data pre-processing component to create input data for a causal machine learning model using pathogenicity evidence data associated with the genetic vanant and the health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources, wherein the plurality of evidence sources comprises at least one of: patient clinical data, population frequency data, published in vitro experimental study data, internally performed in vitro experimental study data, splice site loss predictions, splice site gain predictions, or predictions of nonsense mediated decay curated based on an application of a variant classification or variant interpretation framework; a model training component to apply the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable of the pathogenicity evidence variables related to the health condition, at least one second node of the nodes represents a cause of thepathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and a prediction component to, in response to a request, output predictive data sampled from the trained causal model.

[0334] Clause 35. The system of clause 34, wherein the plurality of evidence sources comprises: patient clinical data evaluated through an evidence modeling platform (EMP), population frequency data evaluated through the EMP, published in vitro experimental study data evaluated through the EMP, internally performed in vitro experimental study data evaluated through the EMP), splice site loss predictions, splice site gain predictions, and predictions of nonsense mediated decay curated based on the application of the variant classification or variant interpretation framework.

[0335] Clause 36. A method comprising: creating input data for a causal machine learning model using pathogenicity evidence data associated with a genetic variant and a health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources, and the plurality of evidence sources comprises at least one of: patient clinical data, population frequency data, published in vitro experimental study data, internally performed in vitro experimental study data, splice site loss predictions, splice site gain predictions, or predictions of nonsense mediated decay curated based on an application of a variant classification or variant interpretation framework; applying the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and in response to a request, outputting predictive data sampled from the trained causal model.

[0336] Clause 37. The method of clause 36, wherein the plurality of evidence sources comprises: patient clinical data evaluated through an evidence modeling platform (EMP), population frequency data evaluated through the EMP, published in vitro experimental study data evaluated through the EMP, internally performed in vitro experimental study data evaluated through the EMP), splice site loss predictions, splice site gain predictions, and predictions of nonsense mediated decay curated based on the application of the variant classification or variant interpretation framework.

[0337] Clause 38. An apparatus comprising: at least one processor; and at least one memory, wherein the at least one memory comprises: a plurality of machine learning -based evidence models comprising at least one of: a first model of patient clinical data, a second model of population frequency data, a third model of published in vitro experimental study data, a fourth model of in vitro experimental study data, a fifth model of splice site loss predictions, a sixth model of splice site gain predictions, or a seventh model of predictions of nonsense mediated decay; a model evaluation component to apply at least one evaluation criterion to output of one or more of the plurality of machine learning-based evidence models; a model calibration component to use output of the model evaluation component to determine and apply weight values to the output of one or more of the plurality of machine learning-based evidence models; and aprediction component to output the output of the model calibration component to at least one device, system, process, model, component, or application.

[0338] Clause 39. The apparatus of clause 38, wherein the at least one memory further comprises a model training component to train at least one model of the plurality of machine learnin -based evidence models using supervised machine learning.

[0339] Clause 40. The apparatus of clause 39, wherein the model training component further trains the at least one model of the plurality of machine learning-based evidence models using gene-specific labeled data.

[0340] Clause 41. The apparatus of any of clauses 38-40, wherein the plurality of machine learningbased evidence models comprises the first model of patient clinical data, the second model of population frequency data, the third model of published in vitro experimental study data, the fourth model of in vitro experimental study data, the fifth model of splice site loss predictions, a sixth model of splice site gain predictions, and the seventh model of predictions of nonsense mediated decay.

[0341] Clause 42. The apparatus of any of clauses 38-41, wherein the model evaluation component is to at least one of evaluate at least one first model of the plurality of machine learning-based evidence models individually or evaluate at least two second models of the plurality of machine learning-based evidence models in combination.

[0342] Clause 43. A method comprising: applying at least one evaluation criterion to output of one or more of a plurality of machine learning-based evidence models, the plurality of machine learning-based evidence models comprising at least one of: a first model of patient clinical data, a second model of population frequency data, a third model of published in vitro experimental study data, a fourth model of in vitro experimental study data, a fifth model of splice site loss predictions, a sixth model of splice site gain predictions, or a seventh model of predictions of nonsense mediated decay; determining one or more weight values for the output of the plurality of machine learning-based evidence models; applying the one or more weight values to the output of the plurality of machine learning-based evidence models; and in response to a request, outputting predictive data based on the applied one or more weight values.

[0343] Clause 44. The method of clause 43, further comprising training at least one model of the plurality of machine learning-based evidence models using supervised machine learning.

[0344] Clause 45. The method of clause 44, wherein the training further comprises training the at least one model of the plurality of machine learning-based evidence models using gene-specific labeled data.

[0345] Clause 46. The method of any of clauses 43-45, wherein the plurality of machine learningbased evidence models comprises the first model of patient clinical data, the second model of population frequency data, the third model of published in vitro experimental study data, the fourth model of in vitro experimental study data, the fifth model of splice site loss predictions, a sixth model of splice site gain predictions, and the seventh model of predictions of nonsense mediated decay.

[0346] Clause 47. The method of any of clauses 38-46, further comprising at least one of: evaluating at least one first model of the plurality of machine learning-based evidence models individually or evaluating at least two second models of the plurality of machine learning -based evidence models in combination.

[0347] In the foregoing specification, embodiments of the disclosure have been described with reference to specific example embodiments thereof. It will be evident that various modifications can be made thereto without departing from the broader spirit and scope of embodiments of the disclosure as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

CLAIMSWhat is claimed is:

1. A system for modeling cause -and-effect relationships between pathogenicity evidence variables for a genetic variant and a health condition, comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises: a data pre-processing component to create input data for a causal machine learning model using pathogenicity evidence data associated with the genetic variant and the health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources; a model training component to apply the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable of the pathogenicity evidence variables related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and a prediction component to, in response to a request, output predictive data sampled from the trained causal model.

2. The system of claim 1, wherein the prediction component is to use the trained causal model to generate and output a prediction as to whether the genetic variant is benign or pathogenic with respect to the health condition.

3. The system of claim 2, wherein the prediction component is to provide the prediction as to whether the genetic variant is benign or pathogenic to a clinician for use in formulating a diagnosis of a patient.

4. The system of claim 2, wherein the prediction component is to store the prediction for retrieval via at least one query.

5. The system of claim 1, wherein, in the graphical representation of the trained causal model, the at least one second node is upstream of the first node and the at least one third node is downstream of the first node and the at least one second node.

6. The system of claim 1, wherein, in the graphical representation of the trained causal model, the at least one second node comprises at least one variable that represents data pertaining to at least onebiological property that causes pathogenicity of the genetic variant with respect to the health condition.

7. The system of claim 6, wherein the at least one variable comprises at least one of: a variable pertaining to an impact of the genetic variant on protein function, a variable pertaining to an impact of the genetic variant on protein stability, a variable pertaining to an impact of the genetic variant on mRNA processing, a variable pertaining to the impact of the genetic variant on nonsense mediated decay, a variable pertaining to an impact of the genetic variant on mRNA, a variable pertaining to an impact of the genetic variant on protein expression, a variable pertaining to a biological criticality of disrupted DNA, a variable pertaining to a biological criticality of a disrupted RNA nucleotide, or a variable pertaining to a biological criticality of a disrupted protein amino acid residue.

8. The system of claim 1, wherein, in the graphical representation of the trained causal model, the at least one third node comprises at least one variable that represents data pertaining to at least one biological property that is an effect of pathogenicity of the genetic variant with respect to the health condition.

9. The system of claim 8, wherein the at least one variable comprises at least one of: a variable pertaining to an impact of pathogenicity on an individual patient phenotype, a variable pertaining to an impact of pathogenicity on a population allele frequency, a variable pertaining to an impact of pathogenicity on a select cohort allele frequency, or a variable pertaining to an impact of pathogenicity on familial genotype-phenotype co-segregation patterns.

10. The system of claim 1, wherein the at least one memory further comprises a model evaluation component to apply at least one performance criterion to the trained causal model.

11. The system of claim 1, wherein, for a category of the pathogenicity evidence data, the plurality of evidence sources comprises alternative types of models for predicting pathogenicity given the category of the pathogenicity evidence data, and the data pre-processing component is to, for the category of the pathogenicity evidence data, use at least one evaluation criterion to create the input data using a model selected from among the alternative types of models.

12. The system of claim 1, wherein the graphical representation of the trained causal model comprises at least one of: a gene-level variable, a gene-level constraint, or a variant-level constraint.

13. The system of claim 1, wherein the prediction component is to sample data from any node of the trained causal model, and anode sampled by the prediction component corresponds to an unobserved variable with respect to the genetic variant and the health condition.

14. A method for creating a training data set for a causal machine learning model, comprising: applying a natural language processing (NLP) model to clinical phenotype data for a genetic variant; using the NLP model, identifying features of the clinical phenotype data that are predictive of a molecular diagnosis of a health condition with respect to the genetic variant; for each subject in a population of subjects, using the identified features and demographic information to compute a first score comprising a probability of the subject being affected with the health condition; and for the population of subjects, including the first scores in the training data set.

15. The method of claim 14, further comprising: including labeled pathogenicity data in the training data set.

16. The method of claim 14, further comprising: using the NLP model to extract the clinical phenotype data from at least one field of a test requisition form, wherein the at least one field comprises at least one of an indication field or a family history field.

17. The method of claim 14, further comprising: obtaining data relating to a genetic variant via a plurality of sources; filtering the data using at least one filtering criterion; converting the filtered data to a standardized format; and including the data in the standardized format in the training data set.

18. The method of claim 17, wherein the data comprises at least one of: output of at least one machine learning model, empirical measurements, predicted variant effects, or public datasets.

19. The method of claim 17, wherein the standardized format comprises a data structure usable as input to the causal machine learning model.

20. The method of claim 17, further comprising: splitting the data into at least one training data set to train the causal machine learning model and at least one holdout data set to validate the trained causal machine learning model.

21. A method for training a causal machine learning model, comprising: creating model input data using pathogenicity evidence data for a variant; applying the causal machine learning model to the model input data, wherein a graphical representation of the causal machine learning model comprises nodes connected via acyclic directed edges, a first node represents a pathogenicity variable, at least one second node represents a cause of thepathogenicity variable, at least one third node represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes; and evaluating data sampled from the causal machine learning model iteratively until at least one performance metric exceeds at least one threshold performance criterion.

22. The method of claim 21, further comprising constructing the causal machine learning model by: linking the at least one second node to the first node via at least one upstream directed acyclic edge; and linking the at least one third node to the first node via at least one downstream directed acyclic edge.

23. The method of claim 21, further comprising constructing the causal machine learning model using domain knowledge to at least one of: identify at least one cause of the pathogenicity variable and assign the at least one cause of the pathogenicity variable to the at least one second node; identify at least one effect of the pathogenicity variable and assign the at least one effect of the pathogenicity variable to the at least one third node; link the at least one second node to the first node via at least one upstream acyclic directed edge; or link the at least one third node to the first node via at least one downstream acyclic directed edge.

24. The method of claim 23, further comprising: obtaining the model input data via a plurality of sources; and using domain knowledge to filter the model input data, wherein the domain knowledge comprises types of relationships between variables, the variables comprise probability distributions, and the method further comprises constructing the causal machine learning model by using the domain knowledge to assign weights to the probability distributions.

25. The method of claim 21, further comprising: constructing the causal machine learning model by fitting a hierarchical Bayesian inference model to the model input data, wherein the model input data comprises phenotype data and variant pathogenicity data.

26. An apparatus for variant interpretation or variant classification, comprising: at least one processor; and at least one memory, wherein the at least one memory comprises a causal machine learning model, wherein a graphical representation of the causal machine learning model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of the pathogenicity variable, at least one third node of the nodes represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes.

27. The apparatus of claim 26, wherein the at least one memory further comprises sub-models and each sub-model is capable of providing output for a corresponding node of the graphical representation of the causal machine learning model.

28. The apparatus of claim 27, wherein the at least one memory further comprises a model abstractions component capable of determining output types of the sub-models and specific interactions between the sub-models.

29. The apparatus of claim 26, wherein the apparatus further comprises at least one first device capable of receiving a request and providing output generated by the causal machine learning model in response to the request to at least one of a data store, a network, a user interface, or at least one device30. A method for interpreting or classifying a genetic variant, comprising: sampling a node of a causal machine learning model to obtain sampled data, wherein the node is associated with the genetic variant, and wherein a graphical representation of the causal machine learning model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity variable, at least one second node of the nodes represents a cause of the pathogenicity variable, at least one third node of the nodes represents an effect of the pathogenicity variable, and at least one acyclic directed edge represents a relationship between two of the nodes; and using the sampled data to determine and output an interpretation for the genetic variant.

31. The method of claim 30, wherein sampling the node further comprises: sampling a posterior predictive distribution associated with the node.

32. The method of claim 30, wherein the interpretation comprises a probability that the genetic variant is pathogenic for a health condition.

33. The method of claim 30, further comprising: receiving a request via a device; and providing the interpretation of the genetic variant to at least one of a data store, a network, or the device.

34. A system for modeling cause -and-effect relationships between pathogenicity evidence variables for a genetic variant and a health condition, comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises: a data pre-processing component to create input data for a causal machine learning model using pathogenicity evidence data associated with the genetic variant and the health condition, whereinthe pathogenicity evidence data is obtained via a plurality of evidence sources, wherein the plurality of evidence sources comprises at least one of: patient clinical data, population frequency data, published in vitro experimental study data, internally performed in vitro experimental study data, splice site loss predictions, splice site gain predictions, or predictions of nonsense mediated decay curated based on an application of a variant classification or variant interpretation framework; a model training component to apply the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable of the pathogenicity evidence variables related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and a prediction component to, in response to a request, output predictive data sampled from the trained causal model.

35. The system of claim 34, wherein the plurality of evidence sources comprises: patient clinical data evaluated through an evidence modeling platform (EMP), population frequency data evaluated through the EMP, published in vitro experimental study data evaluated through the EMP, internally performed in vitro experimental study data evaluated through the EMP), splice site loss predictions, splice site gain predictions, and predictions of nonsense mediated decay curated based on the application of the variant classification or variant interpretation framework.

36. A method comprising: creating input data for a causal machine learning model using pathogenicity evidence data associated with a genetic variant and a health condition, wherein the pathogenicity evidence data is obtained via a plurality of evidence sources, and the plurality of evidence sources comprises at least one of: patient clinical data, population frequency data, published in vitro experimental study data, internally performed in vitro experimental study data, splice site loss predictions, splice site gain predictions, or predictions of nonsense mediated decay curated based on an application of a variant classification or variant interpretation framework; applying the causal machine learning model to the input data to produce a trained causal model, wherein a graphical representation of the trained causal model comprises nodes connected via acyclic directed edges, a first node of the nodes represents a pathogenicity evidence variable related to the health condition, at least one second node of the nodes represents a cause of the pathogenicity evidence variable, at least one third node of the nodes represents an effect of the pathogenicity evidence variable, and an acyclic directed edge represents a relationship between two of the nodes; and in response to a request, outputting predictive data sampled from the trained causal model.

37. The method of claim 36, wherein the plurality of evidence sources comprises: patient clinical data evaluated through an evidence modeling platform (EMP), population frequency data evaluated through the EMP, published in vitro experimental study data evaluated through the EMP, internally performed in vitro experimental study data evaluated through the EMP), splice site loss predictions, splice site gain predictions, and predictions of nonsense mediated decay curated based on the application of the variant classification or variant interpretation framework.

38. An apparatus comprising: at least one processor; and at least one memory, wherein the at least one memory comprises: a plurality of machine learning-based evidence models comprising at least one of: a first model of patient clinical data, a second model of population frequency data, a third model of published in vitro experimental study data, a fourth model of in vitro experimental study data, a fifth model of splice site loss predictions, a sixth model of splice site gain predictions, or a seventh model of predictions of nonsense mediated decay; a model evaluation component to apply at least one evaluation criterion to output of one or more of the plurality of machine learning-based evidence models; a model calibration component to use output of the model evaluation component to determine and apply weight values to the output of one or more of the plurality of machine learning-based evidence models; and a prediction component to output the output of the model calibration component to at least one device, system, process, model, component, or application.

39. The apparatus of claim 38, wherein the at least one memory further comprises a model training component to train at least one model of the plurality of machine learning -based evidence models using supervised machine learning.

40. The apparatus of claim 39, wherein the model training component further trains the at least one model of the plurality of machine learning-based evidence models using gene-specific labeled data.

41. The apparatus of claim 38, wherein the plurality of machine learning-based evidence models comprises the first model of patient clinical data, the second model of population frequency data, the third model of published in vitro experimental study data, the fourth model of in vitro experimental study data, the fifth model of splice site loss predictions, a sixth model of splice site gain predictions, and the seventh model of predictions of nonsense mediated decay.

42. The apparatus of claim 38, wherein the model evaluation component is to at least one of: evaluate at least one first model of the plurality of machine learning-based evidence models individually orevaluate at least two second models of the plurality of machine learning-based evidence models in combination.

43. A method comprising: applying at least one evaluation criterion to output of one or more of a plurality of machine learning-based evidence models, the plurality of machine learning-based evidence models comprising at least one of: a first model of patient clinical data, a second model of population frequency data, a third model of published in vitro experimental study data, a fourth model of in vitro experimental study data, a fifth model of splice site loss predictions, a sixth model of splice site gain predictions, or a seventh model of predictions of nonsense mediated decay; determining one or more weight values for the output of the plurality of machine learning -based evidence models; applying the one or more weight values to the output of the plurality of machine learning-based evidence models; and in response to a request, outputting predictive data based on the applied one or more weight values.

44. The method of claim 43, further comprising training at least one model of the plurality of machine learning-based evidence models using supervised machine learning.

45. The method of claim 44, wherein the training further comprises training the at least one model of the plurality of machine learning -based evidence models using gene-specific labeled data.

46. The method of claim 43, wherein the plurality of machine learning -based evidence models comprises the first model of patient clinical data, the second model of population frequency data, the third model of published in vitro experimental study data, the fourth model of in vitro experimental study data, the fifth model of splice site loss predictions, a sixth model of splice site gain predictions, and the seventh model of predictions of nonsense mediated decay.

47. The method of claim 43, further comprising at least one of: evaluating at least one first model of the plurality of machine learning -based evidence models individually or evaluating at least two second models of the plurality of machine learning-based evidence models in combination.

Images