Calibration of pathogenicity scores from mutant pathogenicity machine learning models

Abstract

This disclosure describes a method, a non-temporal computer-readable medium, and a system that can identify temperature weights and apply those temperature weights to pathogenicity predictions for amino acid variants at specific protein locations to calibrate and improve the accuracy of such predictions. For example, in some cases, a variant pathogenicity machine learning model generates an initial pathogenicity score for a protein or target amino acid at a specific protein location based on the amino acid sequence of the protein. The disclosed system further identifies temperature weights that estimate the degree of certainty for the pathogenicity score output by the variant pathogenicity machine learning model. To generate such weights, the disclosed system can use a novel triangular attention neural network as the temperature prediction machine learning model. Based on the temperature weights and the initial pathogenicity score, the disclosed system generates a calibrated pathogenicity score for the target amino acid at a specific protein location.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims the benefit and priority of U.S. Patent Provisional Application No. 63 / 487,517, entitled “CALIBRATING PATHOGENCITY SCORES FROM A VARIANT PATHOGENCITY MACHINE-LEARNING MODEL,” and U.S. Patent Provisional Application No. 63 / 487,525, entitled “CALIBRATING PATHOGENCITY SCORES FROM A VARIANT PATHOGENCITY MACHINE-LEARNING MODEL,” both filed on 28 February 2023. The aforementioned applications are incorporated herein by reference in their entirety. [Background technology]

[0002] In recent years, biotechnology companies and research institutions have been improving software for predicting the pathogenicity of proteins or genetic variants. For example, some existing pathogenicity prediction models generate predictions that estimate the degree to which amino acid variants are benign or pathogenic. Such pathogenicity predictions can indicate whether amino acid variants may cause various diseases such as certain cancers, developmental disorders, or cardiac abnormalities. In addition to the inherent predictions of such predictions, biotechnology companies and research institutions have developed downstream applications for pathogenicity prediction. For example, pathogenicity predictions output by machine learning models have been used to identify target variants in population subsets for new drugs, as well as target variants that may be subject to gene editing.

[0003] While pathogenicity prediction models have demonstrated significant improvements in accuracy and downstream applications, existing models do not consistently produce accurate predictions across a range of different clinical benchmarks and cell line protocols. Such clinical benchmarks and cell line protocols may include, for example, Deciphering Developmental Disorders (DDD) studies, the United Kingdom (UK) Biobank, cell line experiments for saturated mutagenesis, Clinical Variants (ClinVar) from the National Library of Medicine, and scores for protein variants or benign proteins in data from Genomics England Variants (GELVar). A particular pathogenicity prediction model may produce accurate predictions for variants in the UK Biobank, but the same model may not accurately predict pathogenicity for specific interprotein benchmarks from DDD, for example.

[0004] To address the lack of consistency in cross-benchmarking, more complex pathogenicity prediction models have been developed in the form of transformer machine learning models, which have (i) a self-attention mechanism that processes sequential input data, and (ii) an ensemble of different pathogenicity prediction models that together generate combined or refined predictions. While such transformers have developed highly accurate pathogenicity predictions, in some cases, transformers may consume considerable computing power to generate predictions. Servers and other computing devices to train either transformers or various individual models as part of an ensemble of pathogenicity prediction models may also consume considerable computing power and time. By adding further layers to the architecture of such transformers or additional models, existing models can be improved in accuracy, but this may also further increase computing power.

[0005] To address inconsistencies and inaccuracies in other contexts, some machine learning models apply global temperature scaling to specific machine learning models outside the context of pathogenicity prediction. In such cases, a factor can scale the probabilities output by a particular machine learning model to compensate for inaccuracies. However, such existing temperature scaling factors target the global machine learning model or the entire evaluation dataset, and do not target more specific forms of input or output data. Furthermore, existing temperature scaling factors do not decompose the uncertainty of the global machine learning model from other more specific types of uncertainty.

[0006] These, along with further problems and challenges, exist in existing sequencing systems. [Overview of the project]

[0007] This disclosure describes one or more embodiments of systems, methods, and non-temporary computer-readable storage media that solve one or more of the above problems or provide other advantages over the art. In particular, the disclosed system can identify temperature weights and apply those temperature weights to pathogenicity predictions for amino acid variants at specific protein locations to calibrate and improve the accuracy of such predictions. For example, in some cases, a variant pathogenicity machine learning model generates an initial pathogenicity score for a protein or target amino acid at a specific protein location based on the amino acid sequence of the protein. The disclosed system further identifies temperature weights that estimate the degree of certainty for the pathogenicity score output by the variant pathogenicity machine learning model. To generate such weights, in some cases, the disclosed system uses a novel triangular attention neural network as the temperature prediction machine learning model. Based on the temperature weights and the initial pathogenicity score, the disclosed system generates a calibrated pathogenicity score for a target amino acid at a specific protein location.

[0008] In some cases, the disclosed system employs a unique training technique and loss function to train a temperature prediction machine learning model. For example, after generating a calibrated pathogenicity score for a target amino acid, the disclosed system determines the calibrated score difference between the calibrated pathogenicity score for known benign amino acids and the calibrated pathogenicity score for an unknown pathogenic amino acid. In some cases, the disclosed system uses a unique hybrid loss function to determine the training loss for training iterations. Based on the loss determined by such a hybrid loss function or another loss function, the disclosed system adjusts the parameters of the temperature prediction machine learning model to improve the predicted temperature weights.

[0009] Additional features and benefits of one or more embodiments of the present disclosure are described below, some of which may become apparent from the description or may be acquired by carrying out such exemplary embodiments. [Brief explanation of the drawing]

[0010] This patent or application file includes at least one drawing made in color. A copy of this patent or patent application publication containing the color drawing will be provided by the Office upon request and payment of the required fees.

[0011] The "modes for carrying out the invention" refer to the drawings briefly described below. [Figure 1] A schematic diagram of a computing system in which a calibrated pathogenicity prediction system can operate according to one or more embodiments of this disclosure is shown. [Figure 2] One or more embodiments of a calibrated pathogenicity prediction system are presented, which identifies initial pathogenicity scores for amino acid variants at specific protein locations and applies temperature weights to those initial pathogenicity scores in order to generate a calibrated pathogenicity score. [Figure 3A]A calibrated pathogenicity prediction system that generates temperature weights for target protein positions according to one or more embodiments of the present disclosure, applies a moving average (e.g., Gaussian blur) to the temperature weights, and combines the average temperature weights for the target protein positions with an initial pathogenicity score is shown. [Figure 3B] A calibrated pathogenicity prediction system that generates temperature weights for target protein positions according to one or more embodiments of the present disclosure, applies a moving average (e.g., Gaussian blur) to the temperature weights, and combines the average temperature weights for the target protein positions with an initial pathogenicity score is shown. [Figure 4A] A calibrated pathogenicity prediction system that uses a triangular attention neural network as a temperature prediction machine learning model for generating temperature weights for target protein positions according to one or more embodiments of the present disclosure is shown. [Figure 4B] A calibrated pathogenicity prediction system that uses a triangular attention neural network as a temperature prediction machine learning model for generating temperature weights for target protein positions according to one or more embodiments of the present disclosure is shown. [Figure 5] A calibrated pathogenicity prediction system that trains a temperature prediction machine learning model for generating temperature weights according to one or more embodiments of the present disclosure is shown. [Figure 6] A calibrated pathogenicity prediction system that combines temperature weights with an initial pathogenicity score from a variational autoencoder (VAE) as a mutant pathogenicity machine learning model to determine a calibrated pathogenicity score for a target amino acid at a target protein position within a protein according to one or more embodiments of the present disclosure is shown. [Figure 7] A calibrated pathogenicity prediction system that executes a meta-mutant pathogenicity machine learning model to generate a refined pathogenicity score for a target amino acid according to one or more embodiments of the present disclosure is shown. [Figure 8A]This disclosure provides a graphical visualization of protein position-specific temperature weights for pathogenicity scores output by a mutant pathogenicity machine learning model, according to one or more embodiments of this disclosure. [Figure 8B] This disclosure provides a graphical visualization of protein position-specific temperature weights for pathogenicity scores output by a mutant pathogenicity machine learning model, according to one or more embodiments of this disclosure. [Figure 8C] This disclosure provides a graphical visualization of protein position-specific temperature weights for pathogenicity scores output by a mutant pathogenicity machine learning model, according to one or more embodiments of this disclosure. [Figure 8D] This disclosure provides a graphical visualization of protein position-specific temperature weights for pathogenicity scores output by a mutant pathogenicity machine learning model, according to one or more embodiments of this disclosure. [Figure 8E] This disclosure provides a graphical visualization of protein position-specific temperature weights for pathogenicity scores output by a mutant pathogenicity machine learning model, according to one or more embodiments of this disclosure. [Figure 9] The bar graph shows the relative scores against performance metrics of different models that generate mutant pathogenicity predictions according to one or more embodiments of the present disclosure. [Figure 10] This disclosure describes the architecture, components, and various inputs and outputs of a transformer neural network from PrimateAI3D operating as a mutant pathogenicity machine learning model according to one or more embodiments of this disclosure. [Figure 11] This disclosure describes the architecture, components, and various inputs and outputs of a transformer neural network from PrimateAI3D operating as a mutant pathogenicity machine learning model according to one or more embodiments of this disclosure. [Figure 12] This disclosure describes the architecture, components, and various inputs and outputs of a transformer neural network from PrimateAI3D operating as a mutant pathogenicity machine learning model according to one or more embodiments of this disclosure. [Figure 13]One or more embodiments of this disclosure describe a series of operations for identifying an initial pathogenicity score for an amino acid variant at a specific protein location, and applying temperature weights to that initial pathogenicity score to generate a calibrated pathogenicity score. [Figure 14] This disclosure describes a series of operations for generating a graphical visualization of temperature weights for the location of a target protein within a target protein, according to one or more embodiments of this disclosure. [Figure 15] This disclosure describes a set of operations for training a temperature prediction machine learning model and generating temperature weights according to one or more embodiments of this disclosure. [Figure 16] The following are block diagrams illustrating exemplary computing devices according to one or more embodiments of this disclosure. [Modes for carrying out the invention]

[0012] This disclosure describes one or more embodiments of a calibrated pathogenicity prediction system that can generate temperature weights for amino acid variants at specific protein locations and apply these temperature weights to pathogenicity predictions output by a variant pathogenicity machine learning model. For example, in some cases, the calibrated pathogenicity prediction system runs a variant pathogenicity machine learning model to generate an initial pathogenicity score for a target amino acid at a specific protein location (or across specific protein locations) based on the amino acid sequence of the protein and the corresponding multiple sequence alignment (MSA) of that protein. The calibrated pathogenicity prediction system further identifies or generates temperature weights that estimate the degree of certainty for the pathogenicity score output by the variant pathogenicity machine learning model. To obtain such weights, in some cases, the calibrated pathogenicity prediction system uses a novel triangular attention neural network (or other model) as the temperature prediction machine learning model to output the temperature weights. By further combining the initial pathogenicity score and the temperature weights, in some cases, the calibrated pathogenicity prediction system generates a calibrated pathogenicity score for a target amino acid at a specific protein location.

[0013] As described above, the disclosed temperature weights may be protein-specific or protein-location-specific for a particular protein. In some cases, the disclosed temperature weights estimate the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model. Thus, the disclosed temperature weights can adjust for noise or other uncertainties caused by the mutant pathogenicity machine learning model itself or by the data input to the mutant pathogenicity machine learning model. In certain cases, the temperature weights estimate the degree of certainty for the pathogenicity score, but are designed not to affect the desired uncertainty of the mutant pathogenicity machine learning model caused by either evolutionary constraints (or pathogenicity constraints) of a given protein that allow for multiple variants at a particular protein location.

[0014] To identify temperature weights, a calibrated pathogenicity prediction system can access previously generated temperature weights for a protein or a specific protein location, or it can run a temperature prediction machine learning model. The temperature prediction machine learning model can take the form of various neural networks or other machine learning models, as further described below. For example, a temperature prediction machine learning model could include a multiplayer perceptron (MLP) that generates temperature weights for a target protein based on an initial pathogenicity score for that protein location and the amino acid sequence for that protein. In contrast, as described below, a temperature prediction machine learning model could include a triangular attention neural network having a triangular attention layer that processes residue pair representations of a particular protein based on new inputs and intermediate embeddings.

[0015] In addition to accessing or generating temperature weights, in some cases, a calibrated pathogenicity prediction system reduces weight noise by (i) deriving average temperature weights from initial temperature weights at the target protein location of a particular protein, and (ii) using the average temperature weights as temperature weights for the target protein location. For example, a calibrated pathogenicity prediction system may perform Gaussian blurring (or other moving averages) to determine average temperature weights for the target protein location based on initial temperature weights generated by a temperature prediction machine learning model for different amino acids at the target protein location. Such average temperature weights can then be applied to the initial pathogenicity score for mutant amino acids at the target protein location of a particular protein.

[0016] Since protein-specific temperature weights or protein location-specific temperature weights for pathogenicity scores can be identified here, in some embodiments, the calibrated pathogenicity prediction system generates graphics depicting temperature weights for specific proteins or protein locations within a protein. For example, the calibrated pathogenicity prediction system may generate graphics that include colors, patterns, or numerical values ​​representing the temperature weights determined for specific protein locations within a protein. Examples of such graphics are further illustrated and described below.

[0017] In addition to generating temperature weights and applying them to the pathogenicity score, in some embodiments, the calibrated pathogenicity prediction system uses meta-mutant pathogenicity machine learning models to refine and improve the accuracy of the pathogenicity score. For example, the calibrated pathogenicity prediction system can use a first mutant pathogenicity machine learning model and a second mutant pathogenicity machine learning model to generate a first initial pathogenicity score and a second pathogenicity score for a target amino acid in the protein at the target protein location, respectively. The calibrated pathogenicity prediction system can further combine calibrated versions (and / or uncalibrated versions) of the first and second pathogenicity scores to create a refined pathogenicity score for the target amino acid at the target protein location. As will be further described below, such meta-mutant pathogenicity machine learning models can combine pathogenicity scores from any number of mutant pathogenicity machine learning models and exhibit superior accuracy when the initial pathogenicity score is specific to a target amino acid rather than multiple amino acids at the protein location.

[0018] To train a temperature prediction machine learning model, in some cases, the calibrated pathogenicity prediction system employs specific training techniques and loss functions. For example, the calibrated pathogenicity prediction system uses a mutant pathogenicity machine learning model to determine an initial pathogenicity score for a target amino acid at a target protein location within a protein, based on the amino acid sequence of the protein. The calibrated pathogenicity prediction system further (i) employs a temperature prediction machine learning model to determine the temperature weights for the target amino acid at the target protein location, and (ii) generates a calibrated pathogenicity score based on the initial pathogenicity score and the temperature weights. The calibrated pathogenicity prediction system then determines the calibrated score difference between the calibrated pathogenicity score for known benign amino acids on the one hand and the calibrated pathogenicity score for unknown pathogenic amino acids on the other hand. Based on the loss determined by a hybrid loss function or another loss function, the calibrated pathogenicity prediction system adjusts the parameters of the temperature prediction machine learning model.

[0019] In some cases, a calibrated pathogenicity prediction system utilizes pathogenicity scores for known benign amino acids as a type of ground truth. To determine the calibration score difference, for example, a calibrated pathogenicity prediction system can determine the calibration score difference between (i) each of a first set of calibrated pathogenicity scores for known benign amino acids and (ii) each of a second set of calibrated pathogenicity scores for unknown pathogenic amino acids in different proteins or at different protein locations within the same protein.

[0020] As described above, in some cases, the disclosed system uses a unique hybrid loss function to determine the training loss. For example, when the calibrated score difference is greater than 0, the disclosed system uses the calibrated score difference as the loss for a given training iteration. In contrast, when the calibrated score difference is less than or equal to 0, the disclosed system determines the hyperbolic tangent of the calibrated score difference as the loss for a given training iteration. However, other training loss functions may also be employed.

[0021] As described above, calibrated pathogenicity prediction systems offer several technical advantages over existing pathogenicity prediction models. For example, calibrated pathogenicity prediction systems improve the accuracy and precision with which pathogenicity prediction models generate pathogenicity predictions for amino acid variants. As mentioned above, existing pathogenicity prediction models only produce raw or uncalibrated pathogenicity scores that cannot demonstrate consistent precision across a particular clinical or other benchmark. Unlike existing pathogenicity prediction models, calibrated pathogenicity prediction systems can generate temperature weights, which can be applied to the initial pathogenicity score for amino acid variants at specific protein locations. Because temperature weights are either protein-specific or protein-location-specific for a particular protein, unlike existing global scaling factors, the weights of the calibrated pathogenicity prediction system are adjusted for the uncertainty of pathogenicity scores output by variant pathogenicity machine learning models with customized precision for a particular protein or specific protein location. As illustrated and described herein, for example, the disclosed calibrated pathogenicity prediction system generates temperature weights that calibrate pathogenicity scores to exhibit consistent accuracy across clinical benchmarks and protocols not shown by existing pathogenicity prediction models, including pathogenicity scores that accurately predict pathogenicity for target amino acids across Developmental Disorders (DDD) studies, the UK Biobank, saturated mutagenesis, clinical variants (ClinVar), and Genomics England Variants (GELVar). As further depicted and demonstrated by the various tables and results reported below, in some cases, such calibrated pathogenicity scores perform better than uncalibrated pathogenicity scores in each of the aforementioned benchmarks and protocols.

[0022] In addition to improving accuracy and precision, in some embodiments, the calibrated pathogenicity prediction system generates graphics that existing models cannot and do not support, namely graphics depicting temperature weights for specific proteins or protein locations within proteins. As suggested above, existing temperature scaling factors cannot separate uncertainty for global machine learning models from other, more specific types of uncertainty. In contrast, in some embodiments, the calibrated pathogenicity prediction system identifies or generates temperature weights that estimate the degree of certainty for the pathogenicity score output by a variant pathogenicity machine learning model for a specific protein or target protein location within that particular protein. As a result, the calibrated pathogenicity prediction system can also generate graphic colors, patterns, or numerical values ​​representing the temperature weights determined for a specific protein or specific protein location within a protein, for display on a graphical user interface. As shown in the accompanying drawings, such graphical visualizations can provide a concise snapshot of the certainty or uncertainty associated with the pathogenicity score for a particular protein location. As will be further explained below, the graphical visualizations described and depicted in this disclosure represent a novel and unprecedented visualization that depicts model-induced or data-induced uncertainty for pathogenicity scores corresponding to specific locations, separate from (or independent of) the uncertainty caused by evolutionary constraints or pathogenicity constraints.

[0023] Furthermore, as described above, in some embodiments, the calibrated pathogenicity prediction system uses a Type I machine learning model as a temperature prediction machine learning model. Some existing models can predict three-dimensional protein structure based on the amino acid sequence of a protein. In contrast, this disclosure introduces a triangular attention neural network that determines temperature weights for pathogenicity scores corresponding to target protein locations based on an input representing a particular three-dimensional protein structure. For example, the triangular attention neural network processes, as its intrinsic inputs, an amino acid pairwise index difference embedding representing the differences between amino acids in the amino acid sequence of a protein, and an amino acid pairwise atomic distance matrix representing the pairwise distances between atoms in the protein. Unlike existing models, in some cases, the triangular attention neural network also extracts the diagonal residue pair representation of the protein from the modified residue pair representation of the protein. This disclosure describes additional intrinsic aspects of the novel triangular attention neural network, which are described below.

[0024] Beyond novel graphical visualizations or new networks, in some embodiments, calibrated pathogenicity prediction systems improve the computational efficiency by which pathogenicity prediction models adjust accurate pathogenicity scores for amino acid variants. As described above, existing pathogenicity prediction models have partially improved the accuracy of pathogenicity scores by adding neural network layers or more complex architectures designed for deep learning neural networks, such as transformer machine learning models. However, such additional layers or complex architectures increase both the number of calculations and computer processes performed by existing pathogenicity prediction models. Rather than adding layers or more complex architectures, in some embodiments, calibrated pathogenicity prediction systems efficiently improve the accuracy of pathogenicity scores by identifying temperature weights and applying those temperature weights to the initial pathogenicity score. For example, by accessing previously generated pathogenicity scores for target protein locations within a protein, calibrated pathogenicity prediction systems can quickly and simply improve the initial pathogenicity score using temperature weights that, when applied to the initial pathogenicity score, result in a calibrated pathogenicity score, without using more complex neural network layers.

[0025] As illustrated by the preceding discussion, this disclosure uses a variety of terms to describe the features and benefits of calibrated pathogenicity prediction decision systems. As used herein, for example, the term “machine learning model” refers to a computer algorithm or set of computer algorithms that automatically improves for a particular task through experience based on the use of data. For example, a machine learning model may utilize one or more learning techniques to improve accuracy and / or effectiveness. Exemplary machine learning models include various types of decision trees (e.g., gradient boosted trees), support vector machines, Bayesian networks, or neural networks (e.g., transformer neural networks, recurrent neural networks, triangular attention neural networks).

[0026] In some cases, a calibrated pathogenicity prediction system uses a variant pathogenicity machine learning model to generate, modify, or update pathogenicity scores for target amino acids. As used herein, the term “variant pathogenicity machine learning model” refers to a machine learning model that generates pathogenicity scores for either a protein (e.g., a protein variant) or an amino acid at a specific protein location within a protein. For example, a variant pathogenicity machine learning model includes a machine learning model that generates initial or uncalibrated pathogenicity scores for a variant amino acid at a target protein location within a protein, based on the amino acid sequence of that protein. In some cases, in addition to, or as part of, the amino acid sequence of a protein as input, the variant pathogenicity machine learning model processes other inputs, such as a multiple sequence alignment (MSA) corresponding to the protein or a reference amino acid sequence for that protein. As shown below, mutant pathogenicity machine learning models can take the form of different models, including but not limited to transformer machine learning models, convolutional neural networks (CNNs), sequence-to-sequence models, variational autoencoders (VAEs), multilayer perceptrons (MLPs), recurrent neural networks (RNNs), long short-term memory (LSTMs), or decision tree models.

[0027] In relation to this, as used herein, the term “pathogenicity score” refers to a measure, numerical value, or score indicating the degree to which a protein or an amino acid at a protein site within a protein is benign or pathogenic. In some cases, for example, the pathogenicity score includes a logit or other numerical value indicating the probability of a mutant amino acid at a target protein site in a protein relative to a reference amino acid in the target protein. Since the pathogenicity score can indicate that a particular amino acid at a protein site is benign, in some cases, the pathogenicity score represents the suitability of a particular amino acid at a protein site. As just one example of a pathogenicity score, in some embodiments, the target substituted amino acid (s) at the target protein site may be used. alt The pathogenicity score for ) is determined by the alternative amino acid (p) at the target protein site. alt Logits about ) and reference amino acids (p ref This includes a numerical value determined from the usual difference with the logit for ). Further details regarding this particular example can be found in U.S. Patent Application No. 17 / 975,547, “Pathogenicity Language Model,” filed October 27, 2022, by Tobias Hamp, Anastasia Dietrich, Yibing Wu, Jeffrey Ede, and Kai-How Farh, which is incorporated herein by reference in its entirety. However, other formulations of pathogenicity scores can be used similarly and are described below.

[0028] As suggested above, the term “calibrated pathogenicity score” refers to a pathogenicity score adjusted or modified to account for temperature in a mutant pathogenicity machine learning model. In particular, a calibrated pathogenicity score includes the initial pathogenicity score output by a mutant pathogenicity machine learning model adjusted with temperature weights. As described above, in some cases, the calibrated pathogenicity score is adjusted with temperature weights to account for or reflect the degree of certainty or uncertainty regarding the pathogenicity score output by a given mutant pathogenicity machine learning model.

[0029] In some cases, a calibrated pathogenicity prediction system uses a temperature prediction machine learning model to generate, modify, or update temperature weights. As used herein, the term “temperature prediction machine learning model” refers to a machine learning model that generates temperature weights for a protein or for any amino acids at specific protein locations within a protein. For example, a temperature prediction machine learning model includes a machine learning model that generates temperature weights that estimate the degree of certainty or uncertainty for a pathogenicity score output by a mutant pathogenicity machine learning model. A temperature prediction machine learning model can process a variety of inputs, including but not limited to initial pathogenicity scores, amino acid sequences, amino acid pairwise index difference embeddings, amino acid pairwise atomic distance matrices or multiple matrices, or other inputs described below. As shown below, a temperature prediction machine learning model can take the form of different models, including but not limited to multilayer perceptrons (MLPs), convolutional neural networks (CNNs), triangular attention neural networks, recurrent neural networks (RNNs), long-short-term memory (LSTMs), transformer machine learning models, or decision tree models.

[0030] In relation to this, as used herein, the term “temperature weight” refers to a factor or numerical value that estimates the degree of certainty or uncertainty regarding the pathogenicity score output by a mutant pathogenicity machine learning model. For example, a temperature weight may include a numerical value that estimates (and is designed to correct for) the certainty or uncertainty caused by the mutant pathogenicity machine learning model or the data input to that mutant pathogenicity machine learning model. As shown above, a temperature weight may be specific to a protein or specific to a location within a protein (e.g., a target protein location as described below). Thus, in some embodiments, a temperature weight estimates the degree of certainty or uncertainty regarding the pathogenicity score output by a mutant pathogenicity machine learning model, but is designed not to affect noise or other uncertainties caused by either evolutionary constraints or pathogenicity constraints of a given protein that allow for multiple mutants at a particular protein location. In some cases, as described below, the calibrated pathogenicity prediction system applies a nonlinear activation function to convert potentially positive or negative temperature weights into positive temperature weights before applying positive weights to the initial pathogenicity score.

[0031] As previously stated, temperature weights include factors or numerical values ​​that describe or reflect temperature. As used herein, the term “temperature” refers to a level or measure of certainty or uncertainty. In particular, temperature may include a level or measure of certainty or uncertainty with respect to the pathogenicity score determined by a mutant pathogenicity machine learning model. Thus, as shown above, temperature may be specific to the pathogenicity score output by the mutant pathogenicity machine learning model for a target amino acid at a target protein location within the protein.

[0032] Where used further herein, the term “target amino acid” refers to a specific type of amino acid. In particular, a target amino acid includes a specific alternative or variant residue in the amino acid sequence corresponding to a protein. Thus, as previously stated, a target amino acid may include a specific alternative or variant residue at a target protein position in the amino acid sequence. A target amino acid may also be any of the 20 amino acids that are part of proteins associated with living organisms, such as alanine, arginine, asparagine, aspartic acid, cysteine, etc.

[0033] In relation to this, as used herein, the term “target protein location” refers to a specific location or order of amino acids within the amino acid sequence that forms the polypeptide chain of a protein. In particular, a target protein location includes a numerically identified location of an amino acid within the ordered amino acid sequence representing a protein. For example, a target protein location may include the 7th, 54th, 195th, 200th, or any other numbered position within the amino acid sequence of amino acids representing a protein (e.g., a 300-amino acid sequence). In some cases, a target protein location may be expressed along or as a number within a residue sequence index (e.g., illustrated in the accompanying drawings).

[0034] As further shown above, in some embodiments, the calibrated pathogenicity prediction system trains a temperature prediction machine learning model using known benign amino acids and unknown pathogenic amino acids. As used herein, the term “known benign amino acids” refers to a particular type of amino acid that is unlikely to cause disease in an organism (e.g., with high confidence or high certainty). In particular, known benign amino acids include a particular type of amino acid at a target protein site within a protein that is known not to cause disease in humans or other primates. For example, amino acids labeled as known benign amino acids are benign with a probability of more than 95% (e.g., 95.8%) based on primate data. Thus, the term “potentially benign amino acids” may be used interchangeably with “known benign amino acids.” In contrast, the term “unknown pathogenic amino acids” refers to a particular type of amino acid for which it is unknown whether the type of amino acid causes disease in an organism. In particular, unknown pathogenic amino acids include a particular type of amino acid at a target protein site within a protein for which it is unknown whether the particular type of amino acid causes disease in humans or other primates.

[0035] The following paragraphs describe a calibrated pathogenicity prediction system with respect to exemplary embodiments and exemplary figures illustrating those embodiments. For example, Figure 1 shows a schematic diagram of a computing system 100 on which a calibrated pathogenicity prediction system 104 operates, according to one or more embodiments. As shown, the computing system 100 includes one or more server devices 102 connected via a network 116 to client devices 110 and therapeutic agent analyzers 114. While Figure 1 shows an embodiment of the calibrated pathogenicity prediction system 104, this disclosure describes the following alternative embodiments and configurations.

[0036] As shown in Figure 1, the server device 102, the client device 110, and the therapeutic agent analyzer 114 are connected via a network 116. Therefore, each component of the computing system 100 can communicate via the network 116. The network 116 includes any suitable network to which the computing devices can communicate. An exemplary network will be discussed in more detail below with respect to Figure 16.

[0037] As shown in Figure 1, the therapeutic analyzer 114 includes a device for analyzing amino acid sequences corresponding to protein and / or nucleotide sequences representing coding and non-coding genomic regions (and identifying candidate therapies). In some embodiments, the therapeutic analyzer 114 analyzes a set of amino acid sequences or a set of nucleotide sequences from a database containing samples exhibiting genetic diversity. From the analyzed set of amino acid sequences and / or nucleotide sequences, the therapeutic analyzer 114 can identify a subset of amino acid sequences and / or nucleotide sequences that show common variant amino acids or variant nucleotides. In combination with, or separately from, such variant identification, the therapeutic analyzer 114 can run a machine learning model (or other model) to identify coding or non-coding genomic regions that are intolerant to mutations and where the variants may cause loss or alteration of biological function. With respect to the identified subset of amino acid sequences and / or nucleotide sequences, in some cases, the therapeutic analyzer 114 identifies candidate biologics, drugs, or gene editing protocols for therapy.

[0038] In addition to, or as an alternative to, communicating via network 116, in some embodiments, the therapeutic agent analyzer 114 bypasses network 116 and communicates directly with server device 102 or client device 110. In addition, as shown in Figure 1, in one or more embodiments, the therapeutic agent analyzer 114 includes a calibrated pathogenicity prediction system 104.

[0039] As further shown in Figure 1, the server device 102 can generate, receive, analyze, store, and transmit digital data, such as data about amino acid sequences or nucleotide sequences. As shown in Figure 1, the therapeutic agent analyzer 114 can transmit (and the server device 102 can receive) various data from the therapeutic agent analyzer 114, including data representing amino acid sequences or nucleotide sequences. The server device 102 can also communicate with the client device 110. In particular, the server device 102 can transmit to the client device 110 data representing amino acid sequences or nucleotide sequences (or their variants), pathogenicity scores, or data for temperature weight graphics.

[0040] Furthermore, as shown in Figure 1, the server device 102 may include a calibrated pathogenicity prediction system 104. In one or more embodiments, as will be further described below, the calibrated pathogenicity prediction system 104 generates pathogenicity scores output by a mutant pathogenicity machine learning model 106 for amino acid variants at specific protein locations and applies temperature weights thereto. In connection with this, in some cases, the calibrated pathogenicity prediction system 104 may use a temperature prediction machine learning model 108 to generate such temperature weights. For example, for a specific version or haplotype of a gene and the corresponding protein, the calibrated pathogenicity prediction system 104 may run the mutant pathogenicity machine learning model 106 to determine an initial pathogenicity score for a target amino acid at a target protein location within the corresponding protein, run the mutant pathogenicity machine learning model 106 to determine temperature weights, and combine the initial pathogenicity score and temperature weights to generate a calibrated pathogenicity score for a target amino acid at a target protein location. Similarly, in one or more embodiments, the calibrated pathogenicity prediction system 104 trains a temperature prediction machine learning model 108 to generate temperature weights. In addition to identifying or generating such weights, the server device 102 can also transmit data representing the temperature weights to the client device 110 for graphical visualization. The figures shown herein and in the following paragraphs further illustrate such capabilities of the calibrated pathogenicity prediction system 104, the variant pathogenicity machine learning model 106, and / or the temperature prediction machine learning model 108.

[0041] In addition to, or instead of, running either or both of the mutant pathogenicity machine learning model 106 and the temperature prediction machine learning model 108, in some embodiments, the calibrated pathogenicity prediction system 104 accesses a database or table containing calibrated pathogenicity scores. For example, in a particular embodiment, the calibrated pathogenicity prediction system 104 identifies calibrated pathogenicity scores by identifying scores in a table for a specific protein, a target protein location, and a target amino acid at that location. Thus, such a table or database can organize calibrated pathogenicity scores according to the protein, location, and the target amino acid at that location. Consistent with the disclosures above and below, the table or database includes calibrated pathogenicity scores pre-calculated from a combination of temperature weights output by the temperature prediction machine learning model 108 and initial pathogenicity scores output by the mutant pathogenicity machine learning model 106.

[0042] In some embodiments, the server device 102 comprises a distributed collection of servers, where the server device 102 includes several server devices distributed across a network 116 and located in the same or different physical locations. Furthermore, the server device 102 may include content servers, application servers, communication servers, web hosting servers, or other types of servers.

[0043] In some cases, the server device 102 is located at or near the therapeutic agent analyzer 114, or at a distance from the therapeutic agent analyzer 114. In fact, in some embodiments, the server device 102 and the therapeutic agent analyzer 114 are integrated into the same computing device. The server device 102 can generate, receive, analyze, store, and transmit digital data by running software on the therapeutic agent analyzer 114 or the calibrated pathogenicity prediction system 104, such as by transmitting or receiving data representing amino acid sequences or nucleotide sequences (or variants thereof), pathogenicity scores, or temperature weights. In addition, or alternatively, in some embodiments, the therapeutic agent analyzer 114 or the calibrated pathogenicity prediction system 104 stores and accesses a database or table of pathogenicity scores or temperature weights corresponding to specific proteins and / or protein locations.

[0044] Furthermore, as illustrated and shown in Figure 1, the client device 110 can generate, store, receive, and transmit digital data. In particular, the client device 110 can receive data on amino acid sequences or nucleotide sequences (or their variants), pathogenicity scores, or temperature weights from the server device 102 and / or the therapeutic agent analyzer 114. Thus, the client device 110 can present data on temperature weights or pathogenicity scores to the user associated with the client device 110 within a graphical user interface.

[0045] The client device 110 shown in Figure 1 can include various types of client devices. For example, in some embodiments, the client device 110 includes non-mobile devices such as desktop computers or servers, or other types of client devices. In yet other embodiments, the client device 110 includes mobile devices such as laptops, tablets, mobile phones, or smartphones. Further details related to the client device 110 are described below with reference to Figure 16.

[0046] As further illustrated in Figure 1, the client device 110 includes an analysis application 112. This analysis application 112 may be a web application or a native application (e.g., a mobile application, a desktop application) stored on and executed on the client device 110. The analysis application 112 may include instructions that, when executed, cause the client device 110 to receive data from the calibrated pathogenicity prediction system 104 and to present data from the therapeutic agent analyzer 114 and / or the server device 102. Furthermore, the analysis application 112 may instruct the client device 110 to display data about pathogenicity scores or temperature weights, such as data for a graphical visualization of temperature weights by protein position for a two-dimensional or three-dimensional representation of proteins.

[0047] As further illustrated in Figure 1, the calibrated pathogenicity prediction system 104 may be located on the client device 110 as part of the analysis application 112, or on the therapeutic agent analyzer 114. Therefore, in some embodiments, the calibrated pathogenicity prediction system 104 is implemented on the client device 110 (e.g., entirely or partially located). As described above, in yet other embodiments, the calibrated pathogenicity prediction system 104 is implemented by one or more other components of the computing system 100, such as the therapeutic agent analyzer 114. In particular, the calibrated pathogenicity prediction system 104 can be implemented in various different ways across the server device 102, the network 116, the client device 110, and the therapeutic agent analyzer 114.

[0048] Figure 1 shows the components of the computing system 100 communicating via network 116, but in certain embodiments, the components of the computing system 100 can also communicate directly with each other, bypassing the network. For example, as mentioned earlier, in some embodiments, the client device 110 communicates directly with the therapeutic agent analyzer 114. Furthermore, in some embodiments, the client device 110 communicates directly with the calibrated pathogenicity prediction system 104. Furthermore, the calibrated pathogenicity prediction system 104 can access one or more databases housed in or accessed by the server device 102 or elsewhere within the computing system 100.

[0049] As described above, the calibrated pathogenicity prediction system 104 generates a calibrated pathogenicity score for the target amino acid at the target protein location. According to one or more embodiments, Figure 2 shows an overview of the calibrated pathogenicity prediction system 104 that generates such a calibrated pathogenicity score. As shown in Figure 2, the calibrated pathogenicity prediction system 104 runs a mutant pathogenicity machine learning model 206 to generate an initial pathogenicity score 208 for the target amino acid 200 at the target protein location within the protein and to identify temperature weights 216 for the same protein or the same target protein location. By combining the initial pathogenicity score 208 and the temperature weights 216, the calibrated pathogenicity prediction system 104 generates a calibrated pathogenicity score 218 for the target amino acid at the target amino acid location.

[0050] As shown previously, in some embodiments, the calibrated pathogenicity prediction system 104 runs a variant pathogenicity machine learning model 206 to generate an initial or uncalibrated pathogenicity score. As shown in Figure 2, for example, the calibrated pathogenicity prediction system 104 is supplied with data representing one or more of the following for a protein: a target amino acid sequence 202, a reference amino acid sequence 204, or a conserved multiple sequence alignment (MSA) 205 (and processed by the variant pathogenicity machine learning model 206). Some embodiments of the variant pathogenicity machine learning model 206 (e.g., a variational encoder) process the target amino acid sequence 202 and the reference amino acid sequence 204, while other embodiments of the variant pathogenicity machine learning model 206 (e.g., a transformer machine learning model) process a masked version of the reference amino acid sequence 204 and additional amino acid sequences masked in the conserved MSA 205. Thus, the target amino acid sequence 202, the reference amino acid sequence 204, and the conserved MSA 205 are shown in Figure 2 as example and candidate inputs.

[0051] Each candidate input encodes data from which the mutant pathogenicity machine learning model 206 extracts information for pathogenicity prediction. For example, as part of the target amino acid sequence 202, the target amino acid 200 is represented by a single-letter code (e.g., A) for a specific amino acid (e.g., alanine) at the target protein location. In some cases, the target amino acid 200 represents a mutant amino acid relative to a reference amino acid sequence 204 for a particular organism. As previously suggested, the reference amino acid sequence 204 represents a consensus or representative sequence of amino acids for a protein of a particular species (e.g., human). Thus, the reference amino acid sequence 204 constitutes the reference sequence of amino acids for the target amino acid sequence 202. In some cases, the conserved MSA 205 contains weights for each candidate amino acid at a given protein location, indicating the probability of a given amino acid at a given protein location based on the MSA. Therefore, a conserved MSA205 may include a position weight matrix (PWM), a position-specific weight matrix (PSWM), or a position-specific scoring matrix (PSSM) derived from the MSA corresponding to the protein, and may include alignments of amino acid sequences from different species (e.g., a conserved MSA for a group of primates). In this context, an MSA represents alignments of multiple amino acid sequences from related primates (e.g., 11 primates) or other organisms (e.g., 50 mammals, 99 vertebrates) for the same protein.

[0052] As previously shown, conserved MSAs constitute, or can take the form of, a position-weighted matrix (PWM), position-specific weighted matrix (PSWM), or position-specific scoring matrix (PSSM) derived from MSAs corresponding to proteins, and can include alignments of amino acid sequences from different species (e.g., conserved MSAs for a group of primates). In fact, although not shown in Figure 2, in some embodiments, the mutant pathogenicity machine learning model 206 processes multiple conserved MSAs (e.g., a first conserved MSA for a group of primates, a second conserved MSA for a group of mammals, and a third conserved MSA for a group of vertebrates).

[0053] For simplicity, Figure 2 shows the target amino acid sequence 202, the reference amino acid sequence 204, and the stored MSA 205 as candidate inputs to the mutant pathogenicity machine learning model 206. However, as shown below, the calibrated pathogenicity prediction system 104 can use other input types depending on the type of mutant pathogenicity machine learning model (e.g., transformer machine learning model, variational autoencoder). For example, in some embodiments, the mutant pathogenicity machine learning model 206 (or other mutant pathogenicity machine learning models described or illustrated herein) includes a unique transformer neural network, sometimes referred to as PrimateAI3D, as described in U.S. Patent Application No. 17 / 975,536, filed October 27, 2022, entitled “Mask Patten for Protein Language Models,” and U.S. Patent Application No. 17 / 975,547, filed October 27, 2022, entitled “Pathogenicity Language Model,” each of which is incorporated herein by reference in its entirety. Alternatively, in certain embodiments, the variant pathogenicity machine learning model 206 (or other variant pathogenicity machine learning models described or illustrated herein) includes a neural network that processes two-dimensional information about proteins, sometimes referred to as PrimateAI2D, as described in U.S. Patent Application No. 17 / 876,481, filed October 27, 2022, entitled "Transfer Learning-Based Use of Protein Contact Maps for Variant Pathogenicity Prediction" by Chen Chen, Hong Gao, Laksshman S. Sundaram, and Kai-How Farh.

[0054] Based on one or more of the target amino acid sequence 202, reference amino acid sequence 204, or conserved MSA 205 as candidate inputs, the mutant pathogenicity machine learning model 206 generates an initial pathogenicity score 208 for the target amino acid 200 at the target protein location. The initial pathogenicity score 208 indicates the degree to which the target amino acid 200 is benign or pathogenic to an organism when located at the target protein location within the protein. As shown in Figure 2, in some cases, the mutant pathogenicity machine learning model 206 generates initial pathogenicity scores for other target amino acids at the same or different target protein locations within the protein. Thus, the mutant pathogenicity machine learning model 206 can generate pathogenicity scores for different amino acid variants at different target protein locations.

[0055] Initial pathogenicity score 208 and other such initial pathogenicity scores are uncalibrated and tend to exhibit inconsistent accuracy across different benchmarks; therefore, initial pathogenicity score 208 may not accurately reflect the pathogenicity of target amino acid 200. In fact, initial pathogenicity scores output by the mutant pathogenicity machine learning model 206 may be inaccurate due to uncertainties in the mutant pathogenicity machine learning model 206 itself, or due to limitations in the data input to the mutant pathogenicity machine learning model 206.

[0056] To illustrate such initial or uncalibrated pathogenicity scores, in some embodiments, the mutant pathogenicity machine learning model 206 includes a transformer machine learning model (or other model) that outputs a logit indicating the probability that an organism (e.g., human) contains each of the 20 candidate amino acids at the target protein site. The true probability distribution of observing the 20 candidate amino acids can be expressed as the sum of the individual probabilities for each candidate amino acid, as shown by the function (1) below.

[0057]

number

[0058] However, rather than generating true probabilities, the initial or uncalibrated pathogenicity scores of the mutant pathogenicity machine learning model 206 are negatively affected by temperature (or measure of uncertainty) at each target protein location. For example, if a logit for a candidate amino acid is output by a transformer machine learning model that includes a softmax layer (e.g., trained using cross-entropy), it is unlikely to be entirely accurate because the logit is affected by the relative softmax temperature T > 1, where the softmax temperature is related to uncertainty caused solely by evolutionary constraints (or pathogenicity constraints) of a given protein that allow for multiple variants at a particular protein location. As shown by function (2) below, the softmax temperature T for a logit output by a transformer machine learning model (or other mutant pathogenicity machine learning model) fluctuates and affects the certainty of such a logit.

[0059]

number

[0060] According to function (2), the probability distribution p for observing candidate amino acids at the target protein location is proportional to the logit for each candidate amino acid at the target protein location, raised to an exponential value of 1 over the corresponding softmax temperature T. The closer the softmax temperature T is to 1, the lower the certainty for the logit at the target protein location. In contrast, when the softmax temperature T is equal to 1, the logit at the target protein location has no uncertainty except for the uncertainty caused by evolutionary or conservation constraints that the temperature weights are not designed to be measured or corrected. Conversely, as the softmax temperature T → ∞, or in other words, approaches infinity, the certainty for the logit at the target protein location increases accordingly. Because the softmax temperature T changes in accordance with the certainty of the mutant pathogenicity machine learning model 206, the initial or uncalibrated pathogenicity score also changes in accordance with the softmax temperature T. Therefore, such a softmax temperature T represents a kind of noise that negatively impacts the performance of the mutant pathogenicity machine learning model 206.

[0061] To compensate for or reduce the effect of softmax temperature T, the calibrated pathogenicity prediction system 104 can train a temperature prediction machine learning model 214 to predict temperature weights t, which represent specific temperatures for either the protein or the target protein location within the protein, as further described below. As shown by function (3) below, the model can represent how the predicted temperature weights t affect the logits output by the transformer machine learning model (or other mutant pathogenicity machine learning model) by multiplying the individual logits over the softmax temperature T by an exponent containing the predicted temperature weights t.

[0062]

number

[0063] However, if the temperature prediction machine learning model 214 generates temperature weights t proportional to the softmax temperature T, then the logit (or other initial pathogenicity score) output by the transformer machine learning model (or other mutant pathogenicity machine learning model) will have its effect of the softmax temperature T removed or reduced when the logit (or other initial pathogenicity score) is multiplied by the corresponding temperature weights t. As shown in function (4), if the temperature weights t approximately represent or coincide with the softmax temperature T such that t = kT, then the probability distribution for observing candidate amino acids at the target protein site can be expressed as a monotonic transformation of function (3), as follows:

[0064]

number

[0065] As represented by function (4), each logic takes the form of a monotonic transformation of a true logit indicating the probability that an organism (e.g., human) contains a target amino acid at the target protein site. Since the clinical benchmarks and cell line protocols measured in this disclosure are invariant to the monotonic transformation of the logit, this disclosure can assess the extent to which temperature weights t improve the accuracy of the initial pathogenicity score. As will be further explained below, temperature weights t do indeed improve the accuracy of the initial pathogenicity score across such clinical benchmarks and cell line protocols.

[0066] To correct for or reduce the effect of softmax temperature T in the initial pathogenicity score 208, the calibrated pathogenicity prediction system 104 identifies temperature weights 216 that predict the temperature of the mutant pathogenicity machine learning model 206, as further shown in Figure 2. To do so, the calibrated pathogenicity prediction system 104 either accesses temperature weights 216 previously generated by the temperature prediction machine learning model 214, or runs the temperature prediction machine learning model 214 to generate the temperature weights 216.

[0067] As shown in Figure 2, for example, the calibrated pathogenicity prediction system 104 is supplied with data representing amino acid sequences 210 for a protein and initial pathogenicity scores 212 corresponding to amino acids at different protein locations within the protein (and processed by the temperature prediction machine learning model 214). For overview, Figure 2 shows the amino acid sequences 210 and initial pathogenicity scores 212 as inputs to the temperature prediction machine learning model 214. However, as shown below, the calibrated pathogenicity prediction system 104 may use other input types depending on the type of temperature prediction machine learning model (e.g., triangular attention machine learning model, multilayer perceptron). Additional or alternative candidate inputs for the temperature prediction machine learning model 308 (such as amino acid pairwise index differences, amino acid pairwise atomic distances, and conserved profiles) are described below with respect to Figure 4A.

[0068] As suggested above, in some embodiments, the temperature weights 216 indicate the degree of uncertainty in the pathogenicity score when output by the mutant pathogenicity machine learning model 206 for either the protein or the target protein location. Such temperature weights generated by the temperature prediction machine learning model 214 may also be specific to a particular version of the mutant pathogenicity machine learning model 206, rather than simply being individual positive weights (e.g., temperature weights generated by a triangular attention neural network for pathogenicity scores output by a transformer machine learning model). As shown in Figure 2, in some cases, the temperature prediction machine learning model 214 generates temperature weights for multiple target protein locations for the same protein. In some such cases, the temperature prediction machine learning model 214 simultaneously generates location-specific temperature weights for the same protein. Thus, the temperature prediction machine learning model 214 can generate temperature weights for different target protein locations depending on the input and the type of machine learning model.

[0069] As further shown in Figure 2, the calibrated pathogenicity prediction system 104 combines the initial pathogenicity score 208 and the temperature weights 216 to generate a calibrated pathogenicity score 218. For example, in a particular embodiment, the calibrated pathogenicity prediction system 104 generates the calibrated pathogenicity score 218 by multiplying the initial pathogenicity score 208 and the temperature weights 216. Alternatively, the calibrated pathogenicity prediction system 104 may combine the initial pathogenicity score 208 and the temperature weights 216 by determining the mean, adding, subtracting, or performing another operation, depending on the form of the initial pathogenicity score 208 and the temperature weights 216.

[0070] Regardless of the calculation, the calibrated pathogenicity score 218 represents a modified version of the initial pathogenicity score 208, which more accurately indicates the degree to which the target amino acid 200 is benign or pathogenic to an organism when located at the target protein site within the protein. As further illustrated by Figure 2, in some cases, the calibrated pathogenicity prediction system 104 generates calibrated pathogenicity scores for other target amino acids at the same or different target protein sites within the protein. Thus, the calibrated pathogenicity prediction system 104 can generate calibrated pathogenicity scores for different amino acid variants at different target protein sites within the protein.

[0071] In addition to generating calibrated pathogenicity scores, or as an alternative thereto, in some embodiments, the calibrated pathogenicity prediction system 104 generates data for graphically visualizing temperature weights. As shown in Figure 2, for example, the calibrated pathogenicity prediction system 104 generates different temperature weights to estimate the temperature of the mutant pathogenicity machine learning model 206 for pathogenicity scores at different protein locations within the protein. In some cases, the calibrated pathogenicity prediction system 104 further provides the client device 110 with data representing a graphic 220 depicting the temperature weights across different protein locations. Based on this data, the client device 110 displays the graphic 220 to visualize the temperature of the mutant pathogenicity machine learning model 206 at different protein locations in the form of temperature weights. As described below, such graphics can take different and more complex forms.

[0072] As shown previously, in some cases, the calibrated pathogenicity prediction system 104 can generate temperature weights by running a temperature prediction machine learning model. According to one or more embodiments, Figures 3A and 3B show the calibrated pathogenicity prediction system 104 which runs a temperature prediction machine learning model to generate temperature weights for target protein locations, applies a moving average to the temperature weights, and combines the average temperature weights for target protein locations with the corresponding initial pathogenicity scores. As shown by Figure 3A, in some cases, the calibrated pathogenicity prediction system 104 determines temperature weights for target protein locations by (i) applying a nonlinear function to convert the initial temperature weights to positive temperature weights (or maintain the positive temperature weights), and (ii) determining the average temperature weights from the initial positive temperature weights at a given target protein location. As shown by Figure 3B, the calibrated pathogenicity prediction system 104 combines the average temperature weights for target protein locations with initial pathogenicity scores for different target amino acids at the target protein locations to improve accuracy and performance across clinical benchmarks.

[0073] The calibrated pathogenicity prediction system 104 can utilize various machine learning models as the temperature prediction machine learning model 308. For example, the temperature prediction machine learning model 308 may include, but is not limited to, a multilayer perceptron (MLP), a convolutional neural network (CNN), a triangular attention neural network, a recurrent neural network (RNN), a long-short-term memory (LSTM), a transformer machine learning model, or a decision tree. This disclosure describes the architecture and inputs for the following specific triangular attention neural network with respect to Figures 4A and 4B. However, to introduce less complex machine learning models, the following paragraphs describe the data inputs for an MLP or CNN as the temperature prediction machine learning model 308. In addition to the additional inputs described below with respect to Figures 4A and 4B, the same or similar data inputs shown in Figure 3A may be used similarly for the triangular attention neural network.

[0074] As shown in Figure 3A, for example, the calibrated pathogenicity prediction system 104 is supplied with data representing one or more amino acid sequences 302 for a protein and an initial pathogenicity score 304 for target amino acids at target protein locations within that protein (and processed by the temperature prediction machine learning model 308). Additional or alternative candidate inputs for the temperature prediction machine learning model 308 (such as amino acid pairwise index differences, amino acid pairwise atomic distances, and conserved profiles) are described below with respect to Figure 4A. For example, in some embodiments, the calibrated pathogenicity prediction system 104 determines an embedding representing the amino acid sequence 302, or the corresponding nucleotide sequence for the protein, and inputs it into the temperature prediction machine learning model 308. Such protein-specific embeddings can take the form of vectors representing single-letter codes for amino acids within the protein. For example, the calibrated pathogenicity prediction system 104 can generate trained vectors representing each amino acid at each location for a given protein, where the trained vectors for amino acids at protein locations are optimized or otherwise adjusted during training. In some cases, the calibrated pathogenicity prediction system 104 combines the amino acid sequence 302 and the initial pathogenicity score 304 to form a combined embedded input 306 for MLP or CNN.

[0075] For example, given an initial pathogenicity score represented as x for a protein corresponding to gene g, the calibrated pathogenicity prediction system 104 can infer temperature weights for x and g using an MLP or CNN. To run the MLP or CNN to determine the temperature weights w, the calibrated pathogenicity prediction system 104 can send or receive a call to infer temperature weights for the data representing protein x and gene g based on an embedded input defined as pathogenicity score for protein x + embedding for gene g. After the MLP or CNN has inferred the temperature weights, the calibrated pathogenicity prediction system 104 can further apply an exponential function to convert the negative or positive temperature weights w from the MLP or CNN into positive weights. For example, using Python syntax, the calibrated pathogenicity prediction system 104 can detect or use the command def infer_weights(self,x,g) based on an input represented as embedded_input=self.score_proj(x)+self.gene_embed(g). For example, when using an MLP, the temperature weights can be expressed as w=self.mlp(embedded_input) or w=self.cnn(embedded_input) according to Python syntax. In some embodiments, a nonlinearity such as torch.exp() is applied at the end of the temperature prediction machine learning model 308 to output positive weights. Thus, as either an MLP or a CNN, the temperature prediction machine learning model 308 can return temperature weights expressed again in Python syntax as w=self.infer_weights(x,g). In some embodiments, to calibrate the initial or uncalibrated pathogenicity score corresponding to the same protein, the calibrated pathogenicity prediction system 104 multiplies the temperature weight w by the initial pathogenicity score x.

[0076] As shown in Table 1 below, the calibrated pathogenicity prediction system 104 improves the accuracy of such pathogenicity scores across various clinical benchmarks by combining the initial pathogenicity score generated by the Transformer as a variant prediction machine learning model with temperature weights from MLP. As shown in Table 1, the calibrated pathogenicity score calibrated with MLP-based temperature weights more accurately identifies mutant amino acids that cause developmental disorders from the Developmental Disorder Elucidation (DDD) database and identifies control or benign amino acids that do not cause such developmental disorders better than the initial pathogenicity score. In particular, the DDDp values ​​in Table 1 demonstrate that the calibrated pathogenicity score distinguishes pathogenic amino acid variants from benign amino acid variants or standard reference residues better than the initial pathogenicity score. R for saturated mutagenesis in Table 1 2 As the values ​​indicate, a calibrated pathogenicity score with temperature weighting based on MLP can also more accurately identify cell lines that die or survive with mutant amino acids, for example, using a saturated mutagenesis protocol. Similarly, R for UK Biobank in Table 1 2 As the values ​​further indicate, calibrated pathogenicity scores with MLP-based temperature weighting also more accurately identify pathogenic amino acid variants associated with specific phenotypes represented in the UK Biobank (along with UKBB) than the initial pathogenicity score.

[0077] [Table 1]

[0078] Therefore, as further shown in Figure 3A, the temperature prediction machine learning model 308 generates temperature weights 310 based on the amino acid sequence 302 and the initial pathogenicity score 304. The temperature weights 310 estimate the temperature for the pathogenicity score output by the mutant pathogenicity machine learning model for either the protein or the target protein location within that protein. As shown in Figure 3A, in some embodiments, the calibrated pathogenicity prediction system 104 determines different temperature weights for mutant amino acids at different target protein locations. When a moving average 316 is applied as described below, in some cases, the calibrated pathogenicity prediction system 104 uses the temperature prediction machine learning model 308 to generate temperature weights for target amino acids at both the target protein location and adjacent protein locations adjacent to that target protein location.

[0079] In some cases, the calibrated pathogenicity prediction system 104 initially generates temperature weights with negative values. Therefore, the calibrated pathogenicity prediction system 104 optionally applies a nonlinear function 312 to convert the initial temperature weights 310, which have negative values, into positive temperature weights 314. For example, in certain embodiments, the calibrated pathogenicity prediction system 104 applies a soft plus activation function, an exponential activation function, an absolute value function, or other appropriate nonlinear function to the temperature weights 310 generated by the temperature prediction machine learning model 308. Therefore, in some cases, the temperature prediction machine learning model 308 has a final layer having a soft plus activation function, an exponential activation function, or an absolute value function.

[0080] As further shown in Figure 3A, the calibrated pathogenicity prediction system 104 determines the average temperature weight for a target protein location by applying a moving average 316 to the initial temperature weights corresponding to the target protein location. More generally, the calibrated pathogenicity prediction system 104 can apply a demonization filter (e.g., a bilateral filter) to the initial temperature weights. For a given target protein location, for example, the calibrated pathogenicity prediction system 104 determines the average temperature weight for the target protein location by determining the average of (i) the initial temperature weight for a given target protein location and (ii) the initial temperature weight for adjacent protein locations adjacent to the target protein location. Adjacent protein locations may be, for example, within a threshold number of adjacent protein locations from the target protein location (e.g., within 5, 10, or 15 locations). Since the temperature of the mutant pathogenicity machine learning model for a pathogenicity score at one protein location is often similar to such temperatures at adjacent protein locations, the moving average 316 can compensate for the noise of the individual temperature weights output by the temperature prediction machine learning model 308. As described above, in some embodiments, the calibrated pathogenicity prediction system 104 applies a Gaussian blurring model, median filter, or bilateral filter to the initial temperature weights for various amino acids at or corresponding to the target protein location to determine the average temperature weights for the target amino acids at the target protein location.

[0081] For example, as shown by the blurred graph 318 depicted in Figure 3A, the calibrated pathogenicity prediction system 104 uses a temperature prediction machine learning model 308 to generate temperature weights for different target protein locations, and further uses a Gaussian blurring model to determine the average temperature weights for different target protein locations. Since relatively close protein locations are more likely to show similar data than data for relatively distant protein locations, such a Gaussian blurring model places greater weight on temperature weights at protein locations relatively close to the target protein location. The blurred graph 318 shows the temperature weights along the y-axis over the enumerated target protein locations in the residue sequence index along the x-axis. As the blurred graph 318 shows, the initial temperature weights determined by the temperature prediction machine learning model 308 for different target protein locations are noisy without Gaussian blurring. In contrast, the average temperature weights determined by the calibrated pathogenicity prediction system 104 for different target protein locations are noise-free and (as shown in Figure 3B) improve performance across clinical benchmarks and cell line protocols.

[0082] As shown in Figure 3B, for example, the calibrated pathogenicity prediction system 104 applies mean temperature weights determined from Gaussian blurs of different sizes or adjacent position thresholds to the initial pathogenicity score. In particular, the calibrated pathogenicity prediction system 104 multiplies the mean temperature weight for the target protein location by the initial pathogenicity score for different target amino acids at the target protein location. As a result of combining the mean temperature weights and the initial pathogenicity score, the calibrated pathogenicity score improves in accuracy across various clinical benchmarks and cell line protocols as the size or adjacent position threshold for the Gaussian blur increases. However, as the size or adjacent position threshold for the Gaussian blur increases excessively, as shown by the second to last and last rows of Table 330 in Figure 3B, the calibrated pathogenicity score slightly decreases in accuracy for Gaussian blur sizes that include only more closely adjacent positions.

[0083] As shown in Figure 3B, the calibrated pathogenicity prediction system 104 multiplies each initial pathogenicity score at each protein location from a set of initial pathogenicity scores 322 by the corresponding average temperature weight for that protein location from a set of average temperature weights 320. For example, the calibrated pathogenicity prediction system 104 generates a calibrated pathogenicity score for each of the 20 different target amino acids at the first protein location by (i) multiplying each initial pathogenicity score 326a to 326n for 20 different target amino acids at the first protein location by (ii) the average temperature weight 324a for that protein location. Similarly, the calibrated pathogenicity prediction system 104 generates a calibrated pathogenicity score for each of the 20 different target amino acids at the nth protein position by (i) multiplying each initial pathogenicity score 328a to 328n for the nth protein position by the average temperature weight 324n for that protein position. Thus, in the illustrated embodiment, the calibrated pathogenicity prediction system 104 generates and applies a single average temperature weight for each protein position.

[0084] However, as shown above, the value for a given average temperature weight for a given protein location depends on the size or adjacent location threshold. As described above, Gaussian blurring (or other moving average models) can identify the initial temperature weight averaged for a single average temperature weight by considering a threshold number of adjacent protein locations (e.g., 5, 10, or 15 locations) that are different from the target protein location. Since the number of such thresholds for adjacent protein locations may differ, or the size of the Gaussian blurring may differ, the range and number of values ​​for adjacent temperature weights from adjacent protein locations will also differ from those for Gaussian blurring (or other moving average models).

[0085] As shown by Table 330 of FIG. 3B, for example, the calibrated pathogenicity prediction system 104 applies an average temperature weight determined from the size of the Gaussian blur gradually increasing or the threshold number of adjacent protein positions from the target protein position to a common set of initial pathogenicity scores. In the first row, Table 330 shows the DDDp value from the initial score or uncalibrated pathogenicity score, the R 2 value for saturation mutagenesis, and the R 2 value for the UK Biobank. As the size of the Gaussian blur or the adjacent position threshold increases up to a point, the calibrated pathogenicity score improves in accuracy across mutants as measured by (i) the p-value in distinguishing pathogenic amino acid mutants from benign amino acid variants or reference residues from DDD, (ii) the R 2 value for identifying cell lines that die or survive with mutant amino acids using saturation mutagenesis, and (iii) the R 2 value for identifying pathogenic amino acid mutants associated with a specific phenotype represented in the UK Biobank. However, as the size of the Gaussian blur or the adjacent position threshold in the case of the Gaussian blur increases excessively as shown by the change from log_num22 and blurred (8) in the second-to-last row of Table 330 of FIG. 3B to log_num24 and blurred (16) in the last row, the calibrated pathogenicity score is slightly less accurate in each of DDD, saturation mutagenesis, and UK Biobank measurements compared to a Gaussian blur size that includes only more closely adjacent positions.

[0086] Furthermore, as described above, in some embodiments, the calibrated pathogenicity prediction system 104 incorporates and uses a first type temperature prediction machine learning model. According to one or more embodiments, Figures 4A and 4B show the calibrated pathogenicity prediction system 104 using a triangular attention neural network 400 as a temperature prediction machine learning model for generating temperature weights for target protein locations. In summary, Figures 4A and 4B show that the calibrated pathogenicity prediction system 104 (i) supplies the triangular attention neural network 400 with data representing the amino acid sequence for the target protein, the amino acids of a reference protein or counterpart protein for the related organism, and the initial pathogenicity score for different amino acids at different protein locations within the protein, and (ii) generates temperature weights for different protein locations based on the data input.

[0087] As shown in Figure 4A, the calibrated pathogenicity prediction system 104 provides the triangular attention neural network 400 with a specific set of data inputs. This set of data inputs includes data representing one or more of the following: amino acid pairwise index differences 402, amino acid pairwise atomic distances 404, reference residues 406, conserved profiles 408, and initial pathogenicity scores 410. As described below, the triangular attention neural network 400 determines temperature weights for the initial pathogenicity scores corresponding to target protein locations based on one or more such data inputs representing a particular three-dimensional protein structure.

[0088] For example, as shown in Figure 4A, the amino acid pairwise index difference 402 represents the difference between amino acids in an amino acid sequence for a given protein. In particular, the amino acid pairwise index difference 402 includes a value that represents the difference or distance between specific amino acids in an amino acid sequence for a given protein, when such amino acids are represented by data as a sequence. For example, in an amino acid sequence represented by the single-letter code MATMC, the pairwise index difference between A for alanine and C for cysteine ​​is 3. Such an amino acid pairwise index difference represents a quantitative expression of the distance between specific amino acids in an amino acid sequence for a given protein. In some cases, the calibrated pathogenicity prediction system 104 limits or reduces the value of pairwise index differences that exceed a threshold of ±32, for example, by reducing any positive or negative difference value exceeding a threshold of ±32 to ±32 as the maximum positive or negative value.

[0089] In contrast, amino acid pairwise atomic distance 404 represents the pairwise distance between atoms in a given protein. In particular, amino acid pairwise atomic distance 404 includes Cα distance, which represents the physical distance between Cα carbon atoms in the amino acids that make up a given protein. In some cases, each Cα distance is determined as the logarithm of the Euclidean distance between Cα carbon atoms. For example, the calibrated pathogenicity prediction system 104 determines the logarithm of the Euclidean distance using the function log(x+c), where x represents the distance and c represents the offset value (e.g., 2). In some such cases, the calibrated pathogenicity prediction system 104 uses -1 for missing values ​​that are not part of the input data, because, for example, relatively small proteins are represented by data with filler values ​​to satisfy the model input size. Thus, the calibrated pathogenicity prediction system 104 can use an offset value of c=2 to ensure that the logarithm of a positive number is calculated and that not-of-number (e.g., NaN) is avoided. In some embodiments, each Cα distance can be determined by a local distance difference test (DDT). Since each amino acid contains a Cα atom that connects its amino chemical group to its acid carboxyl group, the amino acid pairwise atomic distance 404 can include the distance between each pair of amino acids in the sequence and can represent the backbone of the sequence. As an alternative to the pairwise Cα distance, in some cases, the amino acid pairwise atomic distance 404 can include the pairwise distance between heavy atoms measured by the logarithm of the Euclidean distance, DDT, or another suitable distance measure.

[0090] As further shown in Figure 4A, reference residue 406 represents a reference residue of a given protein to be targeted for analysis. For example, reference residue 406 contains a digitally represented reference amino acid assembled into a sequence for a given protein. In some embodiments, reference residue 406 contains acceptable values ​​for residues that constitute a standard protein amino acid sequence, where individual reference residues or reference amino acids are considered benign. For example, a reference residue may contain amino acids for a given protein encoded by a coding region for the corresponding gene from GRCh38 (or other versions of the reference genome) from the Genome Reference Consortium. Because reference residue 406 serves as a reference, the amino acid sequence constitutes a representative example of amino acids for a given organism, and a given amino acid in the sequence constitutes a representative example of amino acids at a given protein location for a given organism. Thus, reference residue 406 contains a reference residue corresponding to a target amino acid (e.g., a mutant amino acid) at a target protein location in a given protein. A target amino acid different from the reference residue or reference amino acid at the target protein location constitutes a mutant amino acid.

[0091] In connection with this, the conserved profile 408 contains data representing multiple sequence alignments (MSAs) or compressed versions of MSAs for a given protein from multiple species. For example, the conserved profile 408 contains data for three or more amino acid sequences from different species for the same given protein. These different species may include 50, 100, 150, or other appropriate number of related species relating to a common ancestor, such as 100 vertebrate species.

[0092] In certain embodiments, the saved profile 408 includes or is input to a triangular attention neural network 400 having learned weights for each species. As shown in the figure, in some embodiments, the saved profile 408 includes data representing a compressed version of such an MSA with learned weights. To compress the MSA, in some embodiments, the calibrated pathogenicity prediction system 104 identifies or determines, for each protein position in a given protein, (i) 20 candidate amino acids that occur in a species (e.g., 100 species), and (ii) the number of occurrences of each amino acid from a gap token representing a position that does not contain residues where the aligned non-human amino acid sequence aligns with the human amino acid sequence, and divides the number of occurrences for each amino acid by the number of species (e.g., 100). For (i) 20 candidate amino acids and (ii) one gap token, in some embodiments, the saved profile 408 considers 21 candidate values ​​per position, including values ​​proportional to each amino acid in the MSA sequence at a given position. Therefore, in the compressed version, the conserved profile 408 contains values ​​indicating the probability of each amino acid at a specific protein position across specific associations for a given protein. Thus, in some embodiments, the conserved profile 408 constitutes or takes the form of a position-weighted matrix (PWM), a position-specific weighted matrix (PSWM), or a position-specific scoring matrix (PSSM) derived from MSAs corresponding to proteins, and includes alignments of amino acid sequences from different species (e.g., conserved MSAs for a group of primates).

[0093] As described above, the initial pathogenicity score 410 represents the initial pathogenicity score generated by a mutant pathogenicity machine learning model for an amino acid at a given protein position in a given protein. In particular, the initial pathogenicity score 410 may be an uncalibrated pathogenicity score output by a mutant pathogenicity machine learning model (e.g., a transformer machine learning model) for each of the 20 candidate amino acids at each protein position in a given protein. Therefore, for each target protein position in a given protein, the initial pathogenicity score 410 includes multiple initial pathogenicity scores for different amino acids.

[0094] As further shown in Figure 4A, the triangular attention neural network 400 transforms the amino acid pairwise index difference 402 and the amino acid pairwise atomic distance 404 into the amino acid pairwise index difference embedding 422 and the amino acid pairwise atomic distance matrix 424, respectively. In particular, the embedding layer 412 of the triangular attention neural network 400 processes the data representing the amino acid pairwise index difference 402 and generates the amino acid pairwise index difference embedding 422. In some cases, the embedding layer 412 constitutes an embedding c layer that performs an embedding function that reflects the learned parameters to determine a particular output embedding. In contrast, the linear layer 414 of the triangular attention neural network 400 processes the data representing the amino acid pairwise atomic distance 404 and generates the amino acid pairwise atomic distance matrix 424. In some cases, the linear layer 414 constitutes a linear c layer that performs a linear projection.

[0095] In addition to transforming such structural information regarding residues and atomic distances of a given protein, as further shown in Figure 4A, the triangular attention neural network 400 transforms the reference residue 406, the conserved profile 408, and the initial pathogenicity score 410 into the reference residue embedding 426, the conserved multiple sequence alignment matrix 428, and the pathogenicity score matrix 430, respectively. In particular, the embedding layer 416 of the triangular attention neural network 400 processes the data representing the reference residue 406 and generates the reference residue embedding 426. In some cases, the embedding layer 416 determines a particular output embedding by configuring an embedding c / 2 layer that executes an embedding function that reflects the learned parameters. In contrast, the linear layer 418 of the triangular attention neural network 400 processes the data representing the conserved profile 408 and generates the conserved profile 408. Similarly, the linear layer 420 of the triangular attention neural network 400 processes the data representing the initial pathogenicity score 410 and generates the pathogenicity score matrix 430. In some cases, linear layers 418 and 420 constitute linear c / 2 layers that perform linear projections, respectively.

[0096] After generating such sequence-based and score-based outputs, the triangular attention neural network 400 supplies reference residue embeddings 426, conserved multiple sequence alignment matrices 428, and pathogenicity score matrices 430 through outer linking layers 432a, 432b, and 432c, as further shown in Figure 4A. By applying the outer linking layers 432a, 432b, or 432c to their respective inputs, the triangular attention neural network 400 performs an outer linking function to generate linked pairwise embeddings, or linked pairwise matrices that group the values ​​of pairs of amino acids. In some cases, the outer linking function involves linking features in the channel dimension (rather than multiplying such features together) to generate linked pairwise embeddings. Thus, such linked pairwise embeddings or linked pairwise matrices contain pairwise values ​​and dimensions similar to the amino acid pairwise index difference embeddings 422 and the amino acid pairwise atomic distance matrices 424. Therefore, the outer linking layers 432a, 432b, and 432c of the triangular attention neural network 400 transform the reference residue embedding 426, the conserved multiple sequence alignment matrix 428, and the pathogenicity score matrix 430 into a linked pairwise reference residue embedding 434, a linked pairwise conserved multiple sequence alignment matrix 436, and a linked pairwise pathogenicity score matrix 438, respectively. As a result of the outer linking, the total number of dimensions increases from the reference residue embedding 426, the conserved multiple sequence alignment matrix 428, and the pathogenicity score matrix 430 to the linked pairwise reference residue embedding 434, a linked pairwise conserved multiple sequence alignment matrix 436, and a linked pairwise pathogenicity score matrix 438, respectively.

[0097] As further shown in Figure 4A, the triangular attention neural network 400 concatenates various outputs to form an unfiltered residue pair representation 442. In particular, the triangular attention neural network 400 concatenates the amino acid pairwise index difference embedding 422, the amino acid pairwise atomic distance matrix 424, the concatenated pairwise reference residue embedding 434, the concatenated pairwise conserved multiple sequence alignment matrix 436, and the concatenated pairwise pathogenicity score matrix 438 to form an unfiltered residue pair representation 442. The unfiltered residue pair representation 442 constitutes a matrix that encodes the amino acid sequence, three-dimensional structure, and initial pathogenicity score corresponding to a given protein.

[0098] As further shown in Figure 4A, in some embodiments, the triangular attention neural network 400 optionally identifies and concatenates triangular position embeddings 440. The triangular attention neural network 400 optionally adds triangular position embeddings 440 to break the symmetry of the entire residue pair representation. Since the residue pair representations 450 (described further below) generated by the triangular attention neural network 400 can contain values ​​that replicate the representation of amino acid pairs, the use of triangular position embeddings 440 can reduce unnecessary computer computation by breaking symmetry. To break the symmetry of the residue pair representations 450, the calibrated pathogenicity prediction system 104 optionally adds different learning vectors for the upper triangle, diagonal, and lower triangles of the residue pair representations 450. In principle, the triangular position embeddings 440 enhance the triangular attention capability by enabling the triangular attention neural network 400 to process information from the upper and lower triangles of the residue pair representations 450 differently.

[0099] After generating unfiltered residue pair representations 442, the triangular attention neural network 400 further filters and refines this intermediate matrix. As shown in Figure 4A, for example, the triangular attention neural network 400 supplies the unfiltered residue pair representations 442 through layer normalization 444, tanh layers 446, and linear layers 448. By processing the unfiltered residue pair representations 442 through layer normalization 444, tanh layers 446, and linear layers 448, the triangular attention neural network 400 performs layer normalization to estimate normalized values ​​from the summed input of the unfiltered residue pair representations 442, performs a hyperbolic tangent function on the normalized values ​​of the unfiltered residue pair representations 442, and performs a linear function to linearly project the hyperbolic tangent of such normalized values.

[0100] After filtering the unfiltered residue pair representations 442 through layer normalization 444, tanh layer 446, and linear layer 448, the triangular attention neural network 400 generates residue pair representations 450. The residue pair representations 450 encode values ​​representing the pairwise relationships between residues (or amino acids) of a given protein. As shown by the various inputs described above, the residue pair representations 450 encode data representing amino acid index differences, physical distances between atoms of a given protein, reference residues for a given protein, conserved MSA corresponding to a given protein, and initial pathogenicity scores.

[0101] As shown in Figure 4B, the calibrated pathogenicity prediction system 104 continues to use the triangular attention neural network 400 to determine positive temperature weights for target locations within a given protein. In summary, the triangular attention neural network 400 (i) uses one or more triangular attention layers to generate modified residue pair representations 458, and (ii) determines diagonal residue pair representations 460 from the modified residue pair representations 458, and projects positive temperature weights 464 for protein locations within a given protein from the diagonal residue pair representations 460. By using triangular attention layers, the triangular attention neural network 400 uses layers with unlimited spatial resolution that can be applied to any protein subgraph.

[0102] As further shown in Figure 4B, for example, the triangular attention neural network 400 supplies residue pair representations 450 via a triangular update layer 452, an axial attention layer 454, and a transition layer 456. For example, in some embodiments, the triangular update layer 452 of the triangular attention neural network 400 may include a coupling layer that performs triangular multiplication updates on values ​​from the residue pair representations 450 by predicting the value of a given node (e.g., node i) based on two other nodes (e.g., nodes k and j) and corresponding edges (e.g., output edges or input edges). Furthermore, in certain embodiments, the axial attention layer 454 of the triangular attention neural network 400 may each supply values ​​from the residue pair representations 450 A coupling layer may be included that performs a triangular self-attention function on the residue pair representation 450. After the triangular update layer 452 and the axial attention layer 454, in some embodiments, a transition layer 456 normalizes the layer for the output of the axial attention layer 454 and applies the MLP to each position in the residue pair representation 450. As further shown by Figure 4B, in certain embodiments, the triangular attention neural network 400 processes the residue pair representation 450 multiple times (e.g., 10 times, 5 times, 3 times) through the triangular update layer 452, the axial attention layer 454, and the transition layer 456.

[0103] To implement triangular attention, the triangular update layer 452 and the axial attention layer 454 may include different layers that perform multiplicative updates or self-attention around different inputs. In some cases, to perform either triangular update or attention function, the triangular attention neural network 400 constructs or determines a triangular graph representing different parts of the residue pair representation 450, where three units from either a combination of two rows and one column, or a combination of one row and two columns, form three nodes connected by edges. For example, row i, column j, and row k from the residue pair representation 450 may each represent a node in the triangular graph. In the triangular graph containing nodes i, j, and k, the corresponding edges i~j, j~k, and i~k each represent an output edge, and the corresponding edges k~i, k~j, and j~k each represent an input edge.

[0104] As shown above, the triangular update layer 452 and the axial attention layer 454 utilize such triangular graphs to perform multiplicative updates or self-attention around different inputs. To perform a first triangular update, for example, one of the triangular update layers in the triangular update layer 452 performs a triangular multiplicative update using the output edge. To perform a second triangular update, a second triangular update layer in the triangular update layer 452 performs a triangular multiplicative update using the input edge. To perform a first triangular self-attention, the first triangular self-attention layer of the axial attention layer 454 performs a triangular self-attention around the start node.

[0105] In some embodiments, the calibrated pathogenicity prediction system 104 and the triangular attention neural network 400 utilize triangular renewal layers, axial attention (or self-attention) layers, transition layers, and corresponding supplementary information from John Jumper et al.'s "Highly Accurate Protein Structure Prediction with AlphaFold," 596 Nature 583-589 (2021) (hereinafter referred to as "Jumper"), both of which are incorporated herein by reference in their entirety.

[0106] Importantly, unlike Jumper, the calibrated pathogenicity prediction system 104 and the triangular attention neural network 400 use a triangular update layer, an axial attention (or self-attention) layer, and transition layers for different directions and different outputs. Rather than predicting three-dimensional protein structure based on the amino acid sequence of the protein and other information, the triangular attention neural network 400 uses such triangular update, axial attention, and transition layers to analyze residue pair representations as input representing specific three-dimensional protein structures and other information. Based on such analysis, the calibrated pathogenicity prediction system 104 uses the triangular attention neural network 400 to determine temperature weights for pathogenicity scores corresponding to target protein locations.

[0107] As shown in Figure 4B, after processing the residue pair representations 450 through one or more triangular attention layers, the triangular attention neural network 400 generates modified residue pair representations 458. In some cases, the modified residue pair representations 458 contain values ​​that mix information between the residue pairs and the compressed MSA information.

[0108] As further shown in Figure 4B, the triangular attention neural network 400 determines or extracts diagonal residue pair representations 460 from the modified residue pair representations 458. By determining the diagonal residue pair representations 460, in some embodiments, the triangular attention neural network 400 identifies features (e.g., feature vectors) representing a combination of amino acid index differences, physical distances between atoms of a given protein, reference residues for a given protein, a conserved MSA corresponding to a given protein, and an initial pathogenicity score for a target protein location. As shown in Figure 4B, in some embodiments, the diagonal residue pair representations 460 include units or values ​​from diagonal pathways across a matrix of modified residue pair representations 458.

[0109] From the diagonal residue pair representations 460, the triangular attention neural network 400 projects positive temperature weights 464. For example, in some embodiments, the triangular attention neural network 400 feeds the diagonal residue pair representations 460 through a linear layer 462 and linearly projects them onto the positive temperature weights 464. After projection, the positive temperature weights include positive temperature weights for each protein location within a given protein, where each positive temperature weight estimates the temperature or certainty of the pathogenicity score output by a mutant pathogenicity machine learning model at the target protein location.

[0110] As shown in Figures 4A and 4B, the calibrated pathogenicity prediction system 104 simultaneously calibrates all (or all) portions of the amino acid sequence for the target protein, minimizing computational requirements and providing temperature weight predictions based on data across multiple protein locations by running the triangular attention neural network 400 to process data representing (i) the amino acid sequence for the target protein, (ii) the amino acids of a reference or counterpart protein for a related organism, and (iii) the initial pathogenicity score for different amino acids at different protein locations. In contrast, in some cases, the calibrated pathogenicity prediction system 104 applies the triangular attention neural network to evaluate one amino acid variant at a time to generate positive temperature weights for amino acid variants at the center of a portion of the protein input to the triangular attention neural network (e.g., a target amino acid variant at the center of a portion or target amino acid sequence, or a target amino acid variant corresponding to a central initial pathogenicity score in the initial pathogenicity score). However, the accuracy or calibration quality of such temperature weights yields results comparable to the positive temperature weights 464 generated by the triangular attention neural network 400 shown in Figure 4B for multiple protein positions of the target protein.

[0111] As shown above, the calibrated pathogenicity prediction system 104 can use its own training techniques and hybrid loss functions to train a triangular attention neural network or other temperature prediction machine learning models. According to one or more embodiments, Figure 5 shows the calibrated pathogenicity prediction system 104 training a temperature prediction machine learning model 520 to generate temperature weights. By utilizing the hybrid loss function 530, in certain embodiments, the calibrated pathogenicity prediction system 104 learns to adjust the parameters of the temperature prediction machine learning model 520 to generate temperature weights that facilitate the distinction between benign mutant amino acids and unknown pathogenic mutant amino acids at a given protein location. By implementing a trained version of the temperature prediction machine learning model 520, the calibrated pathogenicity prediction system 104 generates a calibrated pathogenicity score that facilitates the distinction between benign mutant amino acids and pathogenic mutant amino acids.

[0112] As outlined in Figure 5, the calibrated pathogenicity prediction system 104 uses a mutant pathogenicity machine learning model 510 (e.g., a transformer machine learning model from PrimateAI3D) to determine initial pathogenicity scores for known benign amino acids 502 and unknown pathogenic amino acids 504 at the target protein location within the protein, based on the protein's reference residue 506 and conserved MSA 508. The calibrated pathogenicity prediction system 104 further uses a temperature prediction machine learning model 520 to determine temperature weights for the target amino acids at the target protein location. Based on the combination of known benign initial pathogenicity scores and temperature weights, as well as the combination of unknown initial pathogenicity scores, the calibrated pathogenicity prediction system 104 generates calibrated pathogenicity scores for known benign amino acids 502 and / or unknown pathogenic amino acids 504. In some cases, the calibrated pathogenicity prediction system 104 generates calibrated pathogenicity scores only for variants within known benign amino acids 502 or unknown pathogenic amino acids 504 that differ from the reference amino acid sequence due to a single nucleotide change. The calibrated pathogenicity prediction system 104 then determines the calibrated score difference between the calibrated pathogenicity score for known benign amino acids 502 and the calibrated pathogenicity score for unknown pathogenic amino acids 504. Based on the loss determined by the hybrid loss function 530, the calibrated pathogenicity prediction system 104 adjusts the parameters of the temperature prediction machine learning model 520.

[0113] As further shown in Figure 5, in a given iteration, the calibrated pathogenicity prediction system 104 inputs data representing amino acid sequences from either known benign amino acids 502 or unknown pathogenic amino acids 504 into the mutant pathogenicity machine learning model 510. As shown above, known benign amino acids 502 include certain types of amino acids that, when located at the target protein site, are with some certainty (e.g., >95%) unlikely to cause disease in an organism (e.g., human) (or are likely to be benign). Such amino acids constituting known benign amino acids 502 may correspond to or include a “benign” data indicator or “benign” label. As suggested above, in some embodiments, known benign amino acids 502 include only mutants (e.g., missense mutants, nonsense mutants) that differ from the reference amino acid sequence due to a single nucleotide change. In contrast, unknown pathogenic amino acids 504 include certain types of amino acids for which it is unknown whether those particular types of amino acids cause disease in an organism (e.g., human) when located at the target protein site. Such amino acids constituting the unknown pathogenic amino acid 504 may correspond to or include an “unknown pathogenicity” data indicator or “unknown pathogenicity” label. As suggested above, in some embodiments, the unknown pathogenic amino acid 504 includes only variants different from the reference amino acid sequence resulting from a single nucleotide change (e.g., missense variant, nonsense variant). Depending on the target protein location, the calibrated pathogenicity prediction system 104 may be input with data representing the known benign amino acid 502 and the unknown pathogenic amino acid 504 as part of the same or different amino acid sequences corresponding to a given protein. In some embodiments of training, the calibrated pathogenicity prediction system 104 inputs the known benign amino acid 502 and the unknown pathogenic amino acid 504 as part of data representing amino acid sequences for different proteins. For ease of explanation, Figure 5 shows the known benign amino acid 502 and the unknown pathogenic amino acid 504 separately.

[0114] In addition to inputting a known benign amino acid 502 for or in place of an unknown pathogenic amino acid 504, in some embodiments, the calibrated pathogenicity prediction system 104 inputs additional data into the mutant pathogenicity machine learning model 510 to generate an initial pathogenicity score for the unknown pathogenic amino acid 504. For example, the calibrated pathogenicity prediction system 104 optionally inputs data representing a reference residue 506 and a conserved multiple sequence alignment (MSA) 508 corresponding to a given protein into the mutant pathogenicity machine learning model 510. However, depending on the type of machine learning model used for the mutant pathogenicity machine learning model 510, the calibrated pathogenicity prediction system 104 may supply other data inputs in addition to, or instead of, the reference residue 506 and the conserved MSA 508.

[0115] As further shown above, in some training iterations, the calibrated pathogenicity prediction system 104 improves the training results by varying the data input, which represents data randomly selected from different proteins or locations. Batches for such training iterations may include, for example, data randomly sampled from multiple human proteins, and locations within human proteins. For example, in the first set of training iterations, the calibrated pathogenicity prediction system 104 inputs an amino acid sequence for a first protein, including known benign and unknown pathogenic amino acids, a reference residue for the first protein, and a conserved MSA for the first protein into the mutant pathogenicity machine learning model 510. In contrast, in the second set of training iterations, the calibrated pathogenicity prediction system 104 inputs an amino acid sequence for a second protein, including known benign and unknown pathogenic amino acids, a reference residue for the second protein, and a conserved MSA for the second protein into the mutant pathogenicity machine learning model 510. The calibrated pathogenicity prediction system 104 can also continue to input data into additional protein-related mutant pathogenicity machine learning models 510 as part of training a temperature prediction machine learning model 520, as will be further described below.

[0116] For example, in some embodiments, the calibrated pathogenicity prediction system 104 randomly samples data from multiple proteins and locations within those proteins in each training iteration. In a given training iteration, the calibrated pathogenicity prediction system 104 may randomly sample data from the same or different proteins for another training iteration (e.g., the immediate preceding or immediate following one). Further example, in some cases, the calibrated pathogenicity prediction system 104 randomly samples data so that every location in every protein is sampled before the calibrated pathogenicity prediction system 104 samples data again from the same location in a given protein. However, the calibrated pathogenicity prediction system 104 may further or alternatively input data from multiple different random samples from the same or different proteins within the same batch in a training iteration.

[0117] Based on data representing amino acid sequences including known benign amino acids 502 and unknown pathogenic amino acids 504, and / or other data inputs, the mutant pathogenicity machine learning model 510 generates a set of initial pathogenicity scores for known benign amino acids 502 and a set of initial pathogenicity scores for unknown pathogenic amino acids 504. As shown in Figure 5, for example, the mutant pathogenicity machine learning model 510 generates known amino acid initial pathogenicity scores 512 and unknown amino acid initial pathogenicity scores 514. In some embodiments, each initial pathogenicity score from known amino acid initial pathogenicity scores 512 and unknown amino acid initial pathogenicity scores 514 indicates the degree to which a known benign or unknown pathogenic amino acid at a protein location within a protein is benign or pathogenic. As described above, in some embodiments, the mutant pathogenicity machine learning model 510 performs training iterations including amino acid sequences for different proteins, and thus generates known amino acid initial pathogenicity scores 512 and unknown amino acid initial pathogenicity scores 514 for amino acids at different target protein locations within different proteins.

[0118] As further shown in Figure 5, the calibrated pathogenicity prediction system 104 uses a temperature prediction machine learning model 520 to generate temperature weights 522, which are combined with known and unknown amino acid initial pathogenicity scores 512 and unknown amino acid initial pathogenicity scores 514. As part of such a process, the calibrated pathogenicity prediction system 104 supplies data representing amino acid sequences 516 for a given target protein, initial pathogenicity scores 518 for target amino acids at target protein locations within a given target protein, and / or other data inputs. In some embodiments, when the temperature prediction machine learning model 520 includes a triangular attention neural network, the calibrated pathogenicity prediction system 104 inputs the same type of data input as shown in Figure 4A for the triangular attention neural network 400 as the temperature prediction machine learning model. In contrast, in certain embodiments, the calibrated pathogenicity prediction system 104 inputs the same type of data input as shown in Figure 3A for the MLP or CNN as the temperature prediction machine learning model.

[0119] As further shown above, in several training iterations, the calibrated pathogenicity prediction system 104 improves the training results by varying the data input for different proteins. For example, in the first set of training iterations, the calibrated pathogenicity prediction system 104 inputs data representing the amino acid sequence for a first protein, an initial pathogenicity score for the target amino acid at the target protein location within the first protein, and / or other specific inputs for the first protein into the temperature prediction machine learning model 520. In contrast, in the second set of training iterations, the calibrated pathogenicity prediction system 104 inputs data representing the amino acid sequence for a second protein, an initial pathogenicity score for the target amino acid at the target protein location within the second protein, and / or other specific inputs for the second protein into the temperature prediction machine learning model 520. The calibrated pathogenicity prediction system 104 can similarly continue to input data related to additional proteins into the temperature prediction machine learning model 520 as part of training the temperature prediction machine learning model 520, as will be further described below.

[0120] Based on data representing amino acid sequences 516, initial pathogenicity scores 518, and / or other inputs, the temperature prediction machine learning model 520 generates temperature weights 522, as further shown in Figure 5. As shown here, the temperature weights 522 estimate the certainty of each of the initial pathogenicity scores generated by the mutant pathogenicity machine learning model 510. As shown above, the temperature weights from the temperature weights 522 may be specific to a given protein or specific to the location of a target protein within the protein. Similarly, the temperature prediction machine learning model 520 can generate temperature weights based on data inputs using methods and models shown in Figure 3A or Figures 4A and 4B, such as MLPs, CNNs, or triangular attention neural networks. Since batches of input may contain data for different proteins, in some embodiments, the temperature prediction machine learning model 520 generates a first set of temperature weights for a first protein, a second set of temperature weights for a second protein, and / or additional sets of temperature weights for additional proteins.

[0121] As further shown in Figure 5, the calibrated pathogenicity prediction system 104 generates a calibrated pathogenicity score by combining the temperature weights 522 with the initial pathogenicity score. For example, the calibrated pathogenicity prediction system 104 generates a known amino acid calibrated pathogenicity score by multiplying the temperature weights from the temperature weights 522 for the target protein location by the known amino acid initial pathogenicity score from the known amino acid initial pathogenicity score 512. Similarly, the calibrated pathogenicity prediction system 104 generates an unknown amino acid calibrated pathogenicity score by multiplying the temperature weights from the temperature weights 522 for the target protein location by the unknown amino acid initial pathogenicity score from the unknown amino acid initial pathogenicity score 514.

[0122] By multiplying the respective temperature weights and initial pathogenicity scores for target variants at the target protein location, the calibrated pathogenicity prediction system 104 generates known amino acid calibrated pathogenicity scores 524 for known benign amino acids 502 at the target protein location and unknown amino acid calibrated pathogenicity scores 526 for unknown pathogenic amino acids 504 at the target protein location. As described above, in some embodiments, the calibrated pathogenicity prediction system 104 performs training iterations including temperature weights and initial pathogenicity scores for different proteins, and thus generates known amino acid calibrated pathogenicity scores 524 and unknown amino acid calibrated pathogenicity scores 526 for amino acids at different target protein locations within different proteins.

[0123] As further shown in Figure 5, the calibrated pathogenicity prediction system 104 determines the calibrated score difference 528 based on comparing individual scores from known amino acid calibrated pathogenicity scores 524 and unknown amino acid calibrated pathogenicity scores 526. For example, in some embodiments, the calibrated pathogenicity prediction system determines the calibrated score difference between each of the known amino acid calibrated pathogenicity scores 524 for known benign amino acids 502 and each of the unknown amino acid calibrated pathogenicity scores 526 for unknown pathogenic amino acids 504.

[0124] As described above, the calibrated pathogenicity prediction system 104 can determine the calibrated score difference 528 by comparing known amino acid calibrated pathogenicity scores with unknown amino acid calibrated pathogenicity scores for the same or different proteins. For example, in some embodiments, the calibrated pathogenicity prediction system 104 determines the calibrated score difference 528 between (i) known amino acid calibrated pathogenicity scores 524 for known benign amino acids 502 in a set of protein positions within a set of proteins, and (ii) unknown amino acid calibrated pathogenicity scores 526 for unknown pathogenic amino acids 504 in a set of protein positions within a set of proteins. Thus, the calibrated score difference 528 may include the difference between calibrated pathogenicity scores for target amino acids at target protein positions within different proteins.

[0125] Based on the calibrated score difference 528, the calibrated pathogenicity prediction system 104 determines the training loss by executing a hybrid loss function 530. When executing the hybrid loss function 530, in some embodiments, the training loss depends on whether the calibrated score difference between a known amino acid calibrated pathogenicity score and an unknown amino acid calibrated pathogenicity score is greater than or equal to zero. If the calibrated score difference is greater than zero, the calibrated pathogenicity prediction system 104 determines or uses the calibrated score difference as the loss according to the hybrid loss function 530. In contrast, if the calibrated score difference is less than or equal to zero, the calibrated pathogenicity prediction system 104 determines or uses the hyperbolic tangent of the calibrated score difference as the loss according to the hybrid loss function 530.

[0126] As shown in Figure 5, the hybrid loss function 530 can be expressed as a function of x if x > 0. Otherwise, if x ≤ 0, then y = tanh(x). In other words, if a given calibrated score difference x is greater than 0, the loss y for the hybrid loss function 530 is equal to the calibrated score difference x. However, if a given calibrated score difference is 0 or less, the loss y for the hybrid loss function 530 is equal to tanh(x).

[0127] Based on the loss determined from the hybrid loss function 530, the calibrated pathogenicity prediction system 104 modifies the parameters (e.g., network parameters) of the temperature prediction machine learning model 520. By tuning the parameters over training iterations, the calibrated pathogenicity prediction system 104 improves the accuracy of the temperature prediction machine learning model 520 in determining temperature weights that facilitate the distinction between benign and pathogenic mutant amino acids at a given protein location when incorporated into the calibrated pathogenicity score. Based on the loss determined from the hybrid loss function 530, the calibrated pathogenicity prediction system 104 determines the gradients over the weights using, for example, a layer-wise adaptive moment optimizer such as the Layer-wise Adaptive Moment optimizer for Batch training (LAMB), or NVIDIA's implementation of LAMB (NVLAMB) with adaptive learning rates, such as NVLAMB, which is available in "Pretraining BERT with Layer-wise Adaptive Learning Rates" by Sharath Sreenivas et al., NVIDIA Developer Technical Blog (December 5, 2019) https: / / developer.nvidia.com / blog / pretraining-bert-with-layer-wise-adaptive-learning-rates / , which is incorporated herein by reference in its entirety. Alternatively, the calibrated pathogenicity prediction system 104 uses stochastic gradient descent (SGD) to determine the gradient with respect to the weights. In some cases, the calibrated pathogenicity prediction system 104 uses the following function:

[0128]

number

[0129] Following initial training iterations and parameter adjustments, the calibrated pathogenicity prediction system 104 further determines known amino acid calibrated pathogenicity scores and unknown amino acid calibrated pathogenicity scores, as well as the corresponding calibrated score differences, as further shown in Figure 5. Based on further determination of losses from the calibrated score differences and the hybrid loss function 530 in additional training iterations, the calibrated pathogenicity prediction system 104 determines the losses and further adjusts the parameters of the temperature prediction machine learning model 520. In some cases, the calibrated pathogenicity prediction system 104 runs training iterations until the parameters (e.g., values ​​or weights) of the temperature prediction machine learning model 520 no longer change significantly over training iterations, or otherwise until the convergence criteria are met.

[0130] Regardless of the specific training embodiment of the temperature prediction machine learning model, the calibrated pathogenicity prediction system 104 can use different models as mutant pathogenicity machine learning models and can calibrate different forms of pathogenicity scores. According to one or more embodiments, Figure 6 shows the calibrated pathogenicity prediction system 104, which (i) uses a variational autoencoder (VAE) as a mutant pathogenicity machine learning model 604 to determine an initial pathogenicity score for a target amino acid at a target protein location within a protein, (ii) uses a temperature prediction machine learning model 614 to determine temperature weights for the protein, and (iii) determines a calibrated pathogenicity score based on the initial pathogenicity score from the VAE and temperature weights. Although the VAE is different from other models previously used as mutant pathogenicity machine learning models, nevertheless, the calibrated pathogenicity prediction system 104 can predict the pathogenicity of a target amino acid at a target protein location by determining and applying temperature weights that improve the accuracy of the initial pathogenicity score from the VAE.

[0131] To determine the initial pathogenicity score using VAE, the calibrated pathogenicity prediction system 104 can apply some of the functions and assumptions of the VAE as described by Adam J. Riesselman et al., "Deep Generative Models of Genetic Variation Capture the Effects of Mutations," 15 Nat. Methods 816-822 (2018) (hereinafter, Riesselman), the details of which are incorporated herein by reference in their entirety. Unlike Riesselman, and in improvements to Riesselman, as described below, the calibrated pathogenicity prediction system 104 can improve the accuracy of the initial pathogenicity score by (i) determining the difference between the lower limits of the first and second mutant amino acid sequences as a substitute for the initial pathogenicity score for the first mutant amino acid sequence, and (ii) applying temperature weights from a temperature prediction machine learning model. The following paragraphs describe various functions for describing the VAE, but Table 5 and the corresponding explanation demonstrate that the temperature weights of the calibrated pathogenicity prediction system 104 significantly improve the accuracy and performance of pathogenicity scores output by the VAE across clinical benchmarks and cell line protocols.

[0132] By modifying Riesselman's approach, the calibrated pathogenicity prediction system 104 can model the evolutionary process as a sequence generator for amino acid sequences, where such a sequence generator generates an amino acid sequence x using probability p(x|θ) and parameter θ. By using such probabilities that the model assigns functional or evolutionary constraints to the amino acid sequence x, the following function (5) is obtained, as follows, for the reference amino acid x r Given mutant amino acid sequence x v We propose a logarithmic ratio to estimate the relative validity of [the given value].

[0133]

number

[0134] The log-ratio in function (5) has been shown to accurately predict the effects of variations across different types of generative models, expressed as p(x│θ). However, if this model p(x│θ) is considered a nonlinear latent variable model, as in Riesselman, then the nonlinear latent variable model can estimate higher-order interactions between variants in the amino acid sequence. When data is generated under such a model, the calibrated pathogenicity prediction system 104 can sample a hidden variable z from a prior distribution p(z), such as a standard multivariate normal distribution, and generate the amino acid sequence x based on a conditional distribution p(x|z,θ) parameterized by a neural network. If z is hidden, the calibrated pathogenicity prediction system 104 can use the following function (6) to calculate the probability of p(x|z,θ)p(z).

[0135]

number

[0136] Function (6) considers all possible explanations for the hidden variable z by integrating the hidden variable, but function (6) also proposes the direct calculation of the cumbersome probability p(x|z,θ)p(z). Mutant amino acid sequence x v Rather than directly determining the probability, the calibrated pathogenicity prediction system 104 uses VAE to perform variational inference and reference amino acid sequence x r mutant amino acid sequence x v We can infer the lower bound on the (log) probability of [the event]. Such a boundary is generally known as the evidence lower bound (ELBO).

[0137]

number

[0138] In some embodiments, the calibrated pathogenicity prediction system 104 uses the following function (7) to estimate the ELBO for a given amino acid sequence x and to relate the ELBO to the logit or log probability of the given amino acid sequence x using a model with parameter θ.

[0139]

number

[0140] In function (7), q(z|x,φ) represents a variational approximation of the posterior distribution p(z|x,θ) of the hidden variable given the observed variable. Therefore, the calibrated pathogenicity prediction system 104 can use a neural network to model both the conditional distribution p(x|z,θ) of the generative model and the approximate posterior distribution q(z|x,φ) to form a VAE.

[0141] As shown in Figure 6, going further than Riesselman, the calibrated pathogenicity prediction system 104 uses a VAE and function (7) as the mutant pathogenicity machine learning model 604. In particular, in some embodiments, the mutant pathogenicity machine learning model 604 uses the mutant amino acid sequence x v The function (7) is executed on the mutant amino acid sequence x v Regarding

[0142]

number

[0143]

number

[0144]

number

[0145] In order to run the mutant pathogenicity machine learning model 604 as a VAE, in some embodiments, the mutant pathogenicity machine learning model 604 uses mutant amino acid sequences.

[0146]

number

[0147]

number

[0148]

number

[0149] Therefore, as shown in Figure 6, the calibrated pathogenicity prediction system 104 inputs data representing the mutant amino acid sequence 602 into the mutant pathogenicity machine learning model 604. By using functions (3) and (4) described above, the mutant pathogenicity machine learning model 604 acts as a VAE for determining the lower limit difference 606 for the mutant amino acid sequence 602, and thus the initial pathogenicity score 608.

[0150] Similar to other forms of mutant pathogenicity machine learning models and initial pathogenicity scores, the calibrated pathogenicity prediction system 104 can identify the temperature weights 616 generated by the temperature prediction machine learning model 614, or it can determine the temperature weights 616 using the temperature prediction machine learning model 614. As will be further explained below, the calibrated pathogenicity prediction system 104 determines a more accurate calibrated pathogenicity score by using protein-specific temperature weights rather than protein-position-specific temperature weights when the VAE functions as a mutant pathogenicity machine learning model. However, similarly, the initial pathogenicity score output by the VAE as a mutant pathogenicity machine learning model can be calibrated using protein-position-specific temperature weights, and in some cases, this performs better than protein-specific temperature weights.

[0151] As shown in Figure 6, for example, the calibrated pathogenicity prediction system 104 inputs data representing a mutant amino acid sequence 610 containing the target amino acid and an initial pathogenicity score 612 for the amino acids of the mutant amino acid sequence 610 into the temperature prediction machine learning model 614. Consistent with the above disclosure, the temperature prediction machine learning model 614 generates temperature weights 616 for the protein represented by the mutant amino acid sequence 610. By further multiplying the initial pathogenicity score 608 and the temperature weights 616, or by combining them in other ways, the calibrated pathogenicity prediction system 104 generates a calibrated pathogenicity score 618 for the mutant amino acid sequence 610 containing the target amino acid at the target protein location. As will be further described below, the temperature weights 616 improve the accuracy of such initial pathogenicity scores output by the VAE.

[0152] In addition to using different types of mutant pathogenicity machine learning models for calibration, in some embodiments, the calibrated pathogenicity prediction system 104 uses a meta-mutant pathogenicity machine learning model. According to one or more embodiments, Figure 7 illustrates that the calibrated pathogenicity prediction system 104 inputs pathogenicity scores for target amino acids at target protein locations of a given protein into the meta-mutant pathogenicity machine learning model 732, and generates refined pathogenicity scores for target amino acids based on the input calibrated pathogenicity scores. As shown previously, in some embodiments, the calibrated pathogenicity prediction system 104 inputs pathogenicity scores specific to target amino acids (or specific mutant amino acids) for a more accurate refined pathogenicity score output by the meta-mutant pathogenicity machine learning model 732.

[0153] As shown in Figure 7, for example, the calibrated pathogenicity prediction system 104 is supplied with data representing a target amino acid sequence 702 containing the target amino acid (e.g., A) at the target protein location, a reference amino acid sequence 701 for the protein corresponding to the target amino acid sequence 702, and a conserved MSA 703 for the protein corresponding to the target amino acid sequence 702 (and processed by the mutant pathogenicity machine learning model 704). Although not shown in Figure 7, in some embodiments, the calibrated pathogenicity prediction system 104 also supplies other data inputs to the mutant pathogenicity machine learning model 704, depending on the type of machine learning model used for the mutant pathogenicity machine learning model 704. Based on the target amino acid sequence 702, the reference amino acid sequence 701, and the conserved MSA 703, the mutant pathogenicity machine learning model 704 generates an initial pathogenicity score 706 for the target amino acid at the target protein location within the protein. The calibrated pathogenicity prediction system 104 further identifies a temperature weight 708 that estimates the temperature of the mutant pathogenicity machine learning model 704 for the pathogenicity score generated at the target protein location. The calibrated pathogenicity prediction system 104 further combines the initial pathogenicity score 706 and the temperature weight 708 to generate a calibrated pathogenicity score 710 for the target amino acid at the target protein location within the protein.

[0154] As described above, in some cases, the calibrated pathogenicity prediction system 104 identifies and combines pathogenicity scores from multiple variant pathogenicity machine learning models. As further shown in Figure 7, both the variational autoencoder (VAE) 714 and the transformer neural network 724, such as the transformer model from PrimateAI3D, are trained as variant pathogenicity machine learning models. The calibrated pathogenicity prediction system 104 is supplied with data representing the target amino acid sequence 712 containing the target amino acid at the target protein location (and processed by the VAE 714). Similarly, the calibrated pathogenicity prediction system 104 is supplied with data representing the target amino acid sequence 702 containing the target amino acid at the target protein location, the reference amino acid sequence 721 for the protein corresponding to the target amino acid sequence 722, and the conserved MSA 723 for the protein corresponding to the target amino acid sequence 702 (and processed by the transformer neural network 724). Based on the aforementioned data inputs suitable for the given model, the VAE 714 and the transformer neural network 724 generate initial pathogenicity scores 716 and 726, respectively, for target amino acids at target protein locations within the protein.

[0155] As further shown in Figure 7, in some embodiments, the calibrated pathogenicity prediction system 104 further identifies (i) a temperature weight 718 for estimating the temperature of the VAE 714 for the pathogenicity score generated for the protein, and (ii) a temperature weight 728 for estimating the temperature of the transformer neural network 724 for the pathogenicity score generated for the target protein location. The calibrated pathogenicity prediction system 104 optionally combines (i) the initial pathogenicity score 716 with the temperature weight 718 to generate a calibrated pathogenicity score 720 for the target amino acid at the target protein location within the protein, and (ii) combines the initial pathogenicity score 726 with the temperature weight 728 to generate a calibrated pathogenicity score 730 for the target amino acid at the target protein location within the protein.

[0156] After generating or identifying the pathogenicity scores output for the target amino acids, the calibrated pathogenicity prediction system 104 supplies the pathogenicity scores to the meta-mutant pathogenicity machine learning model 732, as further shown in Figure 7. For example, in some embodiments, the calibrated pathogenicity prediction system 104 supplies the calibrated pathogenicity scores 710, 720, and 730 to the meta-mutant pathogenicity machine learning model 732. In contrast, in some cases, the calibrated pathogenicity prediction system 104 supplies the calibrated pathogenicity score 710, the initial pathogenicity score 716, and the initial pathogenicity score 726 to the meta-mutant pathogenicity machine learning model 732. Thus, in some embodiments, the calibrated pathogenicity prediction system 104 supplies both the calibrated and uncalibrated pathogenicity scores to the meta-mutant pathogenicity machine learning model 732.

[0157] In both alternative approaches shown in Figure 7, the calibrated pathogenicity prediction system 104 utilizes a pathogenicity score specific to a single target amino acid at the target protein location (as input for the meta-mutant pathogenicity machine learning model 732), rather than pathogenicity scores for different target amino acids at the same or different protein locations. In some cases, the meta-mutant pathogenicity machine learning model 732 generates a more accurate and refined pathogenicity score based on inputting a single target amino acid-specific pathogenicity score (or single-mutant pathogenicity score) rather than pathogenicity scores for 20 candidate amino acids at the target protein location from among multiple locations on a given protein, because the meta-mutant pathogenicity machine learning model 732 (in some cases) does not receive data input that identifies the type of amino acid (e.g., A, M) corresponding to the input pathogenicity score. However, in other embodiments, the calibrated pathogenicity prediction system 104 is supplied with pathogenicity scores for different target amino acids at the same or different protein locations, output by different mutant pathogenicity machine learning models (and processed by a meta-mutant pathogenicity machine learning model), to generate refined pathogenicity scores for individual target amino acids.

[0158] Based on the input pathogenicity score, the meta-mutant pathogenicity machine learning model 732 generates a refined pathogenicity score 734 for the target amino acid at the target protein location within the protein. In some cases, for example, the meta-mutant pathogenicity machine learning model 732 takes the form of a multilayer perceptron (MLP) or convolutional neural network (CNN) trained to generate a more accurate pathogenicity score. Partly due to the different pathogenicity scores from different types of mutant pathogenicity machine learning models input to the machine learning model 732, the meta-mutant pathogenicity machine learning model 732 generates a refined pathogenicity score that is less susceptible to the fluctuating temperature of different types of mutant pathogenicity machine learning models.

[0159] Figure 7 shows that the meta-mutant pathogenicity machine learning model 732 processes pathogenicity scores from three different mutant pathogenicity machine learning models, but as the dotted ellipse indicates, the meta-mutant pathogenicity machine learning model 732 can be trained and processed from input pathogenicity scores from any number of different mutant pathogenicity machine learning models. For example, in some embodiments, the meta-mutant pathogenicity machine learning model 732 generates refined pathogenicity scores for target amino acids at target protein locations based on input pathogenicity scores from 5, 10, 39, or 51 different mutant pathogenicity machine learning models. As shown in Tables 2 to 4 below, different embodiments of the meta-mutant pathogenicity machine learning model generate refined pathogenicity scores that exhibit consistent accuracy across specific clinical benchmarks.

[0160] To train the meta-mutant pathogenicity machine learning model 732, in some embodiments, the calibrated pathogenicity prediction system 104 uses a binary cross-entropy loss function weighted by the mutation rate. For example, the calibrated pathogenicity prediction system 104 uses the binary cross-entropy loss function to compare input pathogenicity scores (e.g., as probabilities) from different mutant pathogenicity machine learning models with ground truth pathogenicity classifications for target amino acids at target protein locations. For example, a ground truth pathogenicity classification of 0 represents that the target amino acid is benign, and a ground truth pathogenicity classification of 1 represents that the target amino acid is pathogenic. By comparing the input pathogenicity scores with ground truth pathogenicity classifications (e.g., 0 or 1) using the binary cross-entropy loss function, the binary cross-entropy loss function determines the negative mean of the logarithm of the corrected input pathogenicity scores, also known as the binary cross-entropy loss.

[0161] Based on the binary cross-entropy loss over a given training iteration, the calibrated pathogenicity prediction system 104 modifies the parameters (e.g., network parameters) of the meta-mutant pathogenicity machine learning model 732. By tuning the parameters across training iterations, the calibrated pathogenicity prediction system 104 improves the accuracy of the meta-mutant pathogenicity machine learning model 732 in determining a refined pathogenicity score that distinguishes between benign and pathogenic mutant amino acids at a given protein location. Based on the binary cross-entropy loss, for example, the calibrated pathogenicity prediction system 104 determines the gradient over the weights using stochastic gradient descent (SGD). In some cases, the calibrated pathogenicity prediction system 104 uses the following function:

[0162]

number

[0163] In addition to determining or adjusting temperature weights, as described above, the calibrated pathogenicity prediction system 104 can generate data for protein graphics that existing models cannot support, i.e., graphics showing temperature weights for specific proteins or protein locations within a protein. According to one or more embodiments, Figures 8A to 8E show graphical visualizations of protein location-specific temperature weights for pathogenicity scores output by mutant pathogenicity machine learning models. As shown in Figures 8A to 8E, the calibrated pathogenicity prediction system 104 can generate and represent different temperature weights using different colors, patterns, or shading to indicate different degrees of certainty or uncertainty regarding the pathogenicity score at specific protein locations within a protein.

[0164] Figures 8A to 8E show the graphical user interfaces 802a to 802e that appear when the client device 110 executes computer executable instructions for the analysis application 112. Rather than repeatedly referring to computer executable instructions that cause the client device 110 to perform specific operations for the calibrated pathogenicity prediction system 104, this disclosure describes the client device 110 or the calibrated pathogenicity prediction system 104 that perform those operations in the following paragraphs.

[0165] As shown in Figure 8A, for example, the client device 110 displays a position-temperature weight graphical visualization 804a within the graphical user interface 802a that represents the temperature weights for pathogenicity scores at different protein locations within the target protein. As described above, in some cases, the calibrated pathogenicity prediction system 104 represents temperature weights indicating low-uncertainty pathogenicity scores at protein locations with one set of specific colors, patterns, or shadings, and temperature weights indicating high-uncertainty pathogenicity scores at protein locations with another set of specific colors, patterns, or shadings. The following disclosures with respect to Figures 8A to 8E illustrate protein locations in the position-temperature-weight graphical visualization with respect to colored shading and provide examples of colored shading (e.g., blue or red shading) for representing temperature weights, however, in some embodiments, other colored shading (e.g., gray or black shading), colors, or patterns may be used to represent temperature weights at protein locations in the position-temperature-weight graphical visualization.

[0166] For example, as shown in Figure 8A, the position-temperature-weight graphical visualization 804a represents different temperature weights at different target protein locations, each having a different colored shading. In particular, the position-temperature-visualization 804a includes a first portion 806a of a protein having a location depicted with a first set of colored shading (e.g., blue shading) that indicates a temperature weight indicating a less uncertain pathogenicity score. Thus, the first set of colored shading (e.g., specific blue shading) represents a relatively lower temperature or lower temperature weight at a given protein location or portion, and consequently represents a relatively more certain pathogenicity score at a given protein location or portion. In contrast, the position-temperature-weight graphical visualization 804a includes a second portion 808a of a protein having a location depicted with a second set of colored shading (e.g., red shading) that indicates a temperature weight indicating a less uncertain pathogenicity score. Therefore, the second set of colored shading (e.g., red shading) represents a relatively higher temperature or higher temperature weight at a given protein location or portion, and consequently represents a relatively less certain pathogenicity score at a given protein location or portion. As the temperature weight changes from a relatively low temperature (or higher certainty) to a relatively high temperature (or lower certainty), the location-temperature-weight graphical visualization 804a also represents the temperature weight at different locations with a shading gradient, with colored shading that changes between the first set of colored shading (e.g., blue shading) and the second set of colored shading (e.g., red shading) according to the respective values ​​of the corresponding temperature weight for a given location.

[0167] As the graphical visualization of location-temperature-weight 804a demonstrates, the temperature weights generated by the temperature prediction machine learning model may (or may indicate) be related to different protein portions within the target protein. Therefore, the graphical visualization of location-temperature-weight 804a provides a snapshot of which protein locations (or larger portions) of the target protein exhibit pathogenicity scores that are more or less affected by uncertainty caused by the mutant pathogenicity machine learning model itself, or by data input to the mutant pathogenicity machine learning model at different protein locations, without affecting uncertainty caused by either evolutionary constraints or pathogenicity constraints. As described above, existing models and temperature scaling factors cannot separate uncertainty about global machine learning models (e.g., transformer machine learning models) from other, more specific types of uncertainty. Therefore, the graphical visualizations described and depicted in this disclosure represent the first and only visualizations to depict model-induced or data-induced uncertainty about pathogenicity scores corresponding to specific locations, separate (or independent) from uncertainty caused by evolutionary constraints or pathogenicity constraints.

[0168] Similar to Figure 8A, as shown in Figures 8B, 8C, 8D, and 8E, the client device 110 displays graphical position-temperature-weight visualizations 804b, 804c, 804d, and 804e within the graphical user interfaces 802b, 802c, 802d, and 802e, respectively, representing the temperature weights for the pathogenicity score at different protein locations within a given target protein. As shown in Figures 8B to 8E, the graphical position-temperature-weight visualizations 804b to 804e visualize the temperature weights for different protein locations in different target proteins, respectively. Although the internal components of each protein differ, the position-temperature-weight graphical visualizations 804b, 804c, 804d, and 804e each contain the first portion of their respective proteins 806b, 806c, 806d, and 806e, with the position depicted by a set of colored shading (e.g., blue shading) representing temperature weights that indicate a less uncertain pathogenicity score. In contrast, the position-temperature-weight graphical visualizations 804b, 804c, 804d, and 804e each contain the second portion of their respective proteins 808b, 808c, 808d, and 808e, with the position depicted by another set of colored shading (e.g., red shading) representing temperature weights that indicate a less uncertain pathogenicity score.

[0169] As further shown in Figures 8B–8E, the graphical visualizations of location-temperature-weight 804b, 804c, 804d, and 804e also include a third portion of each protein 810b, 810c, 810d, and 810e, respectively, where the location is depicted with a third set of colored shading (e.g., purple shading) representing temperature weights indicating moderate uncertainty about the pathogenicity score. Thus, the third set of colored shading (e.g., purple shading) represents a relatively moderate temperature or moderate temperature weight for a given protein location or portion, and therefore represents a relatively moderate certainty about the pathogenicity score at a given protein location or portion.

[0170] In addition to providing a groundbreaking graphical visualization, the Calibrated Pathogenicity Prediction System 104 improves the accuracy and precision with which pathogenicity prediction models generate pathogenicity predictions for amino acid variants across specific clinical benchmarks and cell-kline protocols. As shown in Table 2 below, researchers have developed (i) a meta-variant pathogenicity machine learning model that combines pathogenicity scores from five different models developed by Illumina (hereinafter referred to as the 5-score combined meta-classifier), (ii) a meta-variant pathogenicity machine learning model that combines pathogenicity scores from a PrimateAI3D-only approach, a triangle attention-only approach, and VAEs or other models described in this paragraph (hereinafter referred to as the combined meta-classifier), and (iii) a combined pathogenicity score that includes a normalized calibrated pathogenicity score based on temperature weights of a triangle attention neural network and a normalized pathogenicity score from an ensemble of 40 models from PrimateAI3D. We measured the performance of pathogenicity scores generated by five models, including (iv) a machine learning model (hereinafter referred to as Additional Triangle Attention + PrimateAI3D), (iv) a PrimateAI3D model only (referred to as PrimateAI3D above and below) which uses an ensemble of 40 models without calibration from temperature weights to generate pathogenicity scores by determining the average of the initial pathogenicity scores output by 40 models, and (v) a mutant pathogenicity machine learning model which generates a calibrated pathogenicity score by combining temperature weights from the Triangle Attention Neural Network with pathogenicity scores from the Transformer machine learning model used in PrimateAI3D (referred to as Triangle Attention above and below).

[0171] As shown in Table 2, researchers measured performance in predicting the pathogenicity of target amino acids from developmental disorder diagnosis (DDD) studies, the UK Biobank, cell line experiments for saturated mutagenesis, clinical variants (ClinVar) from the National Library of Medicine, and Genomics England Variants (GELVar).

[0172] [Table 2]

[0173] As suggested above, in some embodiments of the approach adding Triangle Attention + PrimateAI3D, the calibrated pathogenicity prediction system 104 (a) normalizes the calibrated pathogenicity score calibrated using the temperature weights of the Triangle Attention Neural Network, (b) normalizes the pathogenicity score output by an ensemble of 40 models from PrimateAI3D, and (c) sums the normalized calibrated pathogenicity score and the normalized initial pathogenicity score to generate a combined pathogenicity score for the target amino acid at the target protein location. In certain embodiments for other models, the calibrated pathogenicity prediction system 104 similarly combines a normalized calibrated pathogenicity score calibrated with temperature weights output by another temperature prediction machine learning model and a normalized initial pathogenicity score output by another mutant pathogenicity machine learning model to generate a combined pathogenicity score for the target amino acid at the target protein location.

[0174] As shown in Table 2 above, Triangle Attention using only a single transformer model from PrimateAI3D generates a calibrated pathogenicity score that performs similarly to PrimateAI3D with only an ensemble of 40 models spanning DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar. Therefore, Table 2 shows that by combining temperature weights with the initial pathogenicity score from a single model from PrimateAI3D, the calibrated pathogenicity prediction system 104 can significantly improve performance across benchmarks. Since PrimateAI3D exhibits state-of-the-art performance with an ensemble of 40 models, as shown in Table 2, the Triangle Attention-only approach can exhibit better performance than the state-of-the-art performance with reduced computation from a single model of PrimateAI3D. Furthermore, by normalizing the calibrated pathogenicity scores, which are calibrated using temperature weights from a triangular attention neural network, and normalizing the pathogenicity scores from an ensemble of 40 models from PrimateAI3D, and combining the normalized calibrated pathogenicity scores with the normalized PrimateAI3D pathogenicity scores, the Add Triangle Attention + PrimateAI3D approach shows relatively improved pathogenicity scores for each benchmark, including DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar.

[0175] As suggested above, Table 2 shows performance metrics for different benchmarks for accurately identifying mutant pathogenicity. For example, the calibrated pathogenicity score of the Add Triangle Attention + PrimateAI3D approach more accurately identifies mutant amino acids that cause developmental disorders from the DDD database and better identifies control or benign amino acids that do not cause such developmental disorders than the PrimateAI3D and Triangle Attention alone approach. Table 2 shows R for UK Biobank and saturated mutagenesis. 2As the values ​​indicate, the calibrated pathogenicity score of the Add Triangle Attention + PrimateAI3D approach also more accurately identifies pathogenic amino acid variants associated with specific phenotypes represented in the UKBB database, and more accurately identifies cell lines that die or survive with mutant amino acids using the saturated mutagenesis protocol than the pathogenicity scores of the PrimateAI3D and Triangle Attention-only approaches. As further indicated by the ClinVar AUC and GELVar p values, the calibrated pathogenicity score of the Add Triangle Attention + PrimateAI3D approach also more accurately identifies pathogenic amino acid variants in the ClinVar GELVar database than the pathogenicity scores of the PrimateAI3D and Triangle Attention-only approaches. The values ​​for DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar in the tables and figures described below also illustrate the performance comparisons described above.

[0176] As further shown in Table 2, both the 5-score combined metaclassifier and the combined metaclassifier exhibit relatively improved pathogenicity scores at each benchmark. By combining pathogenicity scores from the Add Triangle Attention + PrimateAI3D, PrimateAI3D only, and Triangle Attention only approaches, the combined metaclassifier generates refined pathogenicity scores with improved performance in identifying pathogenicity from data in each of DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar. By combining pathogenicity scores from five different models from Illumina, the 5-score combined metaclassifier generates refined pathogenicity scores with further improved performance in identifying the pathogenicity of amino acid variants from data in each of DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar.

[0177] To facilitate performance comparisons across clinical benchmarks and cell line protocols, researchers compared scores for PrimateAI3D against each clinical benchmark or cell line protocol. According to one or more embodiments, Figure 9 depicts a bar graph 900 showing relative scores for different models against performance metrics for mutant pathogenicity prediction. In particular, bar graph 900 shows relative scores for different models that identify the pathogenicity of amino acid variants from data in DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar, respectively. Consistent with the above disclosure, bar graph 900 compares the performance of pathogenicity scores from combined metaclassifier, Add Triangle Attention + PrimateAI3D, PrimateAI3D only, and triangle attention only approaches.

[0178] To determine the relative scores shown in Bar Graph 900, the researchers normalized the performance metrics from Table 2 using different techniques. For example, base 10 logarithm was determined for the p-values ​​for DDD and GELVar. 2 Spearman's rank correlation was determined for the values. Furthermore, the area under the curve (AUC) was determined for ClinVar by determining the AUC per gene and then further determining the average AUC across genes.

[0179] As shown by bar graph 900, the combined metaclassifier generates refined pathogenicity scores that better identify pathogenic or benign amino acid variants from data in DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar than other variant pathogenicity machine learning models. By normalizing the calibrated pathogenicity scores calibrated using temperature weights of a triangular attention neural network, normalizing the pathogenicity scores from an ensemble of 40 models from PrimateAI3D, and combining the normalized calibrated pathogenicity scores with the normalized PrimateAI3D pathogenicity scores, the Add Triangle Attention + PrimateAI3D approach exhibits the following best performance for pathogenicity scores in each benchmark compared to PrimateAI3D alone and Triangle Attention alone. As suggested by Table 2 above, the bar graph 900 also confirms that the Triangle Attention-only approach generates calibrated pathogenicity scores that demonstrate performance against clinical benchmarks and cell line protocols similar to the state-of-the-art performance of PrimateAI3D-only.

[0180] To further evaluate the performance of the temperature-weighted and meta-mutant pathogenicity machine learning models described above, the researchers varied the parameters of the specific models described above and determined performance metrics across benchmarks against existing pathogenicity prediction models (e.g., PrimateAI1D and DeepSequence). The performance metrics for these models are shown in Tables 3 and 4 below.

[0181] [Table 3]

[0182] [Table 4]

[0183] As shown above, the mutant pathogenicity machine learning models in Table 3 were tested on a larger set of amino acid variants than the mutant pathogenicity machine learning models in Table 4. The first row of Tables 3 and 4 shows performance metrics for the same mutant pathogenicity machine learning models evaluated for different sets of amino acid variants, although performance metrics for PrimateAI1D and DeepSequence were only available for the smaller sets of amino acid variants shown in Table 4.

[0184] As shown in Tables 3 and 4, the 5-score combination metaclassifier generates refined pathogenicity scores that better distinguish between pathogenic and benign amino acid variants from data in DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar than other variant pathogenicity machine learning models. As shown in Tables 3 and 4, the Triangle Attention only (1B parameter) approach represents a calibrated pathogenicity score generated by applying temperature weights output by a triangle attention neural network to an initial pathogenicity score output by a transformer machine learning model that processes MSA input and has 1 billion parameters. Similarly, the Triangle Attention only (150M parameter) approach represents a calibrated pathogenicity score generated by applying temperature weights output by a triangle attention neural network to an initial pathogenicity score output by a transformer machine learning model that processes MSA input and has 150,000,000 parameters. As shown in Tables 3 and 4, the calibrated pathogenicity scores from the Triangle Attention-only (1B parameter) approach demonstrate performance against clinical benchmarks and cell line protocols similar to the state-of-the-art performance of PrimateAI3D using only an ensemble of 40 models. Furthermore, the Triangle Attention-only (1B parameter) and Triangle Attention-only (150M parameter) approaches generate calibrated pathogenicity scores that perform better against clinical benchmarks and cell line protocols than the state-of-the-art performance of the transformer for PrimateAI3D with 1 billion and 150,000,000 parameters, respectively, based on data from DDD, UKBB, saturated mutagenesis, ClinVar, and GELVar.

[0185] As described above, the calibrated pathogenicity prediction system 104 can use a variational autoencoder (VAE) as a variant pathogenicity machine learning model to determine an initial pathogenicity score for a target amino acid at a target protein location within the protein. Similar to other forms of variant pathogenicity machine learning models, the calibrated pathogenicity prediction system 104 improves the accuracy of the initial pathogenicity score from the VAE by combining temperature weights from a temperature prediction machine learning model with such an initial pathogenicity score. Several initial studies indicate that the calibrated pathogenicity prediction system 104 determines a more accurate calibrated pathogenicity score by using protein-specific temperature weights rather than protein-location-specific temperature weights when the VAE functions as a variant pathogenicity machine learning model. However, as shown in Table 5 below, protein-location-specific temperature weights can also improve the accuracy of the initial pathogenicity score from the VAE.

[0186] As shown in Table 5 below, to test the performance of temperature weights using scores from VAEs, researchers used: (i) a baseline VAE from DeepSequence (hereinafter referred to as VAE baseline); (ii) a calibrated VAE that generates a calibrated pathogenicity score by combining a single temperature weight for the target protein from a temperature prediction machine learning model (e.g., MLP) with a pathogenicity score from the VAE from DeepSequence (hereinafter referred to as VAE + single positive weight); (iii) a calibrated VAE that generates a calibrated pathogenicity score by combining a protein position-specific temperature weight for the target protein from a triangular attention neural network with a pathogenicity score from the VAE from DeepSequence (hereinafter referred to as VAE + triangular attention); and (iv) a mutant pathogenicity machine learning model that generates a calibrated pathogenicity score by combining temperature weights from a triangular attention neural network with 1 billion parameters and pathogenicity scores from a single transformer from PrimateAI3D Triangle Attention (hereinafter referred to as Triangle Attention). We measured the performance of pathogenicity scores generated by four models, including Attention (called the 1B parameter).

[0187] As shown by VAE+Triangle Attention in Table 5 below, to improve the performance of protein location-specific temperature weights, the calibrated pathogenicity prediction system 104 can determine temperature weights by finding the average temperature weights using a modified Gaussian blurring or modified moving average model. Specifically, the calibrated pathogenicity prediction system 104 (i) applies Gaussian blurring to determine the average temperature weight from the initial temperature weights for various amino acids at a particular protein location, and (ii) divides the average temperature weight by the total weight (e.g., sum of temperature weights) for amino acid variants within a window (e.g., 300, 500, 800 amino acids). If the data within the window is not sparse, the total weight is typically a value of 1.

[0188] [Table 5]

[0189] As shown in Table 5, by combining protein-specific temperature weights with initial pathogenicity scores from VAE, the calibrated pathogenicity prediction system 104 can significantly improve its performance across each benchmark compared to the VAE baseline. Similarly, by combining (i) protein position-specific temperature weights generated by a triangular attention neural network and subjected to the modified Gaussian blurring described above with (ii) initial pathogenicity scores from VAE, the calibrated pathogenicity prediction system 104 can significantly improve its performance across each benchmark compared to the VAE baseline. As further shown in Table 5, the calibrated pathogenicity scores in the Triangle Attention (1B parameter) approach show more accurate scores compared to the VAE baseline and the calibrated pathogenicity scores from VAE as a mutant pathogenicity machine learning model.

[0190] As described above, the calibrated pathogenicity prediction system 104 can utilize various different variant pathogenicity machine learning models. According to one or more embodiments, Figures 10, 11, and 12 show the architecture, components, and various inputs and outputs of a transformer neural network from PrimateAI3D operating as a variant pathogenicity machine learning model.

[0191] For example, Figure 10 illustrates an exemplary architecture 1000 of the PrimateAI language model. The PrimateAI language model includes dependent connections of axial attention blocks 1008 (e.g., 12 axial attention blocks). The dependent connections of the axial attention block 1008 take an MSA representation 1006 as input and produce an updated MSA representation 1015 as output. Each axial attention block includes residues that add a bound row-gated self-attention layer 1010, a bound column-gated self-attention layer 1012, and a transition layer 1014.

[0192] In one embodiment, there are 12 heads in a coupled row-gated self-attention layer 1010. In another embodiment, there are 12 heads in a coupled column-gated self-attention layer 1012. Each head generates 64 channels, and the channels are summed up across the 12 heads (768). In one embodiment, the transition layer 1014 projects up to 3072 channels for GELU activation.

[0193] This technology discloses a modified axially gated self-attention that includes combined attention instead of triangular attention. Triangular attention has a high computational cost. Combined attention is calculated by dividing the sum of the dot product similarities between key and value across non-padding rows by the square root of the number of non-padding rows, which substantially reduces the computational load.

[0194] Mask expression reveals unknown values ​​at other mask locations after the dependent connection of axial attention block 1008. Mask expression collects features aligned with the mask sites. For each masked residue in a row, mask expression reveals embedded target tokens at other mask locations in that row.

[0195] The mask expression combines the updated 768-channel MSA representation as the updated MSA representation 1015 with the embedded representation (token embedding) 1034 of the 96-channel target at locations indicated by a Boolean mask 1030 that labels the mask token locations. The Boolean mask 1030, which is a fixed mask pattern with a stride 16, is applied in the row direction to collect features from the MSA representation and target token embedding at the mask token locations.

[0196] Feature collection reduces the row length from 256 to 16, dramatically decreasing the computational cost of the attention block following mask expression. For each location within each row of the collected MSA representation, the row is concatenated with the corresponding row from the collected target token embedding, and that location is also masked in the target token embedding. The MSA representation and the partially expressed target embedding are concatenated in the channel dimension and mixed by linear projection.

[0197] Following the mask expression 1017, the MSA expression 1018 notified here is propagated through the remaining row-gated self-attention layers (e.g., row-gated self-attention layer 1020 and row-gated self-attention layer 1026), as well as the transition layer 1024. Since residues are known for other locations from the MSA expression 1006 provided as input to the PrimateAI language model, attention is applied only to features at mask locations. Thus, attention only needs to be applied to mask locations where new information from the mask expression exists. In some cases, the transition layer 1024 and row-gated self-attention layer 1026 may be repeated four times, as shown by the repeating loop 1022 in Figure 10.

[0198] Following the interpretation of the masked expression by self-attention, the masked collection operation 1028 collects features from the resulting MSA representation at locations where the target token embedding remains masked. The collected MSA representation 1032 is converted by the output head 1036 into predictions 790 for 21 candidates in the amino acid and gap token vocabulary. The output head 1036 includes a transition layer and a perceptron.

[0199] Figure 11 shows various components 1100 of a PrimateAI language model according to one embodiment. The components may include combined row-gated self-attentions, row-gated self-attentions, and column-gated self-attentions. The PrimateAI language model may also use combined attentions. Axial attentions create independent attention maps for each row and column of the input. Sequences in MSA typically have a similar three-dimensional structure. Direct coupling analysis takes advantage of this fact to learn structural contact information. To leverage this shared structure, it is beneficial to combine row attention maps between sequences in the MSA. As an additional benefit, combined attentions reduce the memory footprint of row attentions.

[0200] In embodiments involving recalculation, the combined attention reduces the memory footprint of the row attention to O(ML). 2 ) to O(L 2 ) is reduced to Q. Let M be the number of rows, let d be the hidden dimension, and Q m , K m Let be a matrix of queries and keys for the m-th row of the input. The joined row attention is defined as follows before softmax is applied:

[0201]

number

[0202] The final model uses square root normalization. In other embodiments, the model may use mean normalization. In such embodiments, the denominator l(M,d) is the normalization constant in standard scaled dot product attention.

number

[0203]

number

[0204] In Figure 11, sequence s=32, residue r=256, attention head, h=12 and channel, c=64, and c MSA The dimension of =768 is given.

[0205] In one embodiment, the PrimateAI language model can be trained on four A100 graphical processing units (GPUs). The optimizer step is for a batch size of 80 MSA and is divided into four gradient aggregates to fit the batch into 40 GB of A100 memory. The PrimateAI language model is trained using a LAMB optimizer with the following parameters: β_1=0.9, β_2=0.999, ε=10⁻⁶, and weight decay of 0.01. The gradients are pre-normalized by division by their global L2 norm before applying the LAMB optimizer. Training is normalized by dropout with probability 0.1, which is applied after activation and before residue conjugation.

[0206] To train the described PrimateAI language model, in some embodiments, residue blocks are started as identity operations, which accelerates convergence and enables the PrimateAI language model. "AdamW" refers to the ADAM optimizer with weight decay, "ReZeRO" refers to the zero-redundancy optimizer, and "LR" refers to the LAMB optimizer with gradient pre-normalization. See Large Batch Optimization for Deep Learning Training BERT in 76 minutes, International Conference on Learning Representations (ICLR) 2020 by Yang You, Jing Li, Sashank Reddi et al. As illustrated, the LAMB optimizer with gradient pre-normalization exhibits better performance (e.g., higher accuracy rates over fewer training iterations) and is more effective over a range of learning rates compared to the use of the ADAMW optimizer and the zero-redundancy optimizer.

[0207] Axial dropout can be applied to self-attention blocks before residual connections. Columnar dropout follows softmax post-spatial gating in columnar attention, while row-direction dropout follows softmax post-spatial gating in row-direction attention. Softmax post-spatial gating allows modulation of the exponentially normalized scores or probabilities generated by softmax.

[0208] In one embodiment, the PrimateAI language model can be trained for 100,000 parameter updates. The learning rate over the first 5,000 steps is η = 5 × 10⁻¹⁰. -6 From η = 5 × 10 -4 It increases linearly up to the peak value of η=10 -4It decays linearly up to a certain point. Automatic mixed precision (AMP) can be applied to cast preferred operations from 32-bit precision to 16-bit precision during training and inference. This increases throughput and reduces memory consumption without impacting performance. In addition, a zero-redundancy optimizer reduces memory usage by sharding optimizer states across multiple GPUs.

[0209] Figure 12 shows one embodiment of the output head 1036 that may be used by the disclosed PrimateAI language model. The collected MSA representation 1032 can be converted by the output head 1036 into predictions 790 for 21 candidates in an amino acid vocabulary including gap tokens. In one embodiment, the amino acid vocabulary may be enumerated, and the amino acid enumeration is used to index into a dictionary of learned embeddings. In other embodiments, one-hot embeddings of amino acids may be used and combined with linear projection. In some embodiments, the output head 1036 may comprise a transition layer 1202, a gate 1204, a layer normalization block 1206, a linear block 1208, a GELU block 1210, and another linear block 1212. Dimensions are channel, c MSA This is shown for =768 and vocabulary size, and v=21.

[0210] Referring now to Figure 13, this figure shows a flowchart of a series of operations 1300 according to one or more embodiments of the present disclosure, which identify initial pathogenicity scores for amino acid variants at specific protein locations and apply temperature weights to those initial pathogenicity scores in order to generate a calibrated pathogenicity score. Figure 13 shows operations according to one embodiment, but alternative embodiments may omit, add, rearrange, and / or modify any of the operations shown in Figure 13. The operations in Figure 13 may be performed as part of a method. Alternatively, a non-temporary computer-readable storage medium may, when executed by one or more processors, contain instructions that cause a computing device or system to perform the operations shown in Figure 13. In further embodiments, the system comprises at least one processor and a non-temporary computer-readable medium containing instructions that cause the system to perform the operations in Figure 13 when executed by one or more processors.

[0211] As shown in Figure 13, a series of operations 1300 includes an operation 1302 that determines an initial pathogenicity score. In particular, in some embodiments, operation 1302 includes using a mutant pathogenicity machine learning model to determine an initial pathogenicity score for a target amino acid at a target protein location within the protein, based on the amino acid sequence for the protein. In one or more embodiments, the mutant pathogenicity machine learning model used to generate the initial pathogenicity score includes transformer machine learning models, convolutional neural networks (CNNs), sequence-to-sequence models, variational autoencoders (VAEs), multilayer perceptrons (MLPs), recurrent neural networks (RNNs), long-short-term memory (LSTMs), or decision tree models.

[0212] As further shown in Figure 13, the sequence of operations 1300 includes an operation 1304 that identifies temperature weights. In particular, in some embodiments, operation 1304 includes identifying temperature weights for a protein that estimate the temperature of a mutant pathogenicity machine learning model. As suggested above, in certain cases, identifying temperature weights includes identifying weights that estimate the degree of certainty for a protein or target protein location to the pathogenicity score output by the mutant pathogenicity machine learning model.

[0213] For example, in some embodiments, identifying temperature weights includes identifying temperature weights for the target protein location of the protein. Furthermore, in certain embodiments, identifying temperature weights includes applying a nonlinear activation function to the initial weights to determine positive temperature weights. As further suggested above, in some embodiments, identifying temperature weights includes determining the average temperature weight from the initial temperature weights at the target protein location. For example, in certain cases, determining temperature weights includes using a Gaussian blurring model, median filter, or bilateral filter to determine the average temperature weight from the initial temperature weights for various amino acids at the target protein location.

[0214] In relation to this, in some embodiments, identifying temperature weights involves using a temperature prediction machine learning model to determine the temperature weight for a protein based on an initial pathogenicity score and the amino acid or nucleotide sequence corresponding to that protein. In some cases, the temperature prediction machine learning model used to determine the temperature weights includes multilayer perceptrons (MLPs), convolutional neural networks (CNNs), triangular attention neural networks, recurrent neural networks (RNNs), long-short-term memory (LSTMs), transformer machine learning models, or decision tree models.

[0215] As described above, in one or more embodiments, identifying temperature weights includes determining one or more of the following using a triangular attention neural network: amino acid pairwise index difference embeddings representing the differences between amino acids in the amino acid sequence for the protein; amino acid pairwise atomic distance matrices representing the pairwise distances between atoms in the protein; reference residue embeddings representing reference residues for the protein; a conserved multiple sequence alignment matrix representing multiple sequence alignments for proteins from multiple species; and a pathogenicity score matrix representing pathogenicity scores for amino acids in the protein generated by a mutant pathogenicity machine learning model; determining residue pair representations based on one or more of the amino acid pairwise index difference embeddings, amino acid pairwise atomic distance matrices, reference residue embeddings, conserved multiple sequence alignment matrices, and pathogenicity score matrices; projecting temperature weights for protein locations based on the residue pair representations; and identifying temperature weights for target protein locations within the protein from among the temperature weights.

[0216] To illustrate further, in some embodiments, determining a residue pair representation includes determining the residue pair representation based on a combination of amino acid pairwise index difference embedding, amino acid pairwise atomic distance matrix, reference residue embedding, conserved multiple sequence alignment matrix, and pathogenicity score matrix; generating a modified residue pair representation using one or more triangular attention layers; determining a diagonal residue pair representation from the modified residue pair representation; and projecting temperature weights for protein positions from the diagonal residue pair representation.

[0217] As further shown in Figure 13, the sequence of operations 1300 includes operation 1306, which generates a calibrated pathogenicity score for target amino acids at target protein locations based on the initial pathogenicity score and temperature weights. In particular, in certain embodiments, operation 1306 includes:

[0218] In addition to, or instead of, operations 1302-1306, in certain embodiments, a series of operations 1300 includes, for display purposes, generating a graphical visualization of temperature weights indicating the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model for a protein or target protein location.

[0219] To illustrate further, in some cases, the sequence of operations 1300 further includes: using an additional mutant pathogenicity machine learning model to generate an additional pathogenicity score for the target amino acid at the target protein site; normalizing the additional pathogenicity score and the calibrated pathogenicity score for the target amino acid; and combining the normalized additional pathogenicity score and the normalized calibrated pathogenicity score to generate a combined pathogenicity score for the target amino acid at the target protein site.

[0220] As suggested above, in some cases, the sequence of operations 1300 further includes using an additional mutant pathogenicity machine learning model to generate an additional pathogenicity score for a target amino acid at a target protein site, and using a meta-mutant pathogenicity machine learning model to generate a refined pathogenicity score for a target amino acid at a target protein site based on a calibrated pathogenicity score and an additional pathogenicity score. In connection with this, in some embodiments, the sequence of operations 1300 includes determining an initial pathogenicity score for a particular mutant amino acid at a target protein site based on data representing the particular mutant amino acid and the amino acid sequence for the protein, generating an additional pathogenicity score for the particular mutant amino acid at a target protein site, and generating a refined pathogenicity score for the particular mutant amino acid at a target protein site.

[0221] Referring now to Figure 14, this figure shows a flowchart of a series of operations 1400 that generate a graphical visualization of temperature weights for the location of a target protein within a target protein, according to one or more embodiments of the present disclosure. Figure 14 shows operations according to one embodiment, but alternative embodiments may omit, add, rearrange, and / or modify any of the operations shown in Figure 14. The operations in Figure 14 may be performed as part of a method. Alternatively, a non-temporary computer-readable storage medium may, when executed by one or more processors, include instructions that cause a computing device or system to perform the operations shown in Figure 14. In a further embodiment, the system comprises at least one processor and a non-temporary computer-readable medium that, when executed by one or more processors, includes instructions that cause the system to perform the operations in Figure 14.

[0222] As shown in Figure 14, a series of operations 1400 includes an operation 1402 that identifies a target protein and the location of the target protein. For example, operation 1402 may include identifying a target protein and the location of the target protein within that target protein. In some embodiments, identifying a target protein and the location of the target protein within that target protein includes receiving a user selection of the target protein from a client device or computing device. In connection with this, in certain embodiments, identifying a target protein and the location of the target protein within that target protein includes receiving a user selection of options for viewing temperature weights for the target protein from a client device or computing device.

[0223] As further shown in Figure 14, a series of operations 1400 includes operation 1404, which determines temperature weights for the mutant pathogenicity machine learning model at the target protein location. In some cases, operation 1404 includes determining temperature weights to estimate the temperature of the mutant pathogenicity machine learning model at the target protein location within the target protein.

[0224] As described above, in some embodiments, determining temperature weights involves accessing weights from a database that estimate the degree of certainty for the pathogenicity score output by a mutant pathogenicity machine learning model for the target protein location. Furthermore, in some cases, determining temperature weights involves using a temperature prediction machine learning model to determine the temperature weights corresponding to the target protein location based on the target amino acid at the target protein location and the initial pathogenicity score for the amino acid sequence or nucleotide sequence corresponding to the target protein.

[0225] As further shown above, in certain embodiments, determining temperature weights includes determining temperature weights for target protein locations by using a triangular attention neural network to determine one or more of the following: amino acid pairwise index difference embeddings representing the differences between amino acids in the amino acid sequence of the target protein; amino acid pairwise atomic distance matrices representing the pairwise distances between atoms in the target protein; reference residue embeddings representing reference residues of the target protein; conserved multiple sequence alignment matrices representing multiple sequence alignments for target proteins from multiple species; and pathogenicity score matrices representing pathogenicity scores for amino acids in the target protein generated by a mutant pathogenicity machine learning model; determining residue pair representations based on one or more of the amino acid pairwise index difference embeddings, amino acid pairwise atomic distance matrices, reference residue embeddings, conserved multiple sequence alignment matrices, and pathogenicity score matrices; and projecting temperature weights for target protein locations based on the residue pair representations.

[0226] Similarly, in one or more embodiments, determining a residue pair representation includes determining the residue pair representation based on a combination of amino acid pairwise index difference embeddings, amino acid pairwise atomic distance matrices, reference residue embeddings, conserved multiple sequence alignment matrices, and pathogenicity score matrices. The operation for generating a graphical visualization of temperature weights for target protein locations may further include generating a modified residue pair representation using one or more triangular attention layers, determining a diagonal residue pair representation from the modified residue pair representation, and projecting temperature weights for target protein locations from the diagonal residue pair representation.

[0227] As further shown in Figure 14, a series of operations 1400 includes operation 1406 which generates data representing a graphical visualization depicting the values ​​of temperature weights corresponding to the target protein locations. As further suggested above, in some cases, generating data representing a graphical visualization includes generating data representing a graphical visualization depicting the values ​​of temperature weights indicating the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model for the target protein locations. Also as suggested above, in one or more embodiments, generating data representing a graphical visualization includes generating data indicators for different colors, patterns, or shading representing different values ​​of temperature weights for the target protein locations.

[0228] Referring now to Figure 15, this figure illustrates a flowchart of a series of operations 1500 for training a temperature prediction machine learning model to generate temperature weights, according to one or more embodiments of the present disclosure. While Figure 15 illustrates operations according to one embodiment, alternative embodiments may omit, add, rearrange, and / or modify any of the operations shown in Figure 15. The operations in Figure 15 may be performed as part of a method. Alternatively, a non-temporary computer-readable storage medium may, when executed by one or more processors, contain instructions that cause a computing device or system to perform the operations shown in Figure 15. In a further embodiment, the system comprises at least one processor and a non-temporary computer-readable medium containing instructions that, when executed by one or more processors, cause the system to perform the operations in Figure 15.

[0229] As shown in Figure 15, the sequence of operations 1500 includes an operation 1502 that determines an initial pathogenicity score. In particular, in some embodiments, operation 1502 includes using a mutant pathogenicity machine learning model to determine an initial pathogenicity score for a target amino acid at a target protein location within the protein, based on the amino acid sequence of the protein.

[0230] As further shown in Figure 15, the sequence of operations 1500 includes at least an operation 1504 that identifies temperature weights. In particular, in some embodiments, operation 1504 includes, for a protein, determining temperature weights that estimate the temperature of at least a mutant pathogenicity machine learning model, using a temperature prediction machine learning model. As described above, in some embodiments, the temperature prediction machine learning model used to determine at least the temperature weights includes a multilayer perceptron (MLP), a convolutional neural network (CNN), a triangular attention neural network, a recurrent neural network (RNN), a long-short-term memory (LSTM), a transformer machine learning model, or a decision tree model.

[0231] For example, in some embodiments, determining temperature weights includes determining each temperature weight to estimate the certainty of each pathogenicity score generated by the mutant pathogenicity machine learning model for a target protein location. In contrast, in certain embodiments, determining temperature weights includes determining, for a protein, the degree of certainty of the pathogenicity score generated by the mutant pathogenicity machine learning model for any given protein location within the protein.

[0232] In relation to this, in certain embodiments, determining the temperature weights at least includes determining at least positive temperature weights by applying a nonlinear activation function to at least the initial weights. In some cases, determining the temperature weights at least includes determining the average temperature weight from the initial temperature weight at a target protein location among the target protein locations. Furthermore, in certain embodiments, determining the temperature weights at least includes determining the average temperature weight from the initial temperature weights for various amino acids at the target protein location using a Gaussian blur model, median filter, or bilateral filter.

[0233] In addition, or alternatively, in certain embodiments, determining temperature weights includes, at least, using a triangular attention neural network as a temperature prediction machine learning model to determine temperature weights for at least a protein, by determining one or more of the following: amino acid pairwise index difference embeddings representing the differences between amino acids in the amino acid sequence of the protein; amino acid pairwise atomic distance matrices representing the pairwise distances between atoms in the protein; reference residue embeddings representing reference residues for the protein; conserved multiple sequence alignment matrices representing multiple sequence alignments for proteins from multiple species; and pathogenicity score matrices representing pathogenicity scores for amino acids in the protein generated by a mutant pathogenicity machine learning model; determining residue pair representations based on one or more of the amino acid pairwise index difference embeddings, amino acid pairwise atomic distance matrices, reference residue embeddings, conserved multiple sequence alignment matrices, and pathogenicity score matrices; and projecting temperature weights for target protein locations within the protein based on the residue pair representations.

[0234] In relation to this, in some embodiments, determining a residue pair representation involves determining the residue pair representation based on a combination of amino acid pairwise index difference embeddings, amino acid pairwise atomic distance matrices, reference residue embeddings, conserved multiple sequence alignment matrices, and pathogenicity score matrices. The sequence of operations 1500 may further include generating a modified residue pair representation using one or more triangular attention layers, determining a diagonal residue pair representation from the modified residue pair representation, and projecting temperature weights for target protein locations from the diagonal residue pair representation.

[0235] As further shown in FIG. 15, a series of operations 1500 includes an operation 1506 of generating a calibrated pathogenicity score based on an initial pathogenicity score and at least one temperature weight. In particular, in certain embodiments, operation 1506 includes generating a calibrated pathogenicity score for a target amino acid at a target protein position based on the initial pathogenicity score and at least the temperature weight.

[0236] As further shown in FIG. 15, a series of operations 1500 includes an operation 1508 of determining a calibration score difference. In particular, in certain embodiments, operation 1508 includes determining a calibrated score difference between a first set of calibrated pathogenicity scores for known benign amino acids as at least part of a protein and a second set of calibrated pathogenicity scores for unknown pathogenic amino acids as at least part of the protein.

[0237] In some cases, for example, determining a calibrated score difference includes determining a calibrated score difference between each of a first set of calibrated pathogenicity scores for known benign amino acids and each of a second set of calibrated pathogenicity scores for unknown pathogenic amino acids. In this regard, in certain embodiments, determining a calibrated score difference using a hybrid loss function includes determining the calibrated score difference as a loss generated by the hybrid loss function based on the calibrated score difference exceeding zero, or determining the hyperbolic tangent of the calibrated score difference as a loss generated by the hybrid loss function based on the calibrated score difference being 0 or less.

[0238] As suggested above, in some embodiments, determining a calibration score difference is done by determining the difference between a first set of calibrated pathogenicity scores for known benign amino acids in a set of protein positions within a set of proteins and a second set of calibrated pathogenicity scores for unknown pathogenic amino acids in that set of protein positions within that set of proteins.

[0239] As further shown in Figure 15, the sequence of operations 1500 includes operation 1510, which adjusts the parameters of the temperature prediction machine learning model. In particular, in certain embodiments, operation 1510 includes adjusting the parameters of the temperature prediction machine learning model based on calibrated score differences.

[0240] As further suggested above, tuning the parameters of a temperature prediction machine learning model involves tuning the parameters of the temperature prediction machine learning model to train it to generate temperature weights that facilitate the distinction between benign and pathogenic mutant amino acids at a given protein location.

[0241] The components of the calibrated pathogenicity prediction system 104 may include software, hardware, or both. For example, the components of the calibrated pathogenicity prediction system 104 may include one or more instructions that are stored on a computer-readable storage medium and are executable by the processor of one or more computing devices (e.g., client device 110). When executed by one or more processors, the computer-executable instructions of the calibrated pathogenicity prediction system 104 can cause the computing device to perform the foam detection method described herein. Alternatively, the components of the calibrated pathogenicity prediction system 104 may include hardware such as a dedicated processing unit for performing a particular function or group of functions. Additionally or alternatively, the components of the calibrated pathogenicity prediction system 104 may include a combination of computer-executable instructions and hardware.

[0242] Furthermore, the components of the Calibrated Pathogenicity Prediction System 104 that perform the functions described herein with respect to the Calibrated Pathogenicity Prediction System 104 may be implemented, for example, as part of a standalone application, as a module of an application, as a plug-in to an application, as a single or multiple library functions that can be invoked by other applications, and / or as a cloud computing model. Accordingly, the components of the Calibrated Pathogenicity Prediction System 104 may be implemented as part of a standalone application on a personal computing device or a mobile device. Additionally or alternatively, the components of the Calibrated Pathogenicity Prediction System 104 may be implemented in any application that provides sequencing services, including but not limited to Illumina PrimateAI, Illumina PrimateAI1D, Illumina PrimateAI2D, Illumina PrimateAI3D, or Illumina TruSight software. "Illumina," "PrimateAI," "PrimateAI1D," "PrimateAI2D," "PrimateAI3D," and "TruSight" are registered trademarks or trademarks of Illumina Corporation in the United States and / or other countries.

[0243] Embodiments of the present disclosure may include, or utilize, a dedicated or general-purpose computer, including, for example, one or more processors and system memory, computer hardware, as will be discussed in more detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be embodied in a non-temporary computer-readable medium and at least partially implemented as instructions executable by one or more computing devices (e.g., any of the media content access devices described herein). Generally, a processor (e.g., a microprocessor) receives instructions from a non-temporary computer-readable medium (e.g., memory), executes those instructions, and thereby performs one or more processes, including one or more of the processes described herein.

[0244] A computer-readable medium can be any available medium that can be accessed by a general-purpose computer system or a dedicated computer system. A computer-readable medium that stores computer-executable instructions is a non-temporary computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Thus, embodiments of the present disclosure may include, but are not limited to, two distinctly different types of computer-readable mediums: a non-temporary computer-readable storage medium (device) and a transmission medium.

[0245] Non-temporary computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid-state drives (SSDs) (e.g., RAM-based), flash memory, phase-change memory (PCM), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or dedicated computer.

[0246] A “network” is defined as one or more data links that enable the transfer of electronic data between computer systems and / or modules and / or other electronic devices. When information is transferred to or provided to a computer via a network or another communication connection (either hardwired, wireless, or a combination of hardwired and wireless), the computer appropriately recognizes the connection as a transmission medium. A transmission medium can be used to carry desired program code means in the form of computer-executable instructions or data structures and may include networks and / or data links that can be accessed by general-purpose or dedicated computers. The above combinations should also be included within the scope of computer-readable media.

[0247] Furthermore, upon reaching various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to a non-temporary computer-readable storage medium (device) (or vice versa). For example, computer-executable instructions or data structures received via a network or data link may be buffered in RAM within a network interface module (e.g., NIC) and then ultimately transferred to computer system RAM and / or a less volatile computer storage medium (device) within the computer system. Therefore, it should be understood that non-temporary computer-readable storage media (devices) can be included in computer system components that also (or more primarily) utilize the transmission medium.

[0248] Computer executable instructions include instructions and data that, when executed by a processor, cause a general-purpose computer, a dedicated computer, or a dedicated processing unit to perform a particular function or group of functions. In some embodiments, computer executable instructions are executed on a general-purpose computer and transform the general-purpose computer into a dedicated computer implementing the elements of the Disclosure. Computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, or even source code. While the subject matter is described in language specific to structural features and / or methodological behavior, it should be understood that the subject matter as defined in the appended claims is not necessarily limited to the described features or behaviors. Rather, the described features and behaviors are disclosed as exemplary forms that implement the claims.

[0249] Those skilled in the art will understand that the disclosure can be implemented in network computing environments having many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure can also be implemented in distributed system environments where both local and remote computer systems linked over a network (by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules can reside in both local and remote memory storage devices.

[0250] Embodiments of this disclosure can also be implemented in a cloud computing environment. In this specification, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be adopted in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly configured via virtualization, exposed with low administrative effort or service provider interaction, and then scaled accordingly.

[0251] A cloud computing model can consist of various characteristics such as on-demand self-service, wide-area network access, resource pooling, rapid resilience, and measured service. A cloud computing model can also expose various service models, such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). A cloud computing model can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, and hybrid cloud. In this specification and in the claims, “cloud computing environment” refers to an environment in which cloud computing is employed.

[0252] Figure 16 shows a block diagram of a computing device 1600 that may be configured to perform one or more of the processes described above. It will be understood that one or more computing devices, such as computing device 1600, can implement the calibrated pathogenicity prediction system 104. As shown by Figure 16, computing device 1600 may include a processor 1602, memory 1604, storage device 1606, I / O interface 1608, and communication interface 1610, which may be coupled together communicatively via a communication infrastructure 1612. In certain embodiments, computing device 1600 may include fewer or more components than those shown in Figure 16. The following paragraphs describe in more detail the components of computing device 1600 shown in Figure 16.

[0253] In one or more embodiments, the processor 1602 includes hardware for executing instructions, such as instructions that constitute a computer program. For example, but not limited to, to execute instructions for dynamically modifying a workflow, the processor 1602 may retrieve (or fetch) instructions from internal registers, internal cache, memory 1604, or storage device 1606, decode them, and execute them. Memory 1604 may be volatile or non-volatile memory used to store data, metadata, and programs for execution by the processor. Storage device 1606 includes storage, such as a hard disk, flash disk drive, or other digital storage device, for storing data or instructions for carrying out the methods described herein.

[0254] The I / O interface 1608 allows a user to provide input to and receive output from the computing device 1600, and separately, to transfer and receive data therefrom. The I / O interface 1608 may include a mouse, keypad or keyboard, touchscreen, camera, optical scanner, network interface, modem, other known I / O devices, or a combination of such I / O interfaces. The I / O interface 1608 may include, but is not limited to, one or more devices for presenting output to the user, including a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In a particular embodiment, the I / O interface 1608 is configured to provide graphical data to the display and present it to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be useful in a particular embodiment.

[0255] The communication interface 1610 may include hardware, software, or both. In any case, the communication interface 1610 may provide one or more interfaces for communication (e.g., packet-based communication) between the computing device 1600 and one or more other computing devices or networks. For example, the communication interface 1610 may include, but is not limited to, a network interface controller (NIC) or network adapter for communication with Ethernet or other wired-based networks, or a wireless NIC (WNIC) or wireless adapter for communication with wireless networks such as Wi-Fi.

[0256] Alternatively, the communication interface 1610 can facilitate communication with various types of wired or wireless networks. The communication interface 1610 can also facilitate communication using various communication protocols. The communication infrastructure 1612 may also include hardware, software, or both that connect the components of the computing device 1600 to one another. For example, the communication interface 1610 may enable multiple computing devices connected by a particular infrastructure to communicate with each other using one or more networks and / or protocols to carry out one or more aspects of the processes described herein. For example, a sequencing process may enable multiple devices (e.g., client devices, sequencing devices, and server devices) to exchange information such as sequencing data and error notifications.

[0257] In the aforementioned specification, the disclosure was described with reference to certain exemplary embodiments. Various embodiments and aspects of the disclosure are described with reference to the details considered herein, and the accompanying drawings illustrate various embodiments. The above description and drawings are illustrative of the disclosure and should not be construed as limiting the disclosure. Numerous specific details are described in order to provide a complete understanding of the various embodiments of the disclosure.

[0258] This disclosure may be embodied in other specific forms without departing from its spirit or essential features. The embodiments described herein should be considered in all respects to be illustrative and not limiting. For example, the methods described herein may be carried out using fewer or more steps / operations, or the steps / operations may be carried out in a different order. In addition, the steps / operations described herein may be repeated or carried out in parallel with each other, or in parallel with different occurrences of the same or similar steps / operations. Accordingly, the scope of this application is indicated by the appended claims rather than by the foregoing description. All changes included in the meaning of the claims and equivalents are encompassed within those scopes.

Claims

1. It is a system, At least one processor, A non-temporary computer-readable medium containing instructions, wherein the instructions, when executed by the at least one processor, are transmitted to the system. Using a mutant pathogenicity machine learning model, the initial pathogenicity score for target amino acids at target protein locations within a protein is determined based on the amino acid sequence of the protein. For the aforementioned protein, identify the temperature weights that estimate the temperature in the aforementioned mutant pathogenicity machine learning model. A system for generating a calibrated pathogenicity score for the target amino acid at the target protein location, based on the initial pathogenicity score and the temperature weight.

2. The system according to claim 1, further comprising instructions, the instructions causing the system to identify the temperature weights for the target protein location of the protein when executed by the at least one processor.

3. The system according to claim 1, further comprising instructions, the instructions, when executed by the at least one processor, causing the system to identify the temperature weights by utilizing a temperature prediction machine learning model to determine the temperature weights for the protein based on the initial pathogenicity score and the amino acid sequence or nucleotide sequence corresponding to the protein.

4. The system according to claim 3, wherein the temperature prediction machine learning model used to determine the temperature weights includes a multilayer perceptron (MLP), a convolutional neural network (CNN), a triangular attention neural network, a recurrent neural network (RNN), a long-term short-term memory (LSTM), a transformer machine learning model, or a decision tree model.

5. The system according to claim 1, further comprising an instruction, the instruction causing the system to identify the temperature weights by applying a nonlinear activation function to the initial weights to determine positive temperature weights when executed by the at least one processor.

6. The system according to claim 1, further comprising instructions, the instructions, when executed by the at least one processor, causing the system to identify the temperature weights by identifying weights that estimate the degree of certainty for the protein or the target protein location to a pathogenicity score output by the mutant pathogenicity machine learning model.

7. The system according to claim 1, further comprising instructions, the instructions, when executed by the at least one processor, causing the system to generate a graphical visualization of the temperature weights indicating, for display, the degree of certainty for the protein or the target protein location to the pathogenicity score output by the mutant pathogenicity machine learning model.

8. The system according to claim 1, further comprising instructions, the instructions causing the system to identify the temperature weights by determining the average temperature weights from the initial temperature weights at the target protein location when executed by the at least one processor.

9. The system according to claim 8, further comprising instructions, which, when executed by the at least one processor, cause the system to determine the temperature weights by utilizing a Gaussian blur model, a median filter, or a bilateral filter, and to determine the average temperature weights from the initial temperature weights for various amino acids at the target protein location.

10. The system according to claim 1, wherein the mutant pathogenicity machine learning model used to generate the initial pathogenicity score includes a transformer machine learning model, a convolutional neural network (CNN), a sequence-to-sequence model, a variational autoencoder (VAE), a multilayer perceptron (MLP), a recurrent neural network (RNN), a long-short-term memory (LSTM), or a decision tree model.

11. The system further includes instructions, which, when executed by the at least one processor, use a triangular attention neural network to determine the temperature weights for the protein. Determine one or more of the following: an amino acid pairwise index difference embedding representing the difference between amino acids in the amino acid sequence of the protein; an amino acid pairwise atomic distance matrix representing the pairwise distance between atoms in the protein; a reference residue embedding representing a reference residue for the protein; a conserved multiple sequence alignment matrix representing multiple sequence alignments for the protein from multiple species; and a pathogenicity score matrix representing the pathogenicity score for the amino acids in the protein generated by the mutant pathogenicity machine learning model. Determining residue pair representations based on one or more of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix. Based on the aforementioned residue pair representation, project temperature weights for protein positions, and The system according to claim 1, wherein the temperature weight for the position of the target protein within the protein is identified from the temperature weights to determine the position.

12. The system further includes instructions, which, when executed by the at least one processor, Based on the combination of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix, the residue pair representation is determined. Using one or more triangular attention layers, a modified residue pair representation is generated. From the modified residue pair representation, the diagonal residue pair representation is determined. The system according to claim 11, wherein the temperature weights for protein positions are projected from the diagonal residue pair representation.

13. The system further includes instructions, which, when executed by the at least one processor, Using an additional mutant pathogenicity machine learning model, an additional pathogenicity score is generated for the target amino acid at the target protein location. The additional pathogenicity score and the calibrated pathogenicity score for the target amino acid are normalized. The system according to claim 1, comprising combining the normalized additional pathogenicity score and the normalized calibrated pathogenicity score to generate a combined pathogenicity score for the target amino acid at the target protein site.

14. The system further includes instructions, which, when executed by the at least one processor, Using an additional mutant pathogenicity machine learning model, an additional pathogenicity score is generated for the target amino acid at the target protein location. The system according to claim 1, wherein a meta-mutant pathogenicity machine learning model is used to generate a refined pathogenicity score for the target amino acid at the target protein location based on the calibrated pathogenicity score and the additional pathogenicity score.

15. The system further includes instructions, which, when executed by the at least one processor, The initial pathogenicity score for a specific mutant amino acid at the target protein site is determined based on data representing the specific mutant amino acid and the amino acid sequence for the protein. To generate the additional pathogenicity score for the specific mutant amino acid at the target protein site, The system according to claim 14, which generates the refined pathogenicity score for the specific mutant amino acid at the target protein site.

16. A computer implementation method, Using a mutant pathogenicity machine learning model, the initial pathogenicity score for a target amino acid at a target protein location within a protein is determined based on the amino acid sequence of the said protein. With respect to the aforementioned protein, the temperature weights used to estimate the temperature in the aforementioned mutant pathogenicity machine learning model are identified, A computer-aided method comprising generating a calibrated pathogenicity score for the target amino acid at the target protein location based on the initial pathogenicity score and the temperature weight.

17. The computer implementation method according to claim 16, wherein identifying the temperature weights includes identifying the temperature weights with respect to the target protein position of the protein.

18. The computer implementation method according to claim 16, wherein identifying the temperature weights includes determining the temperature weights for the protein based on the initial pathogenicity score and the amino acid sequence or nucleotide sequence corresponding to the protein, using a temperature prediction machine learning model.

19. The computer implementation method according to claim 18, wherein the temperature prediction machine learning model used to determine the temperature weight includes a multilayer perceptron (MLP), a convolutional neural network (CNN), a triangular attention neural network, a recurrent neural network (RNN), a long-term short-term memory (LSTM), a transformer machine learning model, or a decision tree model.

20. The computer implementation method according to claim 16, wherein identifying the temperature weights includes applying a nonlinear activation function to the initial weights to determine the positive temperature weights.

21. The computer implementation method according to claim 16, wherein identifying the temperature weights includes identifying weights that estimate the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model for the protein or the target protein location.

22. The computer implementation method according to claim 16, further comprising generating a graphical visualization of the temperature weights indicating the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model for the location of the protein or the target protein, for display purposes.

23. The computer implementation method according to claim 16, wherein identifying the temperature weights includes determining the average temperature weights from the initial temperature weights at the target protein location.

24. The computer implementation method according to claim 23, wherein determining the temperature weights includes determining the average temperature weights from the initial temperature weights for various amino acids at the target protein location using a Gaussian blur model, a median filter, or a bilateral filter.

25. The computer implementation method according to claim 16, wherein the variant pathogenicity machine learning model used to generate the initial pathogenicity score includes a transformer machine learning model, a convolutional neural network (CNN), a sequence-to-sequence model, a variational autoencoder (VAE), a multilayer perceptron (MLP), a recurrent neural network (RNN), a long-short-term memory (LSTM), or a decision tree model.

26. Identifying the aforementioned temperature weights is done using a triangular attention neural network to determine the temperature weights for the protein. Determine one or more of the following: an amino acid pairwise index difference embedding representing the difference between amino acids in the amino acid sequence of the protein; an amino acid pairwise atomic distance matrix representing the pairwise distance between atoms in the protein; a reference residue embedding representing a reference residue for the protein; a conserved multiple sequence alignment matrix representing multiple sequence alignments for the protein from multiple species; and a pathogenicity score matrix representing the pathogenicity score for the amino acids in the protein generated by the mutant pathogenicity machine learning model. Determining residue pair representations based on one or more of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix. Based on the aforementioned residue pair representation, project temperature weights for protein positions, and The computer implementation method according to claim 16, further comprising determining the temperature weight for the location of the target protein within the protein from among the temperature weights.

27. The residue pair representation is determined based on a combination of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix. Using one or more triangular attention layers to generate a modified residue pair representation, From the modified residue pair representation, the diagonal residue pair representation is determined, Projecting the temperature weights for protein positions from the aforementioned diagonal residue pair representation, The computer implementation method according to claim 26, further comprising:

28. Using an additional mutant pathogenicity machine learning model, an additional pathogenicity score is generated for the target amino acid at the target protein location. Normalizing the additional pathogenicity score and the calibrated pathogenicity score for the target amino acid, The normalized additional pathogenicity score and the normalized calibrated pathogenicity score are combined to generate a combined pathogenicity score for the target amino acid at the target protein site, The computer implementation method according to claim 16, further comprising:

29. Using an additional mutant pathogenicity machine learning model, an additional pathogenicity score is generated for the target amino acid at the target protein location. Using a meta-mutant pathogenicity machine learning model, a refined pathogenicity score for the target amino acid at the target protein location is generated based on the calibrated pathogenicity score and the additional pathogenicity score. The computer implementation method according to claim 16, further comprising:

30. The initial pathogenicity score for a specific mutant amino acid at the target protein site is determined based on data representing the specific mutant amino acid and the amino acid sequence for the protein. To generate the additional pathogenicity score for the specific mutant amino acid at the target protein site, To generate the refined pathogenicity score for the specific mutant amino acid at the target protein site, The computer implementation method according to claim 29, further comprising:

31. It is a system, At least one processor, A non-temporary computer-readable medium containing instructions, wherein the instructions, when executed by the at least one processor, are transmitted to the system. To identify the target protein and the position of the target protein within the target protein, Determine the temperature weights for estimating the temperature of the mutant pathogenicity machine learning model at the target protein location within the target protein. A system that generates data representing a graphical visualization depicting the temperature weight values ​​corresponding to the target protein positions.

32. The system further includes instructions, which, when executed by the at least one processor, The temperature weights are determined by accessing weights from the database that estimate the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model for the target protein location. The system according to claim 31, which generates data representing the graphical visualization that depicts the values ​​of the temperature weights indicating the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model for the location of the target protein.

33. The system according to claim 32, further comprising instructions, the instructions causing the system to generate the data representing the graphical visualization by generating data indicators for different colors, patterns, or shading that represent different values ​​of the temperature weights for the target protein location, when executed by the at least one processor.

34. The system according to claim 32, further comprising an instruction, the instruction, when executed by the at least one processor, causing the system to determine the temperature weight by utilizing a temperature prediction machine learning model to determine the temperature weight corresponding to the target protein location based on an initial pathogenicity score for a target amino acid at the target protein location and an amino acid sequence or nucleotide sequence corresponding to the target protein.

35. The system further includes instructions, which, when executed by the at least one processor, use a triangular attention neural network to determine the temperature weights for the target protein location. Determine one or more of the following: an amino acid pairwise index difference embedding representing the difference between amino acids in the amino acid sequence of the target protein; an amino acid pairwise atomic distance matrix representing the pairwise distance between atoms in the target protein; a reference residue embedding representing a reference residue for the target protein; a conserved multiple sequence alignment matrix representing multiple sequence alignments for the target protein from multiple species; and a pathogenicity score matrix representing the pathogenicity score for amino acids in the target protein generated by the mutant pathogenicity machine learning model. Determining residue pair representations based on one or more of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix, and The system according to claim 32, wherein the temperature weights for the target protein position are determined by projecting them based on the residue pair representation.

36. The system further includes instructions, which, when executed by the at least one processor, Based on the combination of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix, the residue pair representation is determined. Using one or more triangular attention layers, a modified residue pair representation is generated. From the modified residue pair representation, the diagonal residue pair representation is determined. The system according to claim 35, wherein the temperature weights for the target protein position are projected from the diagonal residue pair representation.

37. A computer implementation method, Identifying the target protein and the location of the target protein within the target protein, To determine the temperature weights for estimating the temperature of the mutant pathogenicity machine learning model at the target protein location within the target protein, A computer method comprising generating data representing a graphical visualization depicting the temperature weight values ​​corresponding to the target protein positions.

38. Determining the temperature weights includes accessing weights from a database that estimate the degree of certainty for the target protein location relative to the pathogenicity score output by the mutant pathogenicity machine learning model, Generating the data representing the graphical visualization means generating the data representing the graphical visualization that depicts the values ​​of the temperature weights indicating the degree of certainty for the pathogenicity score output by the mutant pathogenicity machine learning model for the target protein location, The computer implementation method according to claim 37, including the method described in claim 37.

39. The computer implementation method according to claim 37, wherein generating the data representing the graphical visualization includes generating data indicators for different colors, patterns, or shading representing different values ​​of the temperature weights for the target protein location.

40. The computer implementation method according to claim 37, wherein determining the temperature weights includes using a temperature prediction machine learning model to determine the temperature weights corresponding to the target protein location based on an initial pathogenicity score for the target amino acid at the target protein location and the amino acid sequence or nucleotide sequence corresponding to the target protein.

41. Determining the temperature weights involves using a triangular attention neural network to determine the temperature weights for the target protein location. Determine one or more of the following: an amino acid pairwise index difference embedding representing the difference between amino acids in the amino acid sequence of the target protein; an amino acid pairwise atomic distance matrix representing the pairwise distance between atoms in the target protein; a reference residue embedding representing a reference residue for the target protein; a conserved multiple sequence alignment matrix representing multiple sequence alignments for the target protein from multiple species; and a pathogenicity score matrix representing the pathogenicity score for amino acids in the target protein generated by the mutant pathogenicity machine learning model. Determining residue pair representations based on one or more of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix, and The computer implementation method according to claim 37, comprising determining by projecting the temperature weights for the target protein location based on the residue pair representation.

42. The residue pair representation is determined based on a combination of the amino acid pairwise index difference embedding, the amino acid pairwise atomic distance matrix, the reference residue embedding, the conserved multiple sequence alignment matrix, and the pathogenicity score matrix. Using one or more triangular attention layers to generate a modified residue pair representation, From the modified residue pair representation, the diagonal residue pair representation is determined, Projecting the temperature weights for the target protein position from the aforementioned diagonal residue pair representation, The computer implementation method according to claim 41, further comprising:

Images