Machine learning to predict cancer genotype and treatment response using digital histopathology images

Abstract

A computerized system and method for digital histopathology analysis is disclosed. In one embodiment, a series of deep learning networks are used that train in succession on successively increasing sets of relevant data. In some examples, learned parameters from at least a portion of one deep learning network are transferred to the next deep learning network in the series of deep learning networks. In some examples, at least one of the deep learning networks includes a self-supervised learning network. In some examples, at least one of the deep learning networks includes an attention-based learning network. These and other examples and details are disclosed herein in various contexts including, for example, using histopathology images to evaluate genotypes of cancer tissue (e.g., bladder cancer, prostate cancer, or lung cancer). In some examples, the context is to help predict the presence or absence of a particular cancer genotype and / or predict a patient's response to a new treatment.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 63 / 246,178, filed September 20, 2021, and U.S. Provisional Application No. 63 / 301,023, filed January 19, 2022, the contents of which are incorporated herein by reference. [Background technology]

[0002] The present disclosure relates generally to techniques for computerized prediction of cancer genotype and treatment response using tissue images. Summary of the Invention

[0003] Recruiting patients for clinical trials for drugs that target patients with specific cancer mutations is slow and expensive. One example is in the context of treating bladder cancer with erdafitnib. Another example is in the context of treating prostate cancer with niraparib. Another example is in the context of treating lung cancer with amivantamab. Once a cancer patient is diagnosed, the patient may be referred to an available clinical trial site. A biopsy of the patient is taken as part of the diagnosis, and molecular screening is performed to test for the specific genetic mutation / alteration that the clinical trial drug targets. However, such screening can be expensive and can take an undesirably long time to obtain results. Furthermore, the testing requires irreplaceable tumor tissue. Some patients may decide to forego molecular screening (or undergo a different molecular test) if they perceive that they are unlikely to have one of the eligible mutations. Furthermore, the more quickly suitable candidates for clinical trials can be identified, the more quickly the efficacy of a potentially life-saving new treatment can be determined. For these and other reasons, there is a need for easier and more effective prediction of specific cancer genotypes, particularly in the context of candidate screening clinical trial participation.

[0004] Machine learning systems have been used to analyze digital histopathology images. However, these systems generally rely on a large set of relevant labeled training data to train the computerized system to assist in making an assessment, diagnosis, or prediction based on the histopathology image data. However, in the context of clinical trials for new and potentially promising treatments, a large set of labeled histopathology image data corresponding to patients undergoing such treatments and with known outcomes is not generally available.

[0005] Embodiments of the present disclosure include computerized pipeline systems and methods that can effectively learn to make predictions in situations where the most relevant set of histopathology images is necessarily relatively small. For example, a cohort of interest may include participants (or candidates for participation) in a clinical trial for a new therapeutic treatment for a particular disease. The new treatment may include, for example, administering a new drug, a new combination of drugs, and / or using a new protocol to treat the relevant disease. For example, a small data set may include images from only 100-200 patients, or even fewer than 100 patients.

[0006] The challenge posed by such small histopathology image datasets is that it is generally too difficult or impossible to effectively train a deep learning network from scratch to make sufficiently accurate predictions using such small amounts of training data. However, it may be possible to use such small datasets to fine-tune / train a deep learning network with a pre-trained feature extraction portion (and / or other portions) on other datasets, preferably a series of datasets starting with larger, less relevant datasets followed by progressively more relevant (often smaller) datasets.

[0007] For many diseases, such as cancer, there are publicly available large datasets (e.g., The Cancer Genome Atlas-"TCGA"-data) that may correspond to histopathology images (e.g., whole slide images) from more than 20,000 or even more than 30,000 individuals. While such datasets often identify the type of disease (e.g., type of cancer), they do not necessarily identify other specific information that may be required to predict a specific tumor genotype and / or to use fully supervised learning to predict response to various treatments. Thus, such datasets may be "unlabeled" in the sense that it may not be available to identify specific known information regarding treatment outcome, or even specific tumor genotypes within a specific type of cancer (e.g., specific genotypes for specific mutations associated with bladder, prostate, lung, etc.), and / or may be less relevant to the cohort of interest than other smaller datasets.

[0008] However, such large unlabeled datasets can be effectively used as part of a pipeline system for developing trained deep learning networks using successively more relevant (and potentially significantly smaller) datasets to develop trained networks for analyzing data corresponding to a cohort of interest, such as candidates or participants in a clinical trial for a particular new treatment.

[0009] In one embodiment of a pipeline system according to the present disclosure, a first deep learning network is used to perform self-supervised learning (not requiring labeled training data) on a large histopathology image dataset to begin training the feature extraction portion (and / or other portions) of the deep learning network. In one example, a control learning network is used. However, in other implementations consistent with the principles of the present disclosure, the first deep learning network in the pipeline may be configured to implement other types of self-supervised learning. It may, in other alternatives, be configured to implement learning types other than self-supervised learning.

[0010] In selected embodiments, additional deep learning networks in the pipeline can be successively trained using subsequent datasets of increasing relevance to the cohort of interest. These additional networks can be configured to use supervised learning, which in some examples includes attention-based multiple instance learning. Learned weights can be transferred from the feature extraction portion (and / or additional portion) of one network in the pipeline to the feature extraction portion (and / or additional portion) of the next network in the pipeline until a final trained network is provided that can improve analysis of relatively small datasets, for example datasets that include cohorts of interest that include candidates or participants in clinical trials of new treatments.

[0011] In a broad sense, embodiments of the present disclosure depict a training pipeline that includes the successive training of successive deep learning (or other machine learning) networks using successive datasets, from less relevant to more relevant with respect to a cohort of interest, to assess whether a particular patient is likely to respond to a new treatment, or to improve predictions related to assessing. In some embodiments, certain pre-processing techniques are used to improve performance in the context of a particular histopathology application.

[0012] In various embodiments, subjects predicted to exhibit an altered genotype may undergo further molecular testing to confirm the in silico results. In various embodiments, once confirmed through molecular testing, the subject may be provided with appropriate medical treatment or medical advice based on the altered genotype status. In various embodiments, once confirmed, these subjects may be considered eligible for enrollment in a clinical trial. In various embodiments, subjects predicted to exhibit a wild type genotype are likely to be ineligible for a clinical trial and do not need to undergo further molecular testing, thereby saving resources (e.g., time and money). In various embodiments, the methods and / or non-transitory computer readable media described herein may operate as Software as a Medical Device (SaMD), provided on disk or via download, or as web-based software. In such embodiments, the methods described herein may operate independently of a clinical trial environment.

[0013] Further aspects of these and other embodiments are described herein. [Brief description of the drawings]

[0014] [Figure 1] A computerized pipeline system is shown for developing a trained computerized deep learning network for genotype prediction and treatment response prediction using histopathology images corresponding to a cohort of interest including patients or candidates in clinical trials of new treatments. [Diagram 2] FIG. 2 is a block diagram showing the architecture of a self-supervised deep learning network of the embodiment of FIG. [Diagram 3] FIG. 2 is a block diagram showing the architecture of the attention-based learning network of the embodiment of FIG. 1. [Figure 4] 13 shows further details of pre-processing of histopathology image data in one embodiment. [Diagram 5]3 shows further details of the enhanced processing of histopathology image data implemented by the self-supervised learning network shown in FIG. 2. [Figure 6] 3 shows further details of the feature extraction network and the projection network of the self-supervised learning network shown in FIG. 2. [Figure 7] FIG. 1 illustrates a computer-implemented method for using a pipeline system such as that of FIG. 1 to provide a final trained deep learning network for predicting tumor genotype and / or predicting response in members of a cohort of interest to a new treatment. [Figure 8] 7 illustrates an example of a computer system 7000, one or more of which may be used to implement one or more of the devices, systems, and methods described herein. [Figure 9] 1 depicts an overview of a system environment for determining a subject's genotype prediction, according to some embodiments. [Figure 10] FIG. 10 depicts a block diagram of the genotype prediction system referenced in FIG. [Figure 11] 1 depicts an exemplary flow diagram for determining a subject's genotype prediction, according to some embodiments. [Figure 12] 1 is an exemplary flow process for determining a subject's genotype prediction, according to one embodiment. [Figure 13] 1 illustrates an exemplary computer for use in implementing various embodiments presented herein. [Figure 14] 1 depicts an exemplary process for generating patient-level genotype predictions using slide images. [Figure 15] 1 depicts another exemplary process for generating patient-level genotype predictions using slide images. [Figure 16A] This demonstrates the lack of cross-cohort generalizability when TCGA and Tempus samples are differentiated. [Figure 16B]13 shows the bias of the neural network against pen artifacts in distinguishing TCGA samples from Test #1 specimens when no pre-processing steps are implemented. [Figure 16C] 13 shows the improved treatment of the neural network for TCGA and Test #1 samples when the pre-processing step is implemented. [Figure 17A] 15 depicts the performance of a model deployed with the exemplary process of FIG. 14 compared to a model deployed with the exemplary process of FIG. [Figure 17B] FIG. 15 illustrates the performance of a model deployed on TCGA bladder cancer slides according to the exemplary process. [Figure 18] 1 depicts another exemplary process for generating patient-level genotype predictions using slide images. [Figure 19] 17 depicts the performance of a model deployed with the exemplary process of FIG. 18 compared to a model deployed with the exemplary process of FIG. 14 . [Figure 20] 5 and 18, respectively, demonstrating that the exemplary process of FIG. 18 is more robust against image artifacts such as pen marks. [Figure 21] 1 depicts another exemplary process for generating patient-level genotype predictions using slide images. [Figure 22] 1 depicts an exemplary pipeline workflow that includes several quality control steps to improve the percentage of high-quality images analyzed by the neural network.

[0015] Although the embodiments are described with reference to the above drawings, the drawings are intended to be exemplary and other embodiments are consistent with the spirit and scope of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Various embodiments are described more fully below with reference to the accompanying drawings, which form a part of this specification and which show, by way of illustration, specific examples of implementations of the embodiments. However, this specification may be embodied in a wide variety of forms and should not be construed as being limited to the embodiments set forth herein, but rather, these embodiments are provided so that this specification will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. In particular, this specification may be embodied as a method or device. Thus, any of the various embodiments herein may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Thus, the following description should not be construed in a limiting sense.

[0017] FIG. 1 illustrates a computerized pipeline system 1000 for developing a trained computerized deep learning network (or other computerized machine learning model) to make cancer genotype predictions, treatment response predictions, and / or other predictions using histopathology images corresponding to a cohort of interest, particularly a cohort of interest that may include, for example, patients or candidates in clinical trials of a new treatment (or new combination of treatments).

[0018] FIG. 1 is illustrated and explained using an example of a series of neural networks 111, 112, 113, and 114. However, the underlying principles are applicable to other deep learning or machine learning computer systems, regardless of whether they include any neural network processing layers. A "neural network" in this example simply means a computerized system that implements machine learning processing, including one or more processing layers or elements known as neural network layers or elements. In various examples, this may include a feedforward neural network layer (also known as a fully connected layer), a convolutional neural network layer (which may or may not include residual connections), a recurrent neural network, or other types of neural network layers or elements. Such a computerized machine learning system is referred to herein as a "neural network" if the system includes any processing layer or element known to be of the type of neural network processing layer or element. Such a system is referred to herein as a "neural network" even if it has many additional components or layers other than the neural network elements or layers. Again, however, those skilled in the art will appreciate that certain inventive principles of the illustrated embodiment are applicable to other types of machine learning systems embodying the present disclosure, even if those systems do not include any neural network components.

[0019] Specifically, the pipeline system 1000 comprises a series of respective neural networks (sometimes referred to as neural network modules, deep learning modules, and / or machine learning modules) 111, 112, 113, and 114. Respective histopathology image datasets 101, 102, 103, and 104 are used in succession to train the neural networks 111, 112, 113, and 114, respectively. In this example, the pipeline shows four neural networks in the pipeline system. However, in alternative implementations, more or fewer neural networks (or other types of machine learning modules) and more or fewer datasets may be used.

[0020] The pipeline system 1000 is necessarily tailored for use in relatively small situations, since the most relevant histopathology image dataset 104 relates to a target cohort that typically includes participants (or candidates for participation) in clinical trials for new therapeutic treatments for a particular disease, such as the administration of a new drug, a new drug combination, and / or the use of a new protocol to treat a related disease. For example, a small dataset such as dataset 104 may include images from only 100-200 patients. In many instances, it may correspond to histopathology images from 100 patients or even less than 100 patients (e.g., 40-50 patients, 51-60 patients, 61-70 patients, 71-80 patients, 81-90 patients, 91-100 patients).

[0021] The challenge posed by such small histopathology image datasets is that it is generally too difficult or impossible to effectively train a deep learning network from scratch to make sufficiently accurate predictions using such a small amount of training data. However, it is possible to use such small datasets to fine-tune / train a deep learning network whose feature extractor portion has been pre-trained (or whose weights have been transferred from a suitably pre-trained network) on other datasets, preferably a series of datasets starting with a large, less relevant dataset followed by progressively more relevant (usually smaller) datasets.

[0022] In the illustrated example, the dataset 101 is a publicly available large dataset. For many diseases, such as cancer, there are publicly available large datasets (e.g., The Cancer Genome Atlas-"TCGA"-data) that may correspond to histopathology images from more than 20,000 or even more than 30,000 individuals. Such datasets usually identify the type of cancer, but they do not necessarily identify other specific information that may be required to use fully supervised learning to predict the response to various treatments. Thus, such datasets may be "unlabeled" in the sense that they may not be able to identify specific known information about treatment outcomes, or even specific tumor genotypes within a specific type of cancer (e.g., lung).

[0023] Dataset 101 may be labeled for several purposes, but in the particular example described herein, it is not labeled for the specific purpose of training a treatment response classifier (or, if it has treatment response information, the corresponding treatment and / or cancer information is not sufficiently relevant to the cohort of interest to use the labels for useful training purposes).

[0024] However, even if the dataset 101 is unlabeled (or does not otherwise have sufficient relevant information for fully supervised learning), it may be effectively used for self-supervised learning, for example, to pre-train a feature extractor to extract relevant features from histopathology images (or other medical images) in the context of cancer or another other disease. Thus, in the pipeline system 1000, the first deep learning network 111 performs self-supervised learning (not requiring the use of labeled training data) to begin training the feature extraction portion of the deep learning network 111 using the dataset 101. As will be further explained in the context of FIG. 2 below, in this example, the deep learning network 111 is configured to implement contrastive learning. However, in other implementations consistent with the principles of the present disclosure, the first deep learning network in the pipeline may be configured to implement other types of self-supervised learning.

[0025] At least a portion of the learned parameters (which may be referred to as weights, kernel values, filter values, or other names) resulting from self-supervised learning using the control training neural network 111 and the histopathology image dataset 101 are transferred to a second network 112. In this example, the network 112 is an attention-based multiple-instance deep learning network.

[0026] In the pipeline system 1000, the second network 112 is trained with the histopathology image dataset 102. In this example, the histopathology image dataset 102 is a labeled dataset, and the network 112 performs attention-based supervised learning using the histopathology image dataset 102, as will be further explained in the context of FIG.

[0027] In the illustrated example, dataset 102 is a smaller, more relevant dataset than dataset 101 with more corresponding relevant information. For example, it may be a commercially available dataset (i.e., private) that includes treatment outcomes and / or molecular test results that identify genotypes for specific mutations associated with a particular cancer corresponding to the tissue in the histopathology image. It may be of the same disease or a similar disease context as the disease corresponding to the cohort of interest, but the treatment and other aspects surrounding the data are different. After the second network 112 is trained using the histopathology image dataset 102, at least some of the resulting learned parameters are transferred to a third deep learning network 113, which in this example is also an attention-based multiple instance deep learning network.

[0028] In the pipeline system 1000, the third network 113 is trained with a histopathology image dataset 103. In this example, the histopathology image dataset 103 is a labeled dataset corresponding to a clinical cohort that is not the cohort of interest, but is more relevant to the cohort of interest than dataset 102. In one example, the histopathology image dataset 103 corresponds to a clinical trial cohort that received a treatment that is somewhat related to, but not the same treatment as, the treatment administered to the cohort of interest. In this example, the clinical trial corresponding to dataset 103 has similar inclusion / exclusion criteria as a clinical trial for a new treatment administered to the cohort of interest.

[0029] After the third network 113 has been trained using the histopathology image dataset 103, at least some of the resulting learned parameters are transferred to the final deep learning network 114, which in this example is also an attention-based multi-instance deep learning network.

[0030] The final deep learning network 114 is fine-tuned (further trained) using a histopathology image dataset 104 that corresponds to at least a portion of a subject cohort whose members are in, or candidates for, clinical trials of a new treatment. In one example, the histopathology image dataset 104 corresponds to a portion of a subject cohort that has already received the new treatment and for which some data is available regarding response to the treatment.

[0031] After fine-tuning (i.e., training) of the final network 114 using the histopathology image dataset 104, the network 114 may then be used to help predict treatment response for other members of the cohort of interest using their corresponding histopathology images. In this manner, a final network such as network 114 developed using the pipeline system 1000 may enable improved participant selection in related ongoing clinical trials or future clinical trials involving the same or similar treatments.

[0032] The illustrated example of a pipeline system is described herein in terms of the relevant learned weights being transferred from one deep learning network to the next in the pipeline. However, one skilled in the art will understand that in some cases (e.g., when successive networks in the pipeline have the same architecture), this can be considered equivalent to training the same network first on one dataset and then on another dataset in succession, or, as another example, training the same network first on one dataset in succession and then reinitializing the weights of some parts of the network (e.g., the attention layer and classification layer, where the weights from the training of the previous dataset are not retained when starting the training of the next dataset in the pipeline), but reinitializing the weights of other parts (e.g., the feature extraction layer, where the weights from the training of the previous dataset are retained as starting values ​​for the training of the next dataset) when starting the training of the next dataset. Such examples are considered to be consistent with the spirit and scope of the present disclosure.

[0033] 2 is a block diagram illustrating the architecture of a self-supervised learning neural network 111 (an example of a deep learning network that corresponds to the first deep learning network in a pipeline system embodying principles underlying the present disclosure) of the pipeline system 1000. Specifically, the network 111 comprises a pre-processing block 201, an augmentation block 202, a feature extraction network 203, a projection network 204, and a contrastive learning module 205.

[0034] The contrast learning network 111 shown in FIG. 2 is similar to the contrast learning network disclosed in the applicant's co-pending U.S. Provisional Application No. 63 / 301,023, filed January 19, 2022, the entire contents of which are incorporated herein by reference. The contrast learning network 111 of the present disclosure is adapted to process data from histopathology images. In one embodiment, the system operates to train a feature extraction neural network 203 and a projection neural network 204 (networks 203 and 204 may be considered simply different sets of layers of a single neural network) to extract features from the histopathology images, which enable effective classification of tissue samples captured within the histopathology images (or other supervised learning tasks therefor).

[0035] Pre-processing module 201 receives data from the histopathology image dataset 101 and pre-processes it to provide tiles, such as tile 21 and tile 22, to enhancement module 202. Further details of pre-processing module 201 are discussed in the context of Figure 4. Tiles 22 and 21 are pre-processed pixel data corresponding to portions of the histopathology image from dataset 101.

[0036] The pre-processing module 201 receives histopathology image data from the histopathology image dataset 101. The histopathology images from the dataset 101 are typically digital images of stained (dyed) tissue samples that may be obtained, for example, from a biopsy of a patient suspected of having a particular disease. The pre-processing module 201 divides each histopathology image into multiple tiles, also referred to herein as patches.

[0037] The augmentation module 202 may receive the pre-processed tiles and perform two different executions of the augmentation process on each tile to generate two different augmented versions of each tile. For example, in the illustrated example, the augmentation module 202 receives two different tiles 21 and 22 and generates from tile 21 an augmented tile 21a and an augmented tile 21b corresponding to two different iterations of the augmentation process, and similarly generates from tile 22 an augmented tile 22a and an augmented tile 12b corresponding to two different iterations of the augmentation process. In one example, the augmentation process may include a series of steps performed on the tiles, such as random cropping followed by resizing to the original size, random horizontal and / or vertical flipping, color jitter using randomly selected multipliers within specified ranges for brightness, contrast, hue, and saturation, and / or random conversion of the image tile to grayscale (e.g., with a probability of 0.5). Such a process may be performed twice on the same tile (e.g., tile 21) to generate two different augmented versions of the tile (e.g., tile 21a and 21b). An example of a particular enhancement process is detailed in FIG.

[0038] Returning to the discussion of Figure 2, in the example shown, two augmented tile pairs are generated from each tile provided to the augmentation module 202. However, additional augmented tile versions may be generated from each tile.

[0039] The augmented tiles are processed by a feature extraction network 203. The feature extraction network 203 may be based on the convolutional and average pooling layers of ResNet34, a 34-layer residual network as described in Deep Residual Learning for Image Recognition by He et al., available at arXiv:1512.03385v1, 10 December 2015, and incorporated herein by reference (the "ResNet paper"). The classification layers (fully connected and softmax layers) of ResNet34 are not utilized, but are replaced by a projection network 204. However, this is by way of example only. Feature extraction networks of different sizes and / or types can be used in accordance with the principles of the present disclosure.

[0040] The feature extraction network 203 provides feature vectors (one per augmented tile) to a projection network 204. The projection network 204 (described further below in the context of FIG. 4) processes the feature vectors and provides projected feature vectors to a contrastive learning module 205.

[0041] The contrast learning module 205 applies a loss function to calculate a loss measure for feature vectors corresponding to processed tiles in the same batch. The loss measure can be the relevant difference between feature vectors derived from different augmented versions of the same tile in the batch, and also related to the difference between feature vectors corresponding to augmented versions of different tiles in the batch. The loss is then back-propagated through the projection network 204 and the feature extraction network 203 and used to adjust the learnable parameters (weights) of those networks. In simple terms, as the system learns to produce better feature vectors, it reduces the difference between feature vectors from different augmented versions of the same tile and increases the difference between feature vectors from augmented versions of different tiles.

[0042] The contrastive learning module 205 may implement the “SimCLR” contrastive learning process, which is based on the “A Simple Framework for Contrastive Learning of Visual Representation,” by Chen et al., Proceedings of the 37th International Conference on Visual Representation and Representation (ISR), 2014. th International Conference on Machine Learning, Vienna, PMLR 119, 2020, which is incorporated herein by reference (the "SimCLR paper"). Specifically, the NT-Xent loss referenced therein may be used as a loss function for the control learning module 205. In a particular example, a loss temperature of 0.1 is used. In alternative examples, other loss temperatures may be used. Also, in alternative examples, other loss functions may be used.

[0043] In a particular example, a batch size of 768 is used. In alternative examples, smaller or larger batch sizes are used. In other examples, the batch size ranges from about 250 to 4,000. Also, in one example, an Adam optimizer is used, with an unplanned learning rate of about 5×10^(-4).

[0044] 3 is a block diagram illustrating the architecture of the attention-based learning network 112 (an example of a deep learning network that corresponds to the second deep learning network in a pipeline system embodying the principles underlying the present disclosure) of the pipeline system 1000. Specifically, the network 112 comprises a pre-processing block 301, an augmentation module 302, a feature extraction network 303, an attention network 305, an aggregator 306, a classifier network 307, and a supervised learning module 308.

[0045] In the illustrated example, pre-processing block 301 is substantially similar to pre-processing block 301 of FIG. 2, which is discussed further below in the context of FIG.

[0046] In one example, the augmentation module 302 implements different and / or fewer augmentation steps than those implemented by the augmentation module 202 of FIG. 2. For example, the augmentation module 302 may implement only color jittering (also known as stain-based augmentation) but no other augmentation steps. In other examples, the same augmentation may be used. In other examples, some augmentation steps implemented by the augmentation module 302 are different from any of the augmentation module 202 steps. Those skilled in the art will recognize that in the context of attention-based learning networks 112-114 (or other deep learning networks), augmentation is potentially useful to improve training robustness. However, the same augmentation steps used in the context of self-supervised learning are not necessarily necessary or optimal for attention-based supervised learning. Also, in some examples, augmentation may or may not be used during supervised learning without necessarily departing from the spirit and scope of the present disclosure. In the illustrated embodiment, augmentation is applied to the training data used during training of the attention-based networks 112-114, but when the final fully trained network is applied to untrained histopathology image data to make decision-supporting predictions, no augmentation processing is applied to the analyzed histopathology image data.

[0047] In the illustrated example, the details of feature extraction network 303 are substantially the same as the details of feature extraction network 203 of Figure 2 described above. However, the architecture of these blocks may differ for different steps in a deep learning pipeline consistent with this disclosure. For example, if a first data set used to train a first network in the pipeline is different (e.g., different types of images, different data dimensions, etc.) from a second data set used to train the next network in the pipeline, the pre-processing and / or feature extraction portions of each network may be different.

[0048] In the example shown above, weights from feature extraction network 203 learned in training deep learning network 112 are transferred to feature extraction network 303 before training deep learning network 111 using dataset 102. In the current example, the feature extraction networks in each deep learning network 111, 112, 113, and 114 have the same architecture and dimensions. Thus, weights learned in training each network in the pipeline can be transferred to provide all of the initial weights to the feature extraction portion of the next deep learning network in the pipeline. However, even if the feature extraction networks are different for each pipeline network, weights from the previous network in the pipeline can be transferred to an equivalent position in the next network in the pipeline, e.g., any other position in the next network can be initialized to zero or some other value before training the next network. If necessary, the weights can be further processed during transfer via other techniques such as averaging, interpolation, or other methods, so that the benefits of training the feature extraction portion of the previous network can benefit the training of the next network in the pipeline.

[0049] Continuing with FIG. 3, feature extraction network 303 receives pre-processed tiles such as tiles 21 and 22 and processes them to extract feature vectors for each tile such as feature vectors 31 and 32. The feature vectors for each tile are fed to attention network 305 and aggregation function 306. In this example, attention network 305 is simply a feed-forward neural network with one or more fully connected layers with corresponding weights. The input layer size corresponds to the feature vector size, and the output layer provides a single attention value for each tile. The output scores (attention values) are normalized to be greater than or equal to 0 and less than or equal to 1.

[0050] The aggregator 306 multiplies each tile's feature vector by its corresponding attention score and then averages all feature vectors for a given histopathology image of a patient to produce a summarized feature vector 30, such that one summarized feature vector is produced for each patient's histopathology image. In the current example, the summarized feature vector 30 has the same dimensions as the feature vectors of the individual tiles. For example, the summarized feature vector 30 has the same dimensions as the feature vectors of a single tile, such as feature vector 31. However, in alternative examples, the size of the summarized feature vector does not necessarily match exactly the size of the feature vectors of the individual tiles.

[0051] The summarized feature vectors (one for each patient's histopathology image) are then provided to a classifier network 307 (which in this example includes a typical feed-forward network with one or more fully connected layers), which produces a predicted value (or class) for each summarized feature vector. The supervised learning module 308 uses a loss function to calculate a loss value based on the label corresponding to the associated histopathology image in the dataset 102 and based on the class value provided by the classifier network 307. Depending on the prediction type, different loss functions may be used. For example, if the prediction is in the form of survival time or time to recurrence (which may be one of many values, e.g., weeks or weeks), a cross-entropy loss function may be used. However, if the prediction is binary (e.g., response / non-response to treatment) or presence / absence of a particular mutation), a binary cross-entropy loss function may be used. These are merely examples. A variety of loss functions for supervised learning are within the capabilities of one of ordinary skill in the art, and the particular loss function used is not intended to limit the broader aspects of the present disclosure.

[0052] The supervised learning module 308 then back-propagates the loss value (sometimes called an error value) through the classifier network 307, the attention network 305, and the feature extraction network 303 and uses it to adjust the weights (learnable parameters) in those networks. Various known techniques for back-propagation and weight adjustment can be used, and learning rates and other learning parameters can be selected and modified to improve performance for a particular application.

[0053] In one example, the overall architecture of the deep learning network 112, after the pre-processing block 301, is based on aspects of the attention-based multi-instance deep learning architecture presented in Attention-based Deep Multiple Instance Learning by Isle et al., which was published on arXiv:1802.04712v4[cs.LG] on 28 June 2018 (referred to herein as “the attention-based MI learning paper” and incorporated herein by reference in its entirety (see, e.g., Figure 6c in the Appendix of that paper). However, this is only one example. In other examples, different network architectures may be used for different networks in a pipeline system consistent with the present disclosure (e.g., other attention-based architectures, or types of deep learning networks other than attention-based networks).

[0054] In the particular example of the pipeline system 1000 of Figure 1, the architecture of the third network 113 and the final network 114 are the same as that shown for the second network 112 shown in Figure 3 and are not separately discussed herein. However, it should be noted that the architecture of the pipeline networks from the second network to the final network need not be identical.

[0055] Further, in the particular example of the pipeline system 1000 of FIG. 1, only weights from a feature extraction network, such as feature extraction network 303, are transferred from one network in the pipeline to another. For example, as one skilled in the art will appreciate, during training of the network 112 shown in FIG. 3, weights are learned in the attention network 305 and the classifier network 307, as well as the feature extraction network 303. However, in this example, only the weights in the feature extraction network 303 are transferred to the feature extraction network of the next network in the pipeline (e.g., network 113 of FIG. 1). The weights for the attention and classifier networks in the subsequent deep learning network in the pipeline are reinitialized and retrained from scratch. However, the weights from the feature extraction network are transferred. Nevertheless, in alternative examples, the feature extraction network weights and other weights (e.g., weights from a classification network, such as classifier 307) may be transferred to the next network in the pipeline before training the next network, without necessarily departing from the principles of the present disclosure.

[0056] Figure 4 shows further details of pre-processing block 201 of Figure 2 (which are the same as the details of pre-processing block 301 of Figure 3). Specifically, pre-processing block 201 includes a tiling module 404, quality control (QC) screening for each tile module 402, and QC screening for each slide module 403 ("slide" is used interchangeably herein with "histopathology image" corresponding to a particular tissue sample).

[0057] The tiling module 401 receives a histopathology image 441 (from a histopathology dataset such as dataset 101) and divides the image into multiple tiles (sometimes referred to herein as "patches"). In one particular embodiment, the image is divided into non-overlapping tiles, each tile being 224 pixels by 224 pixels in size. However, in alternative examples, other sizes may be used.

[0058] The per-tile QC screening module 402 processes each tile to identify image quality. In one example, the image quality of a tile is determined by determining the percentage of tissue in the tile relative to background and relative to artifacts such as pen marks. An example of a process for doing this is provided at https: / / github.com / CODAIT / deep-histopath / tree / master / deephistopath / wsi. In one example, the following processing steps are performed by module 402 for each tile: 1. Get a low-resolution thumbnail. 2. Calculate a pen mask (e.g., based on identifying RGB pen colors) and a background mask (e.g., based on identifying "off colors" such as grays and greens that are not H&E stain colors). 3. Remove the calculated masks from the slide. 4. At high resolution, for each tile location, a) get the pen mask and background mask and apply them to the low-resolution version of the tile, b) calculate the percentage of tissue / pen / background in the tile, c) estimate color measurements, and d) calculate a QC score using the tissue percentages and color measurements. In one example, the QC score is determined by the following formula:

[0059]

number

[0060] In the above formula, the "colorfactor" is based on the color of the H&E stain, favoring hematoxylin over eosin. The "saturationfactor" is based on the fact that real tissues have a wide range of HSV saturation, and the "quantityfactor" corresponds to the amount of tissue in the tile image. In one example, the QC score using the above formula is scaled to be between 0 and 1, and tiles below a threshold score are rejected. In one example, the threshold is 0.75. In other examples, it is higher or lower than 0.75 (e.g., numbers between 0.6 and 0.75, numbers between 0.75 and 0.95, etc.).

[0061] A per-slide QC screening module 403 determines whether a sufficient number of tiles corresponding to a particular slide (i.e., a particular tissue image) pass QC screening 402. In one example, if an insufficient number of tiles pass screen 402, module 403 discards all tiles from that slide. Pre-processing 201 / 301 outputs tiles such as 42 and 41 from the image that passed screen 402 and were not rejected by screening 403. The process shown is merely one example of a QC screening process. Other QC screening processes can be used to determine tiles and slides with sufficient tissue images to be usefully used for further analysis without necessarily departing from this disclosure.

[0062] 5 shows a specific example of an augmentation process 5000 that may be performed by the augmentation module 202 of FIG. 2 in the context of self-supervised learning. The inventors have found that the parameters of the illustrated embodiment work particularly well for implementing contrast learning using histopathology slides. However, other processes and parameters may be used without necessarily departing from this disclosure.

[0063] Step 501 selects a new tile for augmentation. Step 502 randomly crops the image tile. Step 503 resizes it to its original size, in this example 224x224 pixels. Step 504 randomly modifies the color parameters of the tile (this action is called "color jitter" in the context of machine vision). In one example, randomly selected values ​​within predefined ranges are applied to the image for brightness, contrast, saturation, and hue. In one example, the predefined ranges within which brightness, contrast, and hue values ​​are randomly selected are each approximately 0.2 to 1.8, and the predefined range within which hue values ​​are randomly selected is approximately -0.2 to 0.2.

[0064] Next, step 505 randomly converts the tiles output by step 504 to grayscale. In this example, the random function uses a conversion probability of 0.5, i.e., the tile has approximately a 50% chance of being converted to grayscale in this step. In other examples, other probabilities are used.

[0065] In step 506, the output of step 505 is randomly blurred. In a particular embodiment, a probability of 0.5 is used to determine whether a tile is randomly blurred during enhancement.

[0066] Step 507 outputs an augmented version of the current tile. In step 508, if two augmented versions of the same tile have not yet been created, the method proceeds to step 502 to perform another iteration of the augmentation steps 502-506 on the same tile. However, if two augmented versions of the same tile have already been output, the method returns to step 501 to select a new tile for augmentation.

[0067] 6 shows further details of the feature extraction network 203 and the projection network 204 of FIG. 2. Specifically, in the example shown, the feature extraction network 203 uses the convolutional layers associated with conv1, conv2_x, conv3_x, conv4_x, conv5_x, and the average pooling layers, but does not use the fully connected layers of ResNet34 in the ResNet paper. It also uses all the skip connections associated with those layers in the ResNet paper. In this example, the feature extraction network 203 outputs a feature vector having 512 values ​​to the projection network 204.

[0068] The projection network 204 may comprise a linear layer 601, a batch normalization layer 602, an activation layer 604, and a linear layer 603. The linear layer 601 comprises an input layer and a fully connected hidden layer of 128 neurons (without activation function). Thus, the linear layer 601 outputs a feature vector of size 128 to the batch normalization layer 602, which performs a standard batch normalization process. After passing through the batch normalization layer 602, the feature vector passes through an activation function layer 604, which implements a nonlinear activation function such as ReLu, and then to the linear layer 603, which comprises an input layer of size 128 and a fully connected hidden layer of 512 neurons (without activation function), thus projecting the feature vector back to size 512.

[0069] In this context, it will be understood that "projection network" simply refers to the fact that the feature representation undergoes a change in the dimensionality used for the representation as it passes through the network. In this example, the projected features have the same dimensionality as the input features, but are obtained through a process in which the feature representation is first projected into a lower dimensional representation, and then, after batch normalization, is back-projected into a representation having the same dimensionality as the input representation. Alternatively, the use of a projection network such as that illustrated and described herein may be omitted, and better results are obtained by using a projection network and batch normalization before passing the resulting feature vector to the control learning module.

[0070] FIG. 7 shows a computer-implemented method 7000 for using a pipeline system, such as system 1000 of FIG. 1, to provide a final trained deep learning network for predicting response in members of a cohort of interest to a new treatment.

[0071] Step 701 involves training a first deep learning network using a first histopathology image dataset. In this particular example, controlled self-supervised learning is used to train the first feature extractor without requiring the use of labels from the training dataset (first medical image dataset).

[0072] Step 702 includes transferring weights of a first feature extractor learned from self-supervised training of the first deep learning network to a feature extractor of a next deep learning network in the pipeline training system.

[0073] Step 703 includes successively training one or more additional subsequent deep learning networks using successive subsequent medical image datasets and transferring weights from a feature extractor of one network in the pipeline to a feature extractor of a subsequent network in the pipeline. Again, in the illustrated example, only the weights of the feature extraction portion of each deep learning network are transferred from one network to the next in the pipeline. However, in alternative examples, weights of other portions of the network (e.g., classification portion) may be transferred as well. As previously discussed, successive datasets in the pipeline become increasingly relevant to the cohort for which the final network in the pipeline is trained.

[0074] Step 704 involves transferring the weights to a final network in the pipeline, and step 705 involves fine-tuning the final network using data from a first portion of a subject cohort in a clinical trial for a related new treatment.

[0075] Step 706 includes using the final trained deep learning network to make genotype predictions and / or treatment response predictions based on histopathology images corresponding to members of a second portion of the cohort of interest.

[0076] Results and Implications In some examples, the prediction made for each histopathology image (at the slide level) is the presence or absence of a genotypic alteration. In various examples, the genotypic alteration comprises an alteration in a gene corresponding to a fibroblast growth factor receptor (FGFR) gene ("FGFR alteration"). In various examples, the FGFR alteration comprises either an FGFR3 mutation, an FGFR3 fusion, or a combination of an FGFR3 mutation and an FGFR3 fusion. In various embodiments, the FGFR alteration is either a p.R248C mutation, a p.G370C mutation, a p.S249C mutation, or a p.Y373C mutation. In various embodiments, the FGFR alteration is either an FGFR3:TACC3V1 fusion, an FGFR3:TACC3V3 fusion, an FGFR3:BAIAP2L1 fusion, an FGFR2:BICC1 fusion, or an FGFR2:CASP7 fusion. In some examples, the FGFR alteration is in the context of a patient having bladder cancer.

[0077] In some examples, the alteration comprises an alteration (e.g., DNA-repair deficiency (DRD)) in one or more of the following genes: BRCA1, BRCA2, BRIP1, CDK12, CHEK2, FANCA, PALB2, RAD51B, RAD54L, RAD21, or SPOP. In some examples, the alteration in one or more of these genes is in the context of a patient with prostate cancer. In some examples, the alteration comprises an alteration in the MET gene (encoding the c-MET protein) (a "MET alteration"). In some examples, the MET alteration is in the context of a patient with lung cancer.

[0078] The alterations may include one or more of various types of alterations, including single nucleotide polymorphisms (SNPs), copy number variations (CNVs), or gene fusions. In various embodiments, the images are hematoxylin and eosin (H&E) stained histopathology images. In various embodiments, the subject has or is suspected of having cancer.

[0079] In some applications, the method includes (and / or the computer program product includes instructions for) determining whether to administer a therapeutic agent according to at least the determined genotype. In various embodiments, the therapeutic agent is an FGFR kinase inhibitor. In various embodiments, the FGFR kinase inhibitor is erdafitinib (BALVERSA™). In various embodiments, the therapeutic agent is a PARP inhibitor. In various embodiments, the PARP inhibitor is niraparib (ZEJULA™). In various embodiments, the final trained neural network model exhibits an area under the receiver operating curve (auROC) performance metric of at least 0.82 for bladder cancer images. In various embodiments, the final trained neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.22 with 100% recall for a bladder cancer model in a testing dataset having a baseline prevalence of 14% for FGFR. In various examples, the final trained neural network model exhibits an auROC performance metric of at least 0.71 for prostate cancer images. In various embodiments, the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.14 with 100% recall for prostate cancer images in a test dataset having a baseline prevalence of 11% for PARP.

[0080] In some applications, the tumor tissue slide from the patient subject comprises lung cancer. In some applications, the genotype is related to alterations in the mesenchymal epithelial transition (MET) gene (MET gene produces c-MET protein). In some examples, a final trained neural network model consistent with the present disclosure exhibits an auROC performance metric of at least 0.78+ / -0.03 for predicting MET genotype in the context of non-small cell lung cancer tissue images. In one example, the performance is for predicting whether a patient has 5 or more copies of the MET gene. In various embodiments, the therapeutic agent is a bispecific monoclonal antibody targeting epidermal growth factor receptor (EGFR) and MET. In various embodiments, the bispecific monoclonal antibody is amivantamab (Rybrevant™).

[0081] In various embodiments, the methods disclosed herein further include reporting one or more tiles of the image that are most strongly associated with the subject's genotype.

[0082] Results regarding other specific predictions relevant to efficient clinical trial candidate selection are discussed further below in the context of other figures.

[0083] 8 illustrates an example of a computer system 8000, one or more of which may be used to implement one or more of the apparatus, systems, and methods described herein. The computer system 8000 executes instruction code contained in a computer program product 860. The computer program product 860 includes executable code in an electronically readable medium that may instruct one or more computers, such as the computer system 8000, to perform processes to effect the example method steps that are performed.

[0084] The electronically readable medium may be any transitory or non-transitory medium that stores information electronically and may be accessed locally or remotely, for example, via a network connection. The medium may include multiple geographically distributed media, each configured to store different portions of the executable code at different locations and / or different times. The executable instruction code in the electronically readable medium directs the illustrated computer system 8000 to perform the various exemplary tasks described herein. The executable code for directing the performance of the tasks described herein is typically implemented in software. However, one skilled in the art will appreciate that a computer or other electronic device may utilize hardware-implemented code to perform many or all of the identified tasks. One skilled in the art will appreciate that many variations of the executable code implementing the exemplary methods may be found within the spirit and scope of the present disclosure.

[0085] Code or copies of code included in computer program product 860 may reside in one or more storage persistent media (not shown separately) communicatively coupled to system 8000 for loading and storing in persistent storage device 870 and / or memory 810 for execution by processor 820. Computer system 800 also includes an I / O subsystem 830 and peripheral devices 840. The I / O subsystem 830, peripheral devices 840, processor 820, memory 810, and persistent storage device 870 are coupled via bus 850. Memory 810, like persistent storage device 870 and any other persistent storage that may include computer program product 860, is a non-transitory medium (even if implemented as a typical volatile computer memory device). Furthermore, those skilled in the art will appreciate that in addition to storing computer program product 860 for performing the processes described herein, memory 810 and / or persistent storage device 870 may be configured to store various data elements referenced and illustrated herein.

[0086] Those skilled in the art will appreciate that computer system 8000 illustrates only one example of a system in which a computer program product according to the present disclosure may be implemented. As one example, execution of instructions included in a computer program product may be distributed across multiple computers, such as, for example, computers in a distributed computing network.

[0087] Instructions for implementing an artificial neural network or other deep learning network may be present in a computer program product 860. When the processor 820 is executing instructions of the computer program product 860, the instructions, or a portion thereof, are typically loaded into the working memory 810, from where the instructions are easily accessed by the processor 820.

[0088] Processor 820 may comprise multiple processors, each of which may comprise additional working memory (additional processors and memories not separately shown), including one or more graphics processing units (GPUs) with at least several thousand arithmetic logic units supporting massively parallel computations. GPUs are often utilized in deep learning applications because they can perform associated processing tasks more efficiently than typical general-purpose processors (CPUs). Processor 820 may additionally or alternatively comprise one or more dedicated processing units with systolic arrays and / or other hardware devices supporting efficient parallel processing. Such dedicated hardware may work with the CPU and / or GPU to perform various operations described herein. Such dedicated hardware may comprise application specific integrated circuits, etc. (which may refer to a portion of an application specific integrated circuit), field programmable gate arrays, etc., or combinations thereof. However, a processor such as processor 820 may be implemented as one or more general-purpose processors (preferably having multiple cores) without necessarily departing from the spirit and scope of the present disclosure.

[0089] Further embodiments and selected results The terms used in this further embodiment and selected results are defined as set forth below unless otherwise stated.

[0090] The terms "subject" or "patient" are used interchangeably and include cells, tissues, or organisms, whether human or non-human, male or female.

[0091] The term "mammal" encompasses both humans and non-humans, and includes, but is not limited to, humans, non-human primates, canines, felines, murines, bovines, equines, and porcines.

[0092] The term "sample" or "test sample" can include a single cell or multiple cells or fragments of a cell or bodily fluid aliquot, such as a blood sample, taken from a subject by means including venipuncture, voiding, ejaculation, massage, biopsy, needle aspirate, lavage sample, scraping, surgical incision, or other means, or other means known in the art. Examples of bodily fluid aliquots include amniotic fluid, aqueous humor, bile, lymph, breast milk, interstitial fluid, blood, plasma, earwax, cow's fluid, chyle, chyme, female ejaculate, menses, mucus, saliva, urine, vomit, tears, vaginal fluid, sweat, serum, semen, sebum, pus, pleural effusion, cerebrospinal fluid, synovial fluid, intracellular fluid, and vitreous humor. In various embodiments, the sample can be a biopsy of a tissue, such as a tumor. In a particular embodiment, the sample is a bladder tumor biopsy. In certain embodiments, the sample is a prostate tumor biopsy. In certain embodiments, the sample is a lung tumor biopsy.

[0093] The term "acquiring one or more images" encompasses acquiring one or more images captured from an object or acquiring one or more images captured from a sample obtained from an object. Acquiring one or more images can encompass performing a step of capturing one or more images from an object or from a sample obtained from an object. The phrase can also encompass receiving one or more images, for example, from a third party who performed a step of capturing one or more images from an object or from a sample obtained from an object. The one or more images can be acquired by one of skill in the art through a variety of known manners, including being stored in a storage device.

[0094] The phrases "subject genotype," "subject genotype," or "subject tumor genotype" are used herein generally to refer to the genotype of a subject's tumor. In various embodiments, a subject genotype refers to the state of a particular gene, such as a wild-type state or an altered state. For example, an altered state may indicate the presence of a mutation or fusion in a particular gene.

[0095] The term "neural network model" refers to a neural network machine learning model. In various embodiments, the neural network model includes multiple sub-models (e.g., a first sub-model and a second sub-model). For example, the neural network model may include a convolutional neural network and an attention network. In other embodiments, the neural network model is composed of only a convolutional neural network.

[0096] The terms "treating", "treatment" or "therapy" of cancer shall mean administering a treatment to slow, stop or reverse the progression of cancer. In some embodiments, treating cancer means reversing the progression of cancer, ideally to the point of eliminating the cancer itself. In various embodiments, "treating", "treatment" or "therapy" of lung cancer includes administering a therapeutic agent or pharmaceutical composition to a subject. Additionally, as used herein, "treating", "treatment" or "therapy" of cancer further includes administering a therapeutic agent or pharmaceutical composition for preventive purposes. Prevention of cancer refers to the administration of a composition or therapeutic agent to prevent the onset, development, manifestation, progression or recurrence of cancer, or some or all of the symptoms of cancer, or to reduce the likelihood of cancer onset.

[0097] It must be noted that, as used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0098] System Environment Overview FIG. 9 depicts an overview of a system environment for determining a genotype prediction for a subject, according to one embodiment. The system environment 100 provides a context for introducing a subject 110, an image generation system 120, and a genotype prediction system 130 for determining a genotype prediction 140 for the subject 110. Although FIG. 9 depicts one subject 110 for which a genotype prediction 140 is generated, in various embodiments, the system environment 100 includes two or more subjects such that the genotype prediction system 130 generates genotype predictions 140 for two or more subjects (e.g., a genotype prediction for each of the two or more subjects).

[0099] In various embodiments, the genotype prediction may be useful to prioritize subjects for subsequent testing. For example, subjects predicted to exhibit an altered genotype may undergo additional molecular testing to confirm the in silico results. In various embodiments, once confirmed through molecular testing, the subject may be provided with appropriate medical treatment or medical advice based on the altered genotype status. In contrast, subjects predicted to exhibit a wild type genotype do not need to undergo further molecular testing. This therefore conserves resources as a subset of subjects do not need to be additionally tested. In various embodiments, the genotype prediction may be useful to determine whether a subject 110 is likely to respond to an intervention. Thus, subjects 110 who are likely to respond to an intervention due to a predicted genotype may be enrolled in a clinical trial where the clinical trial patients are provided with the intervention.

[0100] In various embodiments, the subject has previously been diagnosed with cancer, examples of which include bladder cancer or prostate cancer. In various embodiments, determining a genotype prediction 140 for the subject may be useful for determining a genotype of interest for cancer. In various embodiments, determining a genotype prediction 140 for the subject may be useful for informing a physician of the likelihood that the patient has a genotype of interest for cancer. For example, a genotype of interest for bladder cancer is one associated with the fibroblast growth factor receptor (FGFR) gene. As another example, a genotype of interest for prostate cancer includes one associated with BRCA1, BRCA2, BRIP1, CDK12, CHEK2, FANCA, PALB2, RAD51B, RAD54L, RAD21, or SPOP. As another example, a genotype of interest for lung cancer includes one associated with the MET gene (c-MET).

[0101] In various embodiments, subjects predicted to have a particular genotype (e.g., genotypic change) may be administered a treatment that delays or prevents the onset, progression, or recurrence of cancer. In various embodiments, subjects predicted to have a particular genotype (e.g., genotypic change) are selected for enrollment in a clinical trial. In various embodiments, subjects predicted to have a particular genotype (e.g., genotypic change) and subsequently confirmed to have the particular genotype through molecular testing may be administered a treatment that delays or prevents the onset, progression, or recurrence of cancer. In various embodiments, subjects predicted to have a particular genotype (e.g., genotypic change) and subsequently confirmed to have the particular genotype through molecular testing are selected for enrollment in a clinical trial.

[0102] 9, a test sample is obtained from a subject 110, and an image generating system 120 captures an image from the test sample. In various embodiments, the test sample is a tissue biopsy. For example, the test sample can be a bladder tissue biopsy. For example, the test sample can be a prostate tissue biopsy.

[0103] In various embodiments, the test specimen is processed to prepare a sample that can be readily imaged by the imaging system 120. In certain embodiments, the test specimen is a tissue biopsy that undergoes tissue preparation and hematoxylin and eosin (H&E) staining so that the imaging system 120 can capture H&E images of the tissue. For example, a conventional H&E staining process may involve 1) preserving the tissue biopsy in formalin or paraffin embedding, 2) slicing the tissue into thin sections (e.g., 4-5 μm thick), 3) removing the embedding medium and rehydrating in xylene, ethanol, and deionized water), 4) staining for the target (e.g., antibody staining), 5) counter staining using hematoxylin, and 6) mounting the stained tissue sections onto slides for imaging.

[0104] The image generation system 120 captures images from processed samples derived from the subject 110. In various embodiments, the images and / or samples can be acquired by a third party, for example, a medical professional. Examples of medical professionals include doctors, emergency medical technicians, nurses, first responders, psychologists, phlebotomists, physical therapy personnel, nurse practitioners, surgeons, dentists, and any other obvious medical professionals as known to those of skill in the art. In various embodiments, the images and / or samples can be acquired at a hospital or medical clinic. In various embodiments, the images and / or samples can be acquired at a central laboratory or clinical research organization (CRO). In various embodiments, the images and / or samples can be captured using an imaging device.

[0105] In various embodiments, the imaging generating system 120 includes an imaging device, which may be one of a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, a positron emission tomography (PET) scanner, an X-ray scanner, an ultrasound imaging device, or an optical microscope, such as any of a bright field microscope, a dark field microscope, a phase contrast microscope, a differential interference contrast microscope, a fluorescence microscope, a confocal microscope, or a two-photon microscope. In a particular embodiment, the imaging device is an optical microscope that captures a control image. In various embodiments, the imaging generating system 120 acquires a control image, which may be any one of a bright field image, a phase contrast image, a dark field image, a Rheinberg illumination image, or a polarized light image.

[0106] In general, the genotype prediction system 130 analyzes one or more images captured from the subject 110 (e.g., images taken by the imaging generation system 120) and generates a genotype prediction 140 for the subject 110. In various embodiments, the genotype prediction 140 determined by the genotype prediction system 130 identifies whether a particular target exhibits a wild-type state or an altered state. For example, the genotype prediction 140 can be an indication that the target (e.g., a gene product such as a protein) exhibits a wild-type state. As another example, the genotype prediction 140 can be an indication that the target (e.g., a gene product such as a protein) exhibits an altered state (e.g., the target is altered). In various embodiments, the genotype prediction 140 can be an indication of a particular type of alteration. Examples of particular alterations can include the presence of a mutation, the presence of a fusion, or the presence of both a mutation and a fusion. As a specific example in the context of bladder cancer, the genotype prediction 140 can be an indication that the subject exhibits a wild-type FGFR state or an altered FGFR state. As a particular example in the context of prostate cancer, the genotype prediction 140 may be an indication that the subject exhibits wildtype BRCA status or altered BRCA status.

[0107] In various embodiments, the genotype prediction 140 may be an indication of one of a wild type state or an altered state, where the altered state is defined by the presence of an alteration in at least one of the multiple target genes. In various embodiments, the genotype prediction 140 may indicate the altered state based on the presence of an alteration in at least one of the at least two genes. In various embodiments, the genotype prediction 140 may indicate the altered state based on the presence of an alteration in at least one of at least three genes, at least four genes, at least five genes, at least six genes, at least seven genes, at least eight genes, at least nine genes, at least ten genes, at least eleven genes, at least twelve genes, at least thirteen genes, at least fourteen genes, or at least fifteen genes. In various embodiments, the genotype prediction 140 may indicate the altered state based on the presence of an alteration in at least one of the eleven genes. To provide an example, in the context of prostate cancer, the genotype prediction 140 may be an indication of an altered state based on the presence of an alteration in any one of the following genes: BRCA1; BRCA2; BRIP1; CDK12; CHEK2; FANCA; PALB2; RAD51B; RAD54L; RAD21; SPOP.

[0108] In various embodiments, the genotypic prediction 140 can include a recommended intervention for the subject 110 based on the predicted genotypic status. Returning to the context of bladder cancer, for example, if the genotypic prediction system 130 determines that the subject 110 exhibits an altered FGFR status, the genotypic prediction 140 can include a recommended intervention likely to slow the progression, prevent the progression, or reduce the size of the bladder cancer exhibiting the altered FGFR status.

[0109] The genotype prediction system 130 may include one or more computers embodied as a computer system 1400, as discussed below with respect to FIG. 13. Thus, in various embodiments, the steps described with reference to the genotype prediction system 130 are performed in silico. In various embodiments, the imaging generation system 120 and the genotype prediction system 130 are used by different parties. For example, a first party operates the imaging generation system 120 to capture one or more images derived from the subject 110 and then provides the captured one or more images to a second party that implements the genotype prediction system 130 to determine the genotype prediction 140. For example, the imaging generation system 120 may capture one or more images, store the one or more images, and / or automatically stream the captured one or more images to the genotype prediction system 130, which may automatically analyze the received one or more images to generate the genotype prediction 140. In some embodiments, the imaging generation system 120 and the genotype prediction system 130 are used by the same party.

[0110] Reference is now made to FIG. 10, which illustrates a block diagram of a genotype prediction system, according to one embodiment. Here, the genotype prediction system 130 includes an image pre-processing module 145, a neural network deployment module 160, a neural network training module 150, and a training data store 170. In various embodiments, the genotype prediction system 130 can be configured differently with additional or fewer modules. For example, the genotype prediction system 130 need not include the neural network training module 150 or the training data store 170 (as shown by the dotted lines in FIG. 10), and instead, the neural network training module 150 and the training data store 170 can be used by different systems and / or participants.

[0111] The components of the genotype prediction system 130 are described below with reference to two phases: 1) a training phase and 2) a deployment phase. More specifically, the training phase refers to constructing and training a machine learning model (e.g., a neural network) by the neural network training module 150 based on training data, such as training images captured from training individuals (e.g., individuals whose genotypes are previously known). Thus, the machine learning model (e.g., a neural network) is trained using the training data such that implementation of the machine learning model by the deployment module 160 in the deployment phase enables prediction of the genotype of a subject (e.g., subject 110 of FIG. 9 ).

[0112] In some embodiments, the components of the genotypic prediction system 130 are applied during one of the training phase and the deployment phase. For example, the neural network training module 150 and the training data store 170 are applied to train the neural network model during the training phase. Additionally, the neural network deployment module 160 is applied during the deployment phase. In various embodiments, the components of the genotypic prediction system 130 can be executed by different parties depending on whether the components are applied during the training phase or the deployment phase. In such a scenario, the training and deployment of the neural network model are executed by different parties. For example, the neural network training module 150 and the training data store 170 applied during the training phase can be used by a first party (e.g., to train the neural network prediction model), and the neural network deployment module 160 applied during the deployment phase can be executed by a second party (e.g., to deploy the neural network prediction model).

[0113] Referring to the image pre-processing module 145, it takes the images captured by the image generation system 120 and pre-processes the images. In general, pre-processing the images allows for a more uniform and accurate analysis of the images by the neural network.

[0114] In various embodiments, the image preprocessing module 145 preprocesses the image by removing uninformative tiles or uninformative regions of one or more tiles of the image. For example, the image preprocessing module 145 may remove tiles or regions of tiles where no tissue is present. As another example, the image preprocessing module 145 may remove tiles where less than a threshold area of ​​the tile (e.g., less than 10%, less than 20%, less than 30%, less than 40%, or less than 50%) contains tissue. As another example, the image preprocessing module 145 may perform image recognition to identify the presence of pen marks on tiles of the image. Upon identifying the presence of pen marks, the image preprocessing module 145 may remove regions of the tile that contain pen marks. The image preprocessing module 145 may remove uninformative regions of one or more tiles of the training image during the training phase and / or remove uninformative regions of one or more tiles of the image during the deployment phase.

[0115] In various embodiments, the image preprocessing module 145 preprocesses the image by performing image stain enhancement of the image. Generally, performing image stain enhancement of the image includes generating additional images. As an example, the image preprocessing module 145 performs image stain enhancement of the image by modifying stain intensity or stain contrast of tiles of the image. As another example, the image preprocessing module 145 performs image stain enhancement of the image by performing color jittering or color normalization of tiles of the image. As another example, the image preprocessing module 145 performs image stain enhancement of the image by performing horizontal, vertical, or rotational flip of tiles of the image.

[0116] In various embodiments, image stain enhancement is performed only on the training images (and not on the images analyzed during the deployment phase), thereby enabling the generation of additional training images that can be used to train the neural network. In other words, the image pre-processing module 145 performs image stain enhancement during the training phase to diversify and supplement the entire training image set used to train the neural network. Thus, by training the neural network on diversified training images, the neural network can handle tiles of different stain intensities, different stain colorations, and different rotations and orientations.

[0117] Image pre-processing module 145 provides the processed images to either neural network training module 150 (during the training phase) or neural network deployment module 160 (during the deployment phase). In some embodiments, image pre-processing module 145 stores the processed images in training data store 170 for later retrieval to train the neural network.

[0118] Neural network training module 150 trains a risk neural network model using training data derived from training individuals. Neural network deployment module 160 implements the neural network model to analyze individual tiles of the image and generate genotype predictions for the subject 110. Training a machine learning model (e.g., a neural network model) and deploying a machine learning model (e.g., a neural network model) are described in further detail below.

[0119] Method for predicting the genotype of a subject The embodiments described herein include a method for predicting a subject's genotype by applying a trained neural network model. Such a method can be performed by a genotype prediction system 130 as described in Figure 10. Further reference is made to Figure 11, which depicts an exemplary flow diagram 200 for determining a subject's cancer genotype prediction 1240, according to one embodiment.

[0120] Flow diagram 200 begins with an image 1210, such as an image captured by image generation system 120. In various embodiments, image 1210 is previously subjected to image pre-processing, as described above in connection with image pre-processing module 145 of Figure 10. In such an embodiment, image 1210 shown in Figure 11 represents a pre-processed image.

[0121] A number of tiles 1220 are generated from the image 1210. Generally, each tile 1220 represents a subset of the image 1210. For example, if the image 1210 has dimensions of M pixels by N pixels, then the tiles 1220 may have dimensions of M / X pixels by N / Y pixels, where the values ​​of M / X and N / Y are constant values. In various embodiments, each tile 1220 has the same dimensions as all other tiles 1220. In various embodiments, each tile 1220 has dimensions of 224 by 224 pixels.

[0122] The individual tiles 1220 are provided as input to a neural network model 1230. Generally, the neural network model 1230 analyzes the individual tiles 1220 and determines a prediction, such as a genotype prediction 1240. In various embodiments, the neural network model 1230 determines an informative prediction to determine the genotype prediction 1240. For example, the neural network model 1230 may determine a score, which may then be converted into the genotype prediction 1240.

[0123] In various embodiments, the genotype prediction is determined by comparing the score with a threshold value. If the score is above the threshold value, the genotype prediction is the first classification. If the score is below the threshold value, the genotype prediction is the second classification.

[0124] As shown in FIG. 11, the neural network model 1230 includes a first sub-model 1250 and a second sub-model 1260. In various embodiments, the neural network model 1230 includes only the first sub-model 1250 and does not include the second sub-model 1260. Thus, the genotype prediction 1240 is determined from the output of the first sub-model 1250. In various embodiments, the first sub-model 1250 analyzes the individual tiles 1220 and determines tile-level predictions or tile-level features that are informative for tile-level predictions, but does not provide a tile-level prediction. For example, in one embodiment as shown in FIG. 11 with both the first sub-model 1250 and the second sub-model 1260, the first sub-model 1250 outputs tile-level features that are then input to the second sub-model. As another example, in an embodiment in which the neural network model 1230 includes only the first sub-model 1250, the first sub-model outputs a tile-level prediction that is used to determine the genotype prediction 1240.

[0125] For example, if there are a total of Z tiles 1220 input to the neural network model 1230, the first sub-model 1250 outputs Z different tile level predictions or Z different tile level features, each tile level prediction or tile level feature associated with a tile 1220. In various embodiments, each tile level prediction may be a categorization of the tile. For example, a first categorization may be a wild type state and a second categorization may be an altered genotype state. In various embodiments, each tile level prediction may be a probability (e.g., between 0 and 1) reflecting the likely probability of the tile's genotype state (e.g., wild type or altered genotype state). The tile level feature may be a vector (e.g., an array of numbers) corresponding to the tile.

[0126] In various embodiments, the first sub-model 1250 is a convolutional neural network consisting of multiple layers, multiple nodes per layer, and parameters associated with the nodes of the layers. Thus, the convolutional neural network analyzes individual tiles by propagating the tiles through multiple layers to generate tile-level predictions (or tile-level features).

[0127] In various embodiments, the neural network model 1230 includes only the first sub-model 1250 and does not include the second sub-model 1260. In such embodiments, the tile-level prediction (or tile-level features) output by the first sub-model 1250 may be analyzed to determine the genotype prediction 1240. For example, the tile corresponding to the highest tile probability may be selected, and the highest tile probability is compared to a threshold. If the highest tile probability is greater than the threshold, the slide is labeled as the first classification. If the highest tile probability is less than the threshold, the slide is labeled as the second classification. In various embodiments, the threshold is 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.80, 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, or 0.99.

[0128] In various embodiments, such as the embodiment shown in FIG. 11, the neural network model 1230 includes both a first sub-model 1250 and a second sub-model 1260. Thus, individual tile-level predictions are provided as input to the second sub-model 1260. In various embodiments, the first sub-model 1250 generates tile-level features (e.g., informative features for aggregating tiles), which are then provided as input to the second sub-model 1260. Here, the second sub-model 1260 generates image-level predictions (also referred to herein as "slide-level predictions") by aggregating across individual tile-level predictions or tile-level features.

[0129] In various embodiments, the second sub-model 1260 is a neural network, hereafter referred to as an attention network. Thus, the attention network may consist of multiple layers, multiple nodes per layer, and parameters associated with the nodes of the layers. In general, the attention network is trained to aggregate tile-level predictions or tile-level features to generate image-level predictions. Thus, the attention network can learn to more heavily weight certain tile-level predictions or tile-level features associated with tiles that are more informative for determining genotypes, and conversely, to less heavily weight certain tile-level predictions or tile-level features associated with tiles that are less informative for determining genotypes.

[0130] In various embodiments, the second sub-model 1260 outputs a genotype prediction 1240. In various embodiments, the second sub-model 1260 outputs an informative score of the genotype prediction 1240. For example, the second sub-model 1260 may output a constant value indicative of a genotype classification (e.g., wildtype or altered). As a particular example, a value of "0" may indicate a wildtype genotype classification, and a value of "1" may indicate an altered genotype classification.

[0131] In various embodiments, the subject's predicted genotype 140 may be displayed to a user, e.g., a clinician-user. Thus, the clinician-user may inform the subject of the predicted genotype. In various embodiments, additional / other information may be displayed to the user, e.g., a clinician-user. For example, the one or more tiles that were most informative in generating the predicted genotype 140 may be displayed to the user. This allows the user to perform a manual check of one or more tiles to ensure that confounding image artifacts (e.g., pen marks or staining discolorations) do not lead to the predicted genotype 140.

[0132] Exemplary Machine Learning Model for Predicting Genotype The embodiments disclosed herein involve training and deploying a machine learning model to predict a subject's genotype. In general, the machine learning model is constructed to analyze individual tiles of an image and output an informative image-level prediction of the subject's genotype. In certain embodiments, the machine learning model is a neural network model (e.g., a feed-forward network, a convolutional neural network (CNN), a deep neural network (DNN), an autoencoder neural network, a generative adversarial network, or a recurrent network (e.g., a long short-term memory network (LSTM), a bidirectional recurrent network, or a deep bidirectional recurrent network)). Although the description herein refers to a neural network model, in other embodiments, the machine learning model is any one of a regression model (e.g., linear regression, logistic regression, or polynomial regression), a decision tree, a random forest, a gradient boosted machine learning model, a support vector machine, a naive Bayes model, a k-means cluster, or any combination thereof.

[0133] In various embodiments, the machine learning model includes two or more sub-models. For example, the machine learning model may include two sub-models: a first sub-model that analyzes individual tiles and outputs a tile-level prediction, and a second sub-model that aggregates the tile-level predictions and outputs an image-level prediction. Thus, the output of the first sub-model serves as an input to the second sub-model. In certain embodiments, both the first sub-model and the second sub-model are neural networks (e.g., feed-forward networks, convolutional neural networks (CNNs), deep neural networks (DNNs), autoencoder neural networks, generative adversarial networks, or recurrent networks (e.g., long short-term memory networks (LSTMs), bidirectional recurrent networks, deep bidirectional recurrent networks), or any combination thereof. In certain embodiments, the first sub-model is a convolutional neural network.

[0134] The machine learning model (and its sub-models) can be trained using a machine learning implementation method such as any one of a linear regression algorithm, a logistic regression algorithm, a decision tree algorithm, a support vector machine classification, a naive Bayes classification, a K-nearest neighbor classification, a random forest algorithm, a deep learning algorithm, a gradient boosting algorithm, and a dimensionality reduction technique such as manifold learning, principal component analysis, factor analysis, autoencoder regularization, and independent component analysis, or a combination thereof. In various embodiments, the machine learning model is trained using a supervised learning algorithm, an unsupervised learning algorithm, a semi-supervised learning algorithm (e.g., partially supervised), weakly supervised, transfer learning, multi-task learning, or any combination thereof. In certain embodiments, the machine learning model is trained using a weakly supervised technique. In certain embodiments, the machine learning model is trained using a deep learning algorithm.

[0135] In various embodiments, a machine learning model (or a submodel thereof) has one or more parameters, such as hyperparameters or model parameters. The hyperparameters are generally established prior to training. In various embodiments, hyperparameter optimization (e.g., grid search) is performed via cross-validation. Examples of hyperparameters include the learning rate, depth, or leaves of a decision tree, the number of hidden layers in a deep neural network, the number of clusters in a k-means cluster, the penalty in a regression model, and a regularization parameter associated with a cost function. Model parameters are generally adjusted during training. Examples of model parameters include weights associated with nodes in a layer of a neural network, support vectors in a support vector machine, node values ​​in a decision tree, and coefficients in a regression model. The model parameters of a machine learning model are trained (e.g., adjusted) using training data to improve the predictive ability of the machine learning model.

[0136] Referring again to FIG. 10, the neural network training module 150 trains the machine learning model using training data. In various embodiments, the training data can be obtained and / or derived from a publicly available database. For example, the training data can be obtained and / or derived from the National Cancer Institute GDC Data Portal and / or the cBioPortal for Cancer Genomics. In some embodiments, the training data can be obtained and collected independently of a publicly available database, for example, by capturing images from multiple training individuals. Such training data can be a custom dataset.

[0137] In various embodiments, training data can be stored in and / or retrieved from the training data store 170. Generally, the training data includes training images of H&E stained tissue slides derived from training individuals (e.g., individuals with known wild type or altered genotypes). As described above, the training images can undergo pre-processing. For example, the training images can undergo a quality control process to remove image artifacts such as pen marks. Additionally, the training images can be generated via stain-based enhancement, which diversifies and increases the number of training images that can be used for training. For example, a single image can be pre-processed using a stain-based enhancement process to generate additional images of different stain colorations. As a specific example, a single training image can undergo a stain-based enhancement process to generate 10 training images of different stain colorations. This prevents the different stain colorations from affecting the predictions generated by the machine learning model.

[0138] In various embodiments, the training data may be obtained from a split of the dataset. For example, the dataset may undergo a 50:50 training:test dataset split. In some embodiments, the dataset may undergo a 60:40 training:test dataset split. In some embodiments, the dataset may undergo an 80:20 training:test dataset split. In some embodiments, the dataset may undergo a 70:15:15 training:test:validation dataset split.

[0139] In various embodiments, the training data used to train the machine learning model includes a reference ground truth that indicates the genotype state of the training individuals. In various embodiments, the reference ground truth in the training data is a binary value, such as "1" or "0." For example, a training individual with a known wild type genotype may be identified with a value of "0" in the training data, while a training individual with a known altered genotype may be identified with a value of "1" in the training data.

[0140] In various embodiments, the neural network training module 150 uses the training data to train the machine learning model to minimize a loss function so that the machine learning model can better predict the genotype status based on the input (e.g., tiles of training images). Here, the neural network training module 150 may back-propagate the loss value and adjust the parameters of the machine learning model to minimize the loss value. In various embodiments, the loss function is constructed for either least absolute shrinkage and selection operator (LASSO) regression, ridge regression, or ElasticNet regression. In certain embodiments, the loss function is a cross-entropy loss between the predicted label and the true label (e.g., the label can be a binary label of true or false for FGFR in bladder cancer). As described above, in some embodiments, the machine learning model is a neural network model having at least a first sub-model (e.g., a convolutional neural network) and a second sub-model (e.g., an attention network). In various embodiments, the first sub-model and the second sub-model are jointly trained. Thus, the neural network training module 150 trains the neural network model by backpropagating the loss values ​​and adjusting both the parameters of the first sub-model and the parameters of the second sub-model together. In other embodiments, the first sub-model and the second sub-model are trained separately. Thus, the neural network training module 150 trains the neural network model to minimize the loss function by adjusting the parameters of the first sub-model and the parameters of the second sub-model separately.

[0141] In various embodiments, the machine learning models disclosed herein achieve a performance metric. Exemplary performance metrics include the area under the receiver operating curve (auROC), positive predictive value, and / or negative predictive value of the receiver operating curve. In various embodiments, the machine learning models disclosed herein exhibit an auROC value of at least 0.5. In various embodiments, the machine learning models disclosed herein exhibit an auROC value of at least 0.6. In various embodiments, the machine learning models disclosed herein exhibit an auROC value of at least 0.7. In various embodiments, the machine learning models disclosed herein exhibit an auROC value of at least 0.8. In various embodiments, the machine learning models disclosed herein exhibit an auROC value of at least 0.9. In various embodiments, the machine learning models disclosed herein exhibit an auROC value of at least 0.95. In various embodiments, the machine learning models disclosed herein exhibit an auROC value of at least 0.99. In various embodiments, the machine learning models disclosed herein have a sigma-like probability of at least 0.51, at least 0.52, at least 0.53, at least 0.54, at least 0.55, at least 0.56, at least 0.57, at least 0.58, at least 0.59, at least 0.60, at least 0.61, at least 0.62, at least 0.63, at least 0.64, at least 0.65, at least 0.66, at least 0.67, at least 0.68, at least 0.69, at least 0.70, at least 0.71, at least 0.72, at least 0.73, at least 0.74, at least 0.75, at least 0.76, at least 0.77, at least 0.78, at least 0.79, at least 0.80, at least 0.81, at least 0.82, at least 0.83, at least 0.84, at least 0.85, at least 0.86, at least 0.87, at least 0.88, at least 0.89, at least 0.90, at least 0.91, at least 0.92, at least 0.93, at least 0.94, at least 0.95, at least 0.96, at least 0.97, at least 0.98, at least 0.99, at least 10 ... exhibiting an auROC value of 0.74, at least 0.75, at least 0.76, at least 0.77, at least 0.78, at least 0.79, at least 0.80, at least 0.81, at least 0.82, at least 0.83, at least 0.84, at least 0.85, at least 0.86, at least 0.87, at least 0.88, at least 0.89, at least 0.90, at least 0.91, at least 0.92, at least 0.93, at least 0.94, at least 0.95, at least 0.96, at least 0.97, at least 0.98, at least 0.99.

[0142] Exemplary Methods for Predicting the Genotype of a Subject 12 is an exemplary flow process for determining a subject's genotype prediction according to the first embodiment. As shown in FIG. 12, a flow process 1305 includes steps 1310, 1320, 1330, and 1340.

[0143] Step 1310 involves obtaining an image of a tissue slide derived from the subject. In various embodiments, the tissue slide is an H&E stained biopsy slide. In various embodiments, the tissue is a bladder or prostate biopsy of the subject.

[0144] Step 1320 involves pre-processing the image using a quality control process. In various embodiments, pre-processing the image involves improving the quality of the image, for example, by removing uninformative regions of the image and / or enhancing tissue staining. Thus, the pre-processed image allows the image to be more easily manageable and allows the neural network to properly analyze the image with limited influence from confounding factors (e.g., from image artifacts).

[0145] Step 1330 involves applying a neural network model to analyze tiles of the preprocessed image. In various embodiments, the neural network model generates a slide level prediction. In various embodiments, the slide level prediction includes a score. In various embodiments, the neural network model generates a tile level prediction based on an analysis of individual tiles and then aggregates the tile level predictions to generate a slide level prediction. In various embodiments, the neural network model includes at least a first sub-model (e.g., a convolutional neural network) and a second sub-model (e.g., an attention network for tile aggregation to generate a slide level prediction).

[0146] Step 1340 involves determining the genotype of the subject according to the slide level prediction. For example, the slide level prediction may include a score, and thus the genotype of the subject is classified according to the score. In various embodiments, the genotype of the subject is classified by comparing the score to a threshold. If the score is above the threshold, the genotype of the subject is classified into a first classification. If the score is below the threshold, the genotype of the subject is classified into a second classification.

[0147] In various embodiments, the subject undergoes additional molecular testing based on slide-level prediction. For example, if slide-level prediction indicates that the subject is likely to show an altered genotype, the subject undergoes additional molecular testing to confirm the in silico result. In general, this allows prioritization of subjects for molecular testing, since subjects predicted to show an altered genotype status can be prioritized over subjects predicted to show a wild-type genotype status.

[0148] In various embodiments, the subject can be provided with appropriate medical treatment or medical advice based on the altered genotype status once confirmed through molecular testing. In various embodiments, a guideline intervention can be selected for the subject based on the subject's genotype. For example, if the predicted genotype indicates the presence of a mutated target (such as, for example, a mutated fibroblast growth factor receptor (FGFR)), a therapeutic intervention (e.g., BALVERSA) can be selected for the subject. In some embodiments, if the predicted genotype indicates the presence of a mutated target, the subject can be enrolled in a clinical trial to receive the therapeutic intervention.

[0149] Cancer and Mutations The methods described herein involve implementing a neural network model to predict a subject's cancer-related genotype. In various embodiments, the cancer in the subject can include one or more of lymphoma, B-cell lymphoma, T-cell lymphoma, mycotic mycosis, Hodgkin's disease, myeloid leukemia, bladder cancer, brain tumor, cancer of the nervous system, head and neck cancer, head and neck squamous cell carcinoma, kidney cancer, lung cancer, neuroblastoma / glioblastoma, ovarian cancer, pancreatic cancer, prostate cancer, skin cancer, liver cancer, melanoma, squamous cell carcinoma of the mouth, throat, larynx, lung, colon cancer, cervical cancer, cervical carcinoma, breast cancer, and epithelial cancer, renal cancer, genitourinary cancer, lung cancer, esophageal cancer, gastric cancer, thyroid cancer, head and neck cancer, colon cancer, hematopoietic cancer, testicular cancer, colon and / or rectal cancer, uterine cancer, or prostate cancer. In some embodiments, the cancer in the subject may be a metastatic cancer, including any one of bladder cancer, breast cancer, colon cancer, kidney cancer, lung cancer, melanoma, ovarian cancer, pancreatic cancer, prostate cancer, rectal cancer, gastric cancer, thyroid cancer, or uterine cancer. In certain embodiments, the cancer is bladder cancer. In certain embodiments, the cancer is prostate cancer. In certain embodiments, the cancer is lung cancer.

[0150] In various embodiments, the methods disclosed herein include predicting genotypes associated with bladder cancer, such as fibroblast growth factor receptor (FGFR). In such embodiments, a guided treatment can be provided to a subject based on the subject's FGFR status. The FGFR status can refer to wild-type FGFR or FGFR alterations. In various embodiments, the FGFR alterations refer to either FGFR3 mutations, FGFR3 fusions, or a combination of FGFR3 mutations and FGFR3 fusions. In various embodiments, the FGFR alterations refer to either FGFR2 mutations, FGFR2 fusions, or a combination of FGFR2 mutations and FGFR2 fusions.

[0151] Examples of point mutations in FGFR3 are listed in Table 1 below.

[0152] [Table 1]

[0153] Examples of FGFR3 fusions are listed in Table 2 below.

[0154] [Table 2]

[0155] In various embodiments, the methods disclosed herein involve predicting a genotype associated with prostate cancer, such as the BRCA1 and / or BRCA2 genes. For example, a genotype associated with prostate cancer can include a single nucleotide polymorphism (SNP), copy number variation (CNV), or gene fusion involving the BRCA1 or BRCA2 genes.

[0156] The embodiments described herein involve implementing a machine learning model (e.g., a neural network model) to predict a subject's genotype. In various embodiments, an intervention is selected for the subject based on the predicted genotype. In various embodiments, the intervention can be any one of application of a diagnostic, application of a preventative treatment, or a subsequent action. An exemplary subsequent action can include subsequent testing of the subject or a test sample from the subject to confirm the in silico genotype prediction.

[0157] In certain embodiments, if the subject's predicted genotype is an altered genotype, the test sample from the subject may be provided for further molecular testing screening to confirm that the subject exhibits an altered genotype. In such embodiments, if the additional molecular testing screening confirms that the subject exhibits an altered genotype, the subject may be considered eligible for enrollment in a clinical trial. In certain embodiments, if the subject's predicted genotype is a wild-type genotype, the subject may be excluded from subsequent analysis for potential enrollment in a clinical trial. In such embodiments, the in silico process acts as a predictive screen to eliminate some subjects who are likely to exhibit a wild-type genotype and therefore should not be enrolled in a clinical trial. This avoids having to perform timely and costly molecular screening for each individual patient.

[0158] In various embodiments, if a subject's predicted genotype indicates the presence of a mutated FGFR, an intervention comprising BALVERSA (erdafitinib) may be selected for the subject. As another example, if a subject's predicted genotype indicates the presence of an altered BRCA gene (e.g., an altered BRCA1 or BRCA2), an intervention comprising a poly(ADP-ribose) polymerase (PARP) inhibitor, in particular ZEJULA (niraparib), may be selected for the subject.

[0159] In various embodiments, a therapeutic agent may be selected and / or administered to a subject based on the subject's predicted genotype. The selected therapeutic agent is likely to delay or prevent the onset of cancer, such as prostate or bladder cancer. Exemplary therapeutic agents include chemotherapy, energy therapy (e.g., external beam, microwave, radiofrequency ablation, brachytherapy, electroporation, cryoablation, photothermal ablation, laser therapy, photodynamic therapy, electrocautery, chemical embodied therapy, high intensity focused ultrasound, low intensity focused ultrasound), antigen-specific monoclonal antibodies, anti-inflammatory drugs, oncolytic virus therapy, or immunotherapy. In various embodiments, the selected therapeutic agent is an energy therapy, and the amount of energy applied (e.g., dose and duration) can be adjusted to achieve a desired therapeutic effect. In various embodiments, the therapeutic agent includes a small molecule or biologic, such as a cytokine, an antibody, a soluble cytokine receptor, an antisense oligonucleotide, an siRNA, and the like. Such biologic agents include mucins and derivatives of biologic agents, which may include, for example, fusion proteins, PEGylated derivatives, cholesterol-conjugated derivatives, and the like, as known in the art. Also included are antagonists of cytokines and cytokine receptors, such as traps and monoclonal antagonists. Also included are biosimilar drugs of the active agents described herein or drugs that are bioequivalent to the active agents.

[0160] Treatments for bladder cancer include BALVERSA (erdafitinib), atezolizumab, avelumab, BAVENCIO (avelumab), cisplatin, doxorubicin hydrochloride, enfortumab vedotin-ejfv, erdafitinib, JELMYTO (mitomycin), KEYTRUDA (pembrolizumab), nivolumab, OPDIVO (nivolumab), PADCEV (enfortumab vedotin-ejfv ... vedotin-ejfv), pembrolizumab, sacituzumab govitecan-hziy, TECENTRIQ (atezolizumab), TEPADINA (thiotepa), thiotepa, TRODELVY (sacituzumab govitecan-hziy, valrubicin, and VALSTAR (valrubicin). In certain embodiments, the therapeutic agent is BALVERSA (erdafinitib).

[0161] Treatments for prostate cancer include poly(ADP-ribose) polymerase (PARP) inhibitors, abiraterone acetate, apalutamide, bicalutamide, cabazitaxel, CASODEX (bicalutamide), darolamide, degarelix, docetaxel, ELIGARD (leuprolide acetate), enzalutamide, ERLEADA (apalutamide), FIRMAGON (degarelix), flutamide, goserelin acetate, JEVTANA (cabazitaxel), leuprolide acetate, and LUPRON. Examples include DEPOT (leuprolide acetate), LYNPARZA (olaparib), mitoxantrone hydrochloride, NILANDRON (nilutamide), nilutamide, NUBEQA (darolutamide), olaparib, ORGOVYX (relugolix), PROVENGE (sipuleucel-T), radium-223 dichloride, relugolix, RUBRACA (rucaparib camsylate), rucaparib camsylate, sipuleucel-T, TAXOTERE (docetaxel), XOFIGO (radium-223 dichloride), XTANDI (enzalutamide), YONSA (abiraterone acetate), ZOLADEX (goserelin acetate), and ZYTIGA (abiraterone acetate). In certain embodiments, the prostate cancer therapeutic agent is a PARP inhibitor, in particular ZEJULA (niraparib). In certain embodiments, the lung cancer treatment is Rybervant (amivantamab).

[0162] In various embodiments, one or more of the described therapeutic agents can be combined as a combination therapy to treat a subject.

[0163] In various embodiments, the pharmaceutical composition may be selected and / or administered based at least on a predicted genotype for the subject, and the selected therapeutic agent is likely to show efficacy against cancer. In various embodiments, the pharmaceutical composition may be selected and / or administered to the subject based on a molecular test performed based on a prediction that the subject's tumor will show a mutant genotype. For example, if the predicted genotype shows a mutant genotype for the subject's tumor and a subsequent molecular test confirms the mutant genotype of the subject's tumor, a pharmaceutical composition may be selected and / or administered to the subject. The pharmaceutical composition administered to the individual includes an active agent, such as a therapeutic agent described above. The active ingredient is present in a therapeutically effective amount, i.e., in an amount sufficient when administered to treat the disease or medical condition mediated thereby. The composition may also include various other agents for enhancing delivery and efficacy, to enhance delivery and stability of the active ingredient. Thus, for example, the composition may also include a pharmaceutically acceptable non-toxic carrier or diluent, depending on the desired formulation, which is defined as a vehicle commonly used to formulate pharmaceutical compositions for animal or human administration. The diluent is selected so as not to affect the biological activity of the combination. Examples of such diluents are distilled water, buffered water, physiological saline, PBS, Ringer's solution, dextrose solution, and Hank's solution. In addition, the pharmaceutical composition or formulation can include other carriers, adjuvants, or non-toxic non-therapeutic, non-immunogenic stabilizers, excipients, and the like. The composition can also include additional substances to approximate physiological conditions, such as pH adjusting and buffering agents, toxicity adjusting agents, wetting agents, and detergents. The composition can also include any of a variety of stabilizers, such as antioxidants.

[0164] The pharmaceutical compositions or therapeutic agents described herein can be administered in a variety of different ways, examples include administering a composition containing a pharma- ceutical acceptable carrier via oral, intranasal, intramodular, intralesional, rectal, topical, intraperitoneal, intravenous, intramuscular, subcutaneous, transdermal, intrathecal, intrabronchial, transthoracic, or intracranial methods.

[0165] Computer implementation Methods of the invention, including methods of implementing neural network models for predicting a subject's genotype, are in some embodiments performed on one or more computers.

[0166] For example, the construction and deployment of the neural network model may be implemented in hardware or software, or a combination of both. In one embodiment of the present invention, a machine-readable storage medium is provided, the medium including a data storage material encoded with machine-readable data, which when used with a machine programmed with instructions for using the data, may perform the training or deployment of the neural network model and / or display any of the datasets or results described herein. The present invention may be embodied in a computer program executed on a programmable computer including a processor, a data storage system (including volatile and non-volatile memory and / or storage elements), a graphics adapter, a pointing device, a network adapter, at least one input device, and at least one output device. A display is coupled to the graphics adapter. The program code is applied to the input data to perform the functions described above and generate output information. The output information is applied to one or more output devices in a known manner. The computer may be, for example, a personal computer, a microcomputer, or a workstation of conventional design.

[0167] Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In either case, the language may be a compiled or interpreted language. Each such computer program is preferably stored in a general-purpose or special-purpose programmable computer-readable storage medium or device (e.g., a ROM or magnetic diskette) to configure and operate the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system may also be considered to be implemented as a computer-readable storage medium configured with a computer program, the storage medium so configured causing the computer to operate in a specific predefined manner to perform the functions described herein.

[0168] The signature patterns and their databases can be provided on a variety of media to facilitate their use. "Media" refers to a product containing the signature pattern information of the present invention. The database of the present invention can be recorded on a computer-readable medium, e.g., any medium that can be read and directly accessed by a computer. Such media include, but are not limited to, magnetic storage media such as floppy disks, hard disk storage media, and magnetic tapes, optical media such as CD-ROMs, electrical storage media such as RAM and ROM, and hybrids of these categories such as magnetic / optical storage media. Those skilled in the art can readily recognize how any of the currently known computer-readable media can be used to create a product containing a recording of the present database information. "Recorded" refers to a process for storing information on a computer-readable medium, using any such method as known in the art. Any convenient data storage structure can be selected based on the means used to access the stored information. A variety of data processor programs and formats can be used for storage, e.g., word processing text files, database formats, and the like.

[0169] In some embodiments, the methods of the present invention, including the method of predicting a subject's genotype involving implementing a neural network model, are executed on one or more computers in a distributed computing system environment (e.g., in a cloud computing environment). In this specification, "cloud computing" is defined as a model for enabling on-demand network access to a shared set of configurable computing resources. Cloud computing can be used to provide on-demand access to a shared set of configurable computing resources. The shared set of configurable computing resources can be rapidly provisioned through virtualization, released with low management effort or service provider interaction, and scaled accordingly. The cloud computing model can be configured with various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid scalability, metered services, etc. The cloud computing model can also expose various service models, such as, for example, software as a service ("Software as a Service, SaaS"), platform as a service ("Platform as a Service, PaaS"), and infrastructure as a service ("Infrastructure as a Service, IaaS"). Cloud computing models may also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, etc. As used herein, a "cloud computing environment" is an environment in which cloud computing is used.

[0170] Figure 13 illustrates an exemplary computer for implementing the entities illustrated in Figures 9, 10, 11, and 12. The computer 1400 includes at least one processor 1402 coupled to a chipset 1404. The chipset 1404 includes a memory controller hub 1420 and an input / output (I / O) controller hub 1422. The memory 1406 and the graphics adapter 1412 are coupled to the memory controller hub 1420, and the display 1418 is coupled to the graphics adapter 1412. The storage device 1408, the input device 1414, and the network adapter 1416 are coupled to the I / O controller hub 1422. Other embodiments of the computer 1400 have different architectures.

[0171] The storage device 1408 is a non-transitory computer-readable storage medium such as a hard drive, a compact disk read-only memory (CD-ROM), a DVD, or a solid-state memory device. The memory 1406 holds instructions and data used by the processor 1402. The input interface 1414 is a touch screen interface, a mouse, a trackball, or other type of pointing device, a keyboard, or some combination thereof, and is used to input data into the computer 1400. In some embodiments, the computer 1400 may be configured to receive input (e.g., commands) from the input interface 1414 via gestures from a user. The network adapter 1416 couples the computer 1400 to one or more computer networks.

[0172] The graphics adapter 1412 displays images and other information on a display 1418. In various embodiments, the display 1418 is configured to allow a user (e.g., a radiologist, oncologist, pulmonologist) to input user selections on the display 1418, for example, to predict the patient's genotype or to order any additional tests or procedures. In one embodiment, the display 1418 may include a touch interface. In various embodiments, the display 1418 may show one or more genotypes predicted for the subject. Thus, a user accessing the display 1418 may inform the subject of the predicted genotype for the subject. In various embodiments, the display 1418 may show information such as the individual tiles of the image that contributed most significantly to the predicted genotype for the subject. For example, such information may be useful to verify that the predicted genotype was due to a particular attribute in the tissue as opposed to an image artifact.

[0173] The computer 1400 is adapted to execute computer program modules for providing functionality described herein. As used herein, the term "module" refers to computer program logic used to provide a specified function. Thus, a module may be implemented in hardware, firmware, and / or software. In one embodiment, the program module is stored in the storage device 1408, loaded into the memory 1406, and executed by the processor 1402.

[0174] The type of computer 1400 used by the entities of Figure 9 or Figure 10 can vary depending on the embodiment and the processing power required by the entities. For example, the genotype prediction system 130 can be implemented on a single computer 1400 or on multiple computers 1400 that communicate with each other via a network such as a server farm. The computer 1400 can lack some of the components described above, such as the graphics adapter 1412 and the display 1418.

[0175] system Further disclosed herein is a system for implementing a neural network model for predicting a subject's genotype. In various embodiments, such a system may include at least the genotype prediction system 130 described above in FIG. 9. In various embodiments, the genotype prediction system 130 is embodied as a computer system, such as the computer system having the exemplary computer 1400 described in FIG. 13.

[0176] In various embodiments, the system includes an imaging device, such as the imaging generation system 120 described above in FIG. 9. In various embodiments, the system includes both a genotype prediction system 130 (e.g., a computer system) and the imaging generation system 120. In such embodiments, the genotype prediction system 130 may be communicatively coupled to the imaging generation system 120 to receive images captured from the subject. Thus, the genotype prediction system 130 implements a neural network model in silico to analyze the images and determine a prediction of a genotype for the subject. EXAMPLES

[0177] Below are examples of specific embodiments for carrying out the present invention. The examples are provided for illustrative purposes only and are not intended to limit the scope of the present invention in any manner. Although efforts have been made to ensure accuracy with respect to the numerical values ​​used, some experimental error and deviation should be allowed for.

[0178] Example 1: First implementation of a neural network model and corresponding problem The goal was to develop and implement a machine learning model to predict patient mutations, thereby enabling pre-screening of patients for eligibility in clinical trials. Using in silico methods to pre-screen patients would reduce the number of patients who need to undergo molecular screening. Furthermore, in silico methods can be fast and cost-effective, which can prompt physicians to send patients who are predicted to have a high probability of having eligible mutations for screening. In short, inexpensive mutation screening tools can lower the barrier to testing in real-world clinical settings and help match patients to the right treatment.

[0179] FIG. 14 depicts a first exemplary process for generating patient-level genotype predictions using slide images. Here, whole-slide H&E images are analyzed. Slides are divided into individual tiles, and then tile-level predictions are performed and analyzed using a neural network. These tile-level predictions are aggregated into a patient-level prediction. Here, the aggregation of the tile-level predictions is done by selecting the highest tile-level prediction (i.e., the maximum aggregation) within the slide. Further description of this first exemplary process is disclosed in Campanella, G., et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med 25, 1301-1309 (2019), the entire contents of which are incorporated herein by reference.

[0180] However, two main problems arose with this analysis. Slide quality (i.e. smudges, pen marks, dust / stains) Batch variance / variability

[0181] These issues are explained in further detail in Example 2 below.

[0182] Example 2: Second implementation of a neural network model incorporating image preprocessing To address the modeling challenges discussed above and improve model performance, a second implementation of the neural network model was developed that includes a quality control step (e.g., slide quality challenge) and a data augmentation step (e.g., batch differences).

[0183] FIG. 15 depicts a second exemplary process for generating patient-level genotype predictions using slide images. Here, whole slide images were divided into smaller tiles. Quality control and data augmentation steps were then performed on each individual tile. The quality control step removed pen marks and background. Pen marks and background intensity are image artifacts that can adversely affect the performance of the neural network analyzing the tiles. The data augmentation step involved stain-based augmentation, which increased and diversified the number of differently stained images in the training set. This allowed the trained neural network to be trained on tiles of different staining intensity and coloration, thus making predictions by the neural network less affected by staining variations across different tiles and samples.

[0184] The preprocessed tiles were provided as input to a neural network that generated tile-level predictions, each indicating whether a particular tile was likely to correspond to an altered or wild-type state. The tile-level predictions were aggregated to generate slide-level predictions, where the aggregation of tile-level predictions was performed by selecting the highest tile-level predictions within a slide (i.e., the maximum aggregation).

[0185] In general, the performance of the neural network improved when incorporating pre-processing steps (e.g., quality control and data augmentation steps). First, FIG. 16A shows the lack of cross-cohort generalizability when TCGA and Tempus samples were differentiated. To verify that the improvement was at least partially due to reduced bias against pen artifacts, the neural network was analyzed when the pre-processing step was implemented and not implemented. FIG. 16B shows the neural network's bias against pen artifacts when the pre-processing step was not implemented. Notably, the neural network treated the TCGA samples significantly differently from the Test #1 samples (e.g., as shown by their different cluster locations) due to differences in the pen marks on the images. This indicates that the presence of pen marks adversely affected the neural network's predictions. Conversely, FIG. 16C shows the improved treatment of the neural network for TCGA and Test #1 samples when the pre-processing step was implemented. Notably, Figure 16C shows that the neural network treated the TCGA and Test #1 samples without bias, as the tiles were indistinguishably clustered together, indicating that the presence of the pen marks did not affect the neural network's predictions.

[0186] 17A depicts the performance of a model deployed in a second exemplary process (described in FIG. 15) compared to a model deployed in a first exemplary process (described in FIG. 14). Specifically, "Version 1" in FIG. 17A refers to the first exemplary process described in FIG. 14, and "Version 2" in FIG. 17A refers to the second exemplary process described in FIG. 15.

[0187] Generally, the incorporation of "Version 2," e.g., the preprocessing steps in the second exemplary process (illustrated in FIG. 15), improved the performance of the neural network when analyzing samples from both TCGA and Study #1. As shown in FIG. 17A, each of the performance metrics (e.g., area under the receiver operating curve (auROC) and positive predictive value (PPV)) improved when using "Version 2."

[0188] FIG. 17B illustrates the performance of the model tested on TCGA bladder cancer slides according to the second exemplary process (described in FIG. 15). Here, the TCGA samples included a total of 407 TCGA bladder cancer slides. 275 slides were used for training, 62 slides were used for validation, and 70 slides were used for testing. 12.5% ​​of the samples had a known eligible fibroblast growth factor receptor (FGFR) alteration (e.g., mutation, fusion, or both). The results shown in FIG. 17B highlight that the model deployed according to the second exemplary process described in FIG. 15 was able to achieve a PPV of 0.57 with a recall of 100%, which reduces the total screening cases by about 75%. This therefore indicates that the inclusion of a pre-processing step significantly improved the model's performance.

[0189] Example 3: A third implementation of a neural network model in which aggregations of tile-level features are learned To further improve the performance of the neural network model compared to the models described in Examples 1 and 2, the neural network model was further constructed using an attention network so that aggregations of tile-level features (e.g., normalized weighted values ​​for the feature vectors of each tile used to summarize all tiles of a particular image) are learned. Such a neural network model can achieve more reliable and informative probability scores and can be more robust against image artifacts. Furthermore, it is desirable to have a trainable aggregation step so that the model automatically learns how to aggregate multiple tiles from the data to give accurate predictions.

[0190] Figure 18 depicts a third exemplary process for generating patient-level genotype predictions using slide images, where the third exemplary process involves the implementation of an attention network that has learned how to aggregate tile-level features to generate slide-level predictions. As shown in Figure 18, the third implementation further includes a pre-processing step.

[0191] FIG. 19 depicts the performance of a model deployed in a third exemplary process (described in FIG. 18) compared to a model deployed in a first exemplary process (described in FIG. 14), where the model was tested against the Tempus dataset including 512 tissue slides with a known prevalence of 11% of known variations. Additionally, "Version 1" in FIG. 19 refers to the first exemplary process described in FIG. 14, and "Version 3" in FIG. 19 refers to the third exemplary process described in FIG. 18.

[0192] First, the model of the third exemplary process was able to achieve improved performance with significantly reduced training time. That is, the "Version 1" model required 3 days of training time, while the "Version 3" model required only 3 hours of training time. Here, a pre-trained CNN may be used for Version 3, and thus the training focus for the Version 3 model is on the attention network. In contrast, in Version 1, the CNN is trained using fixed / unlearnable aggregation, such as the maximum aggregation approach. Because the CNN contains more parameters (i.e., it looks at the 2D image of each tile instead of the probability / feature per tile), training the CNN (as required in the Version 1 model) takes much longer than training the attention model (as required in the Version 3 model). Additionally, even with the reduced training time, the "Version 3" model achieved higher performance metrics (e.g., auROC value=0.71 and PPV=0.14) compared to the "Version 1" model (auROC=0.65 and PPV=0.12).

[0193] Furthermore, considering that the "Version 3" model also incorporated a pre-processing step, the "Version 3" model was more robust to image artifacts such as pen marks compared to the "Version 1" model. Specifically, FIG. 20 shows heat maps of the first exemplary process (illustrated in FIG. 14) and the third exemplary process (illustrated in FIG. 8), respectively, which demonstrate that the third exemplary process was more robust to image artifacts such as pen marks.

[0194] In summary, these results show that the third embodiment predicts subject genotypes with reduced training and adequate performance.

[0195] Example 4: Fourth implementation of a neural network model in which the first and second sub-models are jointly trained FIG. 21 depicts a fourth exemplary process for generating patient-level genotype predictions using slide images. Here, the fourth exemplary process shown in FIG. 21 is an "End2End" model, which means that both the first sub-model (e.g., a neural network that generates tile-level features) and the second sub-model (e.g., an attention network that learns the best aggregation of tile-level features) are trained together. As shown in FIG. 21, the slide-level predictions are analyzed for their accuracy (e.g., as measured by the difference between the ground truth and the slide-level predictions), and the values ​​are back-propagated to jointly train the parameters of the neural network and the attention network.

[0196] The "End2End" version improves on version 3 (described in Example 3) by jointly training a convolutional neural network with an attention module. A single backpropagation loop updates the attention weights in a similar manner to the CNN, which allows the CNN to learn tissue pathology-based features.

[0197] The "End2End" version also performs stain enhancement on-the-fly using color jittering along with flipping / rotation, thus eliminating the need to do it manually beforehand. Memory constraints due to the large size of training patches (tiles) per slide are addressed by randomly sampling "N" patches (tiles) per slide in each batch, thus fitting the network into the GPU memory.

[0198] Example 5: Exemplary deployment of a pipeline workflow FIG. 22 depicts an exemplary pipeline workflow. In particular, FIG. 22 shows several quality control steps that ensure high quality images are analyzed by the neural network. Starting at the top left of FIG. 22, the input to the pipeline workflow includes the image file and associated metadata. The quality control step checks that the tissue is either bladder / prostate tissue and that a 10x zoom image is available. If any of these criteria are not met, an indication may be provided that either criterion was not met. If the criteria are met, the image is pre-processed and the tile positions within the slide are calculated. A second quality control step verifies that the tile positions are successfully calculated within the image. If this step fails, an error may be provided. If not, the tiles are individually assessed for the presence of tissue, artifacts, and / or background.

[0199] Tiles are assigned a quality score according to the presence of tissue, artifacts, and / or background. For example, the presence of tissue will increase the quality score, and the presence of artifacts and / or background will decrease the quality score. Further details of the steps for determining the quality score are shown in Figure 22. A slide with N or more eligible tiles is input to a neural network, which is further connected to an attention network. Thus, the attention network outputs a binary prediction.

[0200] Additional Embodiments The embodiments of the invention disclosed herein include a pipeline workflow for predicting the genotype of a target tumor based on cancer tissue images captured from a sample obtained from a subject. In general, the pipeline workflow involves 1) pre-processing the image, and 2) deploying a trained neural network that analyzes individual tiles of the image and generates a prediction of the genotype of the target tumor. In various embodiments, the trained neural network includes a convolutional neural network that analyzes the individual tiles and generates tile-level predictions, or in alternative embodiments, tile-level features. In various embodiments, the trained neural network further includes an attention network that analyzes the tile-level predictions, or alternatively, the tile-level features, and aggregates the tile-level predictions to generate a prediction of the genotype of the target tumor, or alternatively, aggregates the tile-level features to generate an image-level feature vector that is used by a connected classification layer (or layers) to generate a prediction of the genotype of the target tumor. In summary, the implementation of the pipeline workflow enables accurate prediction of a target genotype that may be a relevant driver in a particular cancer (e.g., bladder cancer or prostate cancer). Thus, this in silico process represents a rapid, low-cost procedure for determining whether a subject has a particular cancer-associated genotype, which can further guide treatment options for the subject.

[0201] In various embodiments, subjects predicted to exhibit an altered genotype may undergo further molecular testing to confirm the in silico results. In various embodiments, once confirmed through molecular testing, the subject may be provided with appropriate medical treatment or medical advice based on the altered genotype status. In various embodiments, once confirmed, these subjects may be considered eligible for enrollment in a clinical trial. In various embodiments, subjects predicted to exhibit a wild-type genotype are likely to be ineligible for a clinical trial and do not need to undergo further molecular testing, thereby saving resources (e.g., time and money). In various embodiments, the methods and / or non-transitory computer readable media described herein may operate as a software as a medical device (SaMD) provided on disk or via download, or as web-based software. In such embodiments, the methods described herein may operate independently of a clinical trial environment.

[0202] Disclosed herein is a method that includes acquiring or having acquired images of tumor tissue slides from a subject, pre-processing the acquired images, applying a neural network model to analyze a plurality of tiles of the pre-processed images to generate slide level predictions, and determining the genotype of the subject using the slide level predictions.

[0203] In various embodiments, pre-processing the acquired image includes performing a quality control process by removing tiles with excessive background and / or pen marks present in the image. In various embodiments, pre-processing the acquired image includes one or more of removing uninformative regions of one or more tiles of the image, performing image stain enhancement of the acquired image. In various embodiments, performing image stain enhancement includes one or more of modifying stain intensity or stain contrast of one or more tiles of the acquired image, performing color jittering or color normalization of one or more tiles of the acquired image, and performing horizontal or vertical flip of one or more tiles of the acquired image. In various embodiments, pre-processing the acquired image is performed through an automated process.

[0204] In various embodiments, the neural network model includes a convolutional neural network sub-model and an attention network sub-model. In various embodiments, the convolutional neural network sub-model generates a tile-level prediction for one or more tiles of the preprocessed image. In various embodiments, the attention network sub-model receives the tile-level prediction of the convolutional neural network sub-model and generates an image-level prediction. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more layers of nodes. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more learned parameters. In various embodiments, the one or more learned parameters of the attention network sub-model guide the aggregation of the tile-level predictions from the convolutional neural network sub-model.

[0205] In various embodiments, the image level prediction is the presence or absence of a genotypic alteration. In various embodiments, the genotypic alteration comprises an FGFR alteration. In various embodiments, the FGFR alteration comprises either an FGFR3 mutation, an FGFR3 fusion, or a combination of an FGFR3 mutation and an FGFR3 fusion. In various embodiments, the FGFR alteration is either a p.R248C mutation, a p.G370C mutation, a p.S249C mutation, or a p.Y373C mutation. In various embodiments, the FGFR alteration is either an FGFR3:TACC3V1 fusion, an FGFR3:TACC3V3 fusion, an FGFR3:BAIAP2L1 fusion, an FGFR2:BICC1 fusion, or an FGFR2:CASP7 fusion.

[0206] In various embodiments, the tumor tissue slide from the subject comprises bladder cancer. In various embodiments, the genotypic alteration comprises a single nucleotide polymorphism (SNP), copy number variation (CNV), or gene fusion involving the BRCA1 or BRCA2 genes. In various embodiments, the tumor tissue slide from the subject comprises prostate cancer. In various embodiments, the image is a hematoxylin and eosin (H&E) stained histopathology image. In various embodiments, the subject has cancer or is suspected of having cancer.

[0207] In various embodiments, the methods disclosed herein further include determining, based on the determined genotype, whether to perform further molecular testing to confirm the determined genotype. In various embodiments, determining whether to perform further molecular testing includes prioritizing the subject for undergoing further molecular testing in response to the determined genotype indicating a genotypic variation. In various embodiments, determining whether to perform further molecular testing includes excluding the subject from undergoing further molecular testing in response to the determined genotype indicating a wild type genotype.

[0208] In various embodiments, the method disclosed herein further comprises determining whether to enroll the subject in a clinical trial according to at least the determined genotype. In various embodiments, determining whether to enroll the subject comprises determining that the determined genotype includes a genotypic variation, and determining that the subject is eligible to enroll in the clinical trial based at least on the determination that the genotype includes a genotypic variation. In various embodiments, determining that the subject is eligible to enroll in the clinical trial is further based on a molecular test confirming that the subject exhibits a genotype that includes a genotypic variation. In various embodiments, determining whether to enroll the subject comprises determining that the determined genotype does not include a genotypic variation, and determining that the subject is ineligible to enroll in the clinical trial based at least on the determination that the genotype does not include a genotypic variation. In various embodiments, based on the determination that the determined genotype does not include a genotypic variation, the subject does not undergo further molecular testing.

[0209] In various embodiments, the method disclosed herein further comprises determining whether to administer a therapeutic agent according to at least the determined genotype. In various embodiments, the therapeutic agent is an FGFR kinase inhibitor. In various embodiments, the FGFR kinase inhibitor is erdafitinib (BALVERSA). In various embodiments, the therapeutic agent is a PARP inhibitor. In various embodiments, the PARP inhibitor is niraparib (ZEJULA). In various embodiments, the neural network model exhibits an auROC performance metric of at least 0.82 for bladder cancer images. In various embodiments, the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.22 with 100% recall for bladder cancer images in a test dataset having a baseline prevalence of 14% for FGFR. In various embodiments, the neural network model exhibits an auROC performance metric of at least 0.71 for prostate cancer images. In various embodiments, the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.14 with 100% recall for prostate cancer images in a testing dataset with a baseline prevalence of 11% for PARP. In some examples, neural network models consistent with the present disclosure exhibit an auROC performance metric of at least 0.78+ / -0.03 for predicting MET genotypes in the context of non-small cell lung cancer tissue images. In various embodiments, the methods disclosed herein further include reporting one or more tiles of the image that are most strongly associated with the genotype of interest.

[0210] Additionally disclosed herein is a method that includes acquiring or having acquired training data including training histopathology images, and training a neural network model by analyzing a plurality of tiles of the training histopathology images, where the neural network model is configured to generate informative image-level predictions to determine a genotype corresponding to the training histopathology images. In various embodiments, the neural network model includes a convolutional neural network sub-model and an attention network sub-model. In various embodiments, the attention network sub-model is trained using training data including the training histopathology images. In various embodiments, the convolutional neural network sub-model and the attention network sub-model are trained separately. In various embodiments, the convolutional neural network sub-model and the attention network sub-model are trained jointly using the training data.

[0211] In various embodiments, the training data further comprises reference ground truth labels indicative of the genotypes of the training histopathology images. In various embodiments, the training histopathology images are obtained from a publicly available database. In various embodiments, the publicly available database is The Cancer Genome Atlas (TCGA) database. In various embodiments, the training histopathology images are obtained from a proprietary database.

[0212] In various embodiments, the convolutional neural network sub-model generates a tile-level prediction for one or more tiles of the preprocessed image. In various embodiments, the attention network sub-model receives the tile-level prediction of the neural network sub-model and generates an image-level prediction. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more layers of nodes. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more learned parameters.

[0213] Additionally disclosed herein is a non-transitory computer readable medium including instructions that, when executed by a processor, cause the processor to acquire an image of a tumor tissue slide from a subject, pre-process the acquired image, apply a neural network model to analyze a plurality of tiles of the pre-processed image to generate a slide level prediction, and use the slide level prediction to determine the genotype of the subject. In various embodiments, the instructions that cause the processor to pre-process the acquired image, when executed by the processor, further include instructions that cause the processor to perform a quality control process by removing tiles with image background and / or pen marks present in the image. In various embodiments, the instructions that cause the processor to pre-process the acquired image, when executed by the processor, further include instructions that cause the processor to perform one or both of removing uninformative regions of one or more tiles of the image and performing image stain enhancement of the acquired image. In various embodiments, the instructions that cause the processor to perform image stain enhancement further include instructions that, when executed by the processor, cause the processor to vary stain intensity or stain contrast of one or more tiles of the acquired image, color jitter or color normalize one or more tiles of the acquired image, and perform horizontal or vertical flip of one or more tiles of the acquired image. In various embodiments, the instructions that cause the processor to preprocess the acquired image enable the processor to perform the preprocessing through an automated process.

[0214] In various embodiments, the neural network model includes a convolutional neural network sub-model and an attention network sub-model. In various embodiments, the convolutional neural network sub-model generates a tile-level prediction for one or more tiles of the preprocessed image. In various embodiments, the attention network sub-model receives the tile-level prediction of the convolutional neural network sub-model and generates an image-level prediction. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more layers of nodes. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more learned parameters. In various embodiments, the one or more learned parameters of the attention network sub-model guide the aggregation of the tile-level predictions from the convolutional neural network sub-model.

[0215] In various embodiments, the image level prediction is the presence or absence of a genotypic alteration. In various embodiments, the genotypic alteration comprises an FGFR alteration. In various embodiments, the FGFR alteration comprises either an FGFR3 mutation, an FGFR3 fusion, or a combination of an FGFR3 mutation and an FGFR3 fusion. In various embodiments, the FGFR alteration is either a p.R248C mutation, a p.G370C mutation, a p.S249C mutation, or a p.Y373C mutation. In various embodiments, the FGFR alteration is either an FGFR3:TACC3V1 fusion, an FGFR3:TACC3V3 fusion, an FGFR3:BAIAP2L1 fusion, an FGFR2:BICC1 fusion, or an FGFR2:CASP7 fusion.

[0216] In various embodiments, the tumor tissue slide from the subject comprises bladder cancer. In various embodiments, the genotypic alteration comprises a single nucleotide polymorphism (SNP), copy number variation (CNV), or gene fusion involving the BRCA1 or BRCA2 genes. In various embodiments, the tumor tissue slide from the subject comprises prostate cancer. In various embodiments, the image is a hematoxylin and eosin (H&E) stained histopathology image. In various embodiments, the subject has cancer or is suspected of having cancer.

[0217] In various embodiments, the non-transitory computer readable medium further comprises instructions, which when executed by a processor, cause the processor to determine, based on the determined genotype, whether to perform further molecular testing to confirm the determined genotype. In various embodiments, the instructions, which when executed by the processor, cause the processor to determine whether to perform further molecular testing further comprise instructions, which when executed by the processor, cause the processor to prioritize the subject for undergoing further molecular testing in response to the determined genotype indicating a genotypic variation. In various embodiments, the instructions, which when executed by the processor, cause the processor to determine whether to perform further molecular testing further comprise instructions, which when executed by the processor, cause the processor to exclude the subject from undergoing further molecular testing in response to the determined genotype indicating a wild type genotype.

[0218] In various embodiments, the non-transitory computer readable medium further comprises instructions, which when executed by the processor, cause the processor to determine whether to enroll the subject in a clinical trial according to at least the determined genotype. In various embodiments, the instructions, which when executed by the processor, cause the processor to determine that the determined genotype includes a genotypic variation and determine that the subject is eligible for enrollment in the clinical trial based at least on the determination that the genotype includes a genotypic variation. In various embodiments, the determination that the subject is eligible for enrollment in the clinical trial is further based on a molecular test that confirms that the subject exhibits a genotype that includes a genotypic variation. In various embodiments, the instructions, which when executed by the processor, cause the processor to determine that the determined genotype does not include a genotypic variation and further comprise instructions that determine that the subject is ineligible for enrollment in the clinical trial based at least on the determination that the genotype does not include a genotypic variation. In various embodiments, based on the determination that the determined genotype does not include a genotypic variation, the subject does not undergo further molecular testing.

[0219] In various embodiments, the non-transitory computer readable medium further comprises instructions that, when executed by a processor, cause the processor to determine whether to administer a therapeutic agent according to at least the determined genotype. In various embodiments, the therapeutic agent is an FGFR kinase inhibitor. In various embodiments, the FGFR kinase inhibitor is erdafitinib (BALVERSA). In various embodiments, the therapeutic agent is a PARP inhibitor. In various embodiments, the PARP inhibitor is niraparib (ZEJULA).

[0220] In various embodiments, the neural network model exhibits an auROC performance metric of at least 0.82 for a bladder cancer model. In various embodiments, the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.22 at 100% recall for a bladder cancer model in a test dataset having a baseline prevalence of 14% for FGFR. In various embodiments, the neural network model exhibits an auROC performance metric of at least 0.71 for a prostate cancer model. In various embodiments, the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.14 at 100% recall for prostate cancer images in a test dataset having a baseline prevalence of 11% for PARP. In some examples, neural network models consistent with the present disclosure exhibit an auROC performance metric of at least 0.78+ / -0.03 for predicting MET genotypes in the context of non-small cell lung cancer tissue images. In various embodiments, the non-transitory computer readable medium further includes instructions that, when executed by the processor, cause the processor to report one or more tiles of the image that are most strongly associated with the genotype of the subject.

[0221] Additionally disclosed herein is a non-transitory computer readable medium including instructions that, when executed by a processor, cause the processor to obtain training data including training histopathology images and train a neural network model by analyzing a plurality of tiles of the training histopathology images, where the neural network model is configured to generate informative image-level predictions to determine genotypes corresponding to the training histopathology images. In various embodiments, the neural network model includes a convolutional neural network sub-model and an attention network sub-model. In various embodiments, the attention network sub-model is trained using training data including the training histopathology images. In various embodiments, the convolutional neural network sub-model and the attention network sub-model are trained separately. In various embodiments, the convolutional neural network sub-model and the attention network sub-model are trained jointly using the training data.

[0222] In various embodiments, the training data further comprises reference ground truth labels indicative of the genotypes of the training histopathology images. In various embodiments, the training histopathology images are obtained from a publicly available database. In various embodiments, the publicly available database is The Cancer Genome Atlas (TCGA) database. In various embodiments, the training histopathology images are obtained from a proprietary database.

[0223] In various embodiments, the convolutional neural network sub-model generates a tile-level prediction for one or more tiles of the preprocessed image. In various embodiments, the attention network sub-model receives the tile-level prediction of the neural network sub-model and generates an image-level prediction. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more layers of nodes. In various embodiments, the convolutional neural network sub-model and the attention network sub-model each comprise one or more learned parameters.

[0224] Embodiment 1: A method comprising: acquiring or having acquired an image of a tumor tissue slide from a subject; pre-processing the acquired image; applying a neural network model to analyze a plurality of tiles of the pre-processed image to generate slide level predictions, or optionally slide level features; and determining the genotype of the subject using the slide level predictions or slide level features.

[0225] Embodiment 2: The method of embodiment 1, wherein pre-processing the acquired image includes performing a quality control process by removing image background and / or tiles with pen marks present in the image.

[0226] Embodiment 3: The method of embodiment 1, wherein pre-processing the acquired image includes one or more of removing uninformative regions of one or more tiles of the image, and performing image staining enhancement of the acquired image.

[0227] Embodiment 4: The method of embodiment 3, wherein performing image staining enhancement comprises performing one or more of the following: modifying staining intensity or staining control of one or more tiles of the acquired image; performing color jittering or color normalization of one or more tiles of the acquired image; and performing horizontal or vertical flipping of one or more tiles of the acquired image.

[0228] Embodiment 5: The method of any one of embodiments 1 to 4, wherein pre-processing the acquired images is performed through an automated process.

[0229] Embodiment 6: The method of embodiment 1, wherein the neural network model includes a convolutional neural network sub-model and an attention network sub-model.

[0230] Embodiment 7: The method of embodiment 6, wherein the convolutional neural network sub-model generates tile-level predictions for one or more tiles of the preprocessed image.

[0231] Embodiment 8: The method of embodiment 6 or 7, wherein the attention network sub-model receives the tile-level predictions of the convolutional neural network sub-model and generates image-level predictions.

[0232] Embodiment 9: The method according to any one of embodiments 6 to 8, wherein the convolutional neural network sub-model and the attention network sub-model each comprise one or more layers of nodes.

[0233] Embodiment 10: The method according to any one of embodiments 6 to 9, wherein the convolutional neural network sub-model and the attention network sub-model each comprise one or more learned parameters.

[0234] Embodiment 11: The method of embodiment 10, wherein one or more learned parameters of the attention network sub-model guide the aggregation of tile-level predictions from the convolutional neural network sub-model.

[0235] Embodiment 12: The method according to any one of embodiments 1 to 11, wherein the image level prediction is the presence or absence of a genotypic alteration.

[0236] Embodiment 13: The method of embodiment 12, wherein the genotypic alteration comprises an FGFR alteration.

[0237] Embodiment 14: The method of embodiment 13, wherein the FGFR alteration comprises either an FGFR3 mutation, an FGFR3 fusion, or a combination of an FGFR3 mutation and an FGFR3 fusion.

[0238] Embodiment 15: The method of embodiment 13, wherein the FGFR alteration is any of a p.R248C mutation, a p.G370C mutation, a p.S249C mutation, or a p.Y373C mutation.

[0239] Embodiment 16: The method of embodiment 13, wherein the FGFR alteration is any of a FGFR3:TACC3V1 fusion, a FGFR3:TACC3V3 fusion, a FGFR3:BAIAP2L1 fusion, a FGFR2:BICC1 fusion, or a FGFR2:CASP7 fusion.

[0240] Embodiment 17: The method of any one of embodiments 13 to 16, wherein the tumor tissue slide from the subject comprises bladder cancer.

[0241] Embodiment 18: The method of embodiment 12, wherein the genotypic variation comprises a single nucleotide polymorphism (SNP), a copy number variation (CNV), or a gene fusion involving the BRCA1 or BRCA2 gene.

[0242] Embodiment 19: The method of embodiment 18, wherein the tumor tissue slide from the subject comprises prostate cancer.

[0243] Embodiment 20: The method of any one of embodiments 1 to 19, wherein the image is a hematoxylin and eosin (H&E) stained histopathology image.

[0244] Embodiment 21: The method of any one of embodiments 1 to 20, wherein the subject has or is suspected of having cancer.

[0245] Embodiment 22: The method according to any one of embodiments 1 to 21, further comprising determining whether to perform further molecular testing to confirm the determined genotype based on the determined genotype.

[0246] Embodiment 23: The method of embodiment 22, wherein determining whether to perform further molecular testing comprises prioritizing subjects for undergoing further molecular testing in response to the determined genotype indicating a genotypic change.

[0247] Embodiment 24: The method of embodiment 22, wherein determining whether to perform further molecular testing comprises excluding the subject from undergoing further molecular testing in response to the determined genotype indicating a wildtype genotype.

[0248] Embodiment 25: The method according to any one of embodiments 1 to 21, further comprising determining whether to enroll the subject in a clinical trial at least according to the determined genotype.

[0249] Embodiment 26: The method of embodiment 22, wherein determining whether to enroll the subject comprises determining that the determined genotype comprises a genotypic variation, and determining that the subject is eligible for enrollment in the clinical trial based at least on a determination that the genotype comprises a genotypic variation.

[0250] Embodiment 27: The method of embodiment 26, wherein determining that the subject is eligible to enroll in the clinical trial is further based on a molecular test confirming that the subject exhibits a genotype that includes a genotypic alteration.

[0251] Embodiment 20: The method of embodiment 22, wherein determining whether to enroll the subject comprises determining that the determined genotype does not comprise a genotypic variation, and determining that the subject is ineligible for enrollment in the clinical trial based at least on a determination that the genotype does not comprise a genotypic variation.

[0252] Embodiment 29: The method of embodiment 28, wherein the subject does not undergo further molecular testing based on a determination that the determined genotype does not contain a genotypic alteration.

[0253] Embodiment 30: The method of any one of embodiments 1 to 22, further comprising determining whether to administer a therapeutic agent according to at least the determined genotype.

[0254] Embodiment 31: The method of embodiment 30, wherein the therapeutic agent is an FGFR kinase inhibitor.

[0255] Embodiment 32: The method of embodiment 31, wherein the FGFR kinase inhibitor is erdafitinib (BALVERSA).

[0256] Embodiment 33: The method of embodiment 30, wherein the therapeutic agent is a PARP inhibitor.

[0257] Embodiment 34: The method of embodiment 33, wherein the PARP inhibitor is niraparib (ZEJULA).

[0258] Embodiment 35: The method described in any one of embodiments 1 to 5 and 12 to 34, wherein the neural network model exhibits an auROC performance metric of at least 0.82 for the bladder cancer model.

[0259] Embodiment 36: The method of any one of embodiments 1 to 5 and 12 to 34, wherein the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.22 with 100% recall for bladder cancer images in a test dataset having a baseline prevalence of 14% for FGFR.

[0260] Embodiment 37: The method described in any one of embodiments 6 to 34, wherein the neural network model exhibits an auROC performance metric of at least 0.71 for prostate cancer images.

[0261] Embodiment 37.5: The method of any one of embodiments 6 to 34, wherein the neural network model exhibits an auROC performance metric of at least 0.78+ / -0.03 for predicting MET genotypes on non-small cell lung cancer images.

[0262] Embodiment 38: The method according to any one of embodiments 6 to 34, wherein the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.14 with 100% recall against a prostate cancer model in a test dataset having a baseline prevalence of 11% for PARP.

[0263] Embodiment 39: The method of any one of embodiments 1 to 38, further comprising reporting one or more tiles of the image that are most strongly associated with the subject's genotype.

[0264] Embodiment 40: A method comprising: acquiring or having acquired an image of a tumor tissue slide from a subject; pre-processing the acquired image; applying a neural network model to analyze a plurality of tiles of the pre-processed image to generate a slide level prediction; and determining the genotype of the subject using the slide level prediction.

[0265] Embodiment 41: The method described in embodiment 40, wherein pre-processing the acquired image includes performing a quality control process by removing image background and / or tiles with pen marks present in the image.

[0266] Embodiment 42: The method described in embodiment 40, wherein preprocessing the acquired image includes one or more of removing uninformative regions of one or more tiles of the image, and performing image staining enhancement of the acquired image.

[0267] Embodiment 43: The method of embodiment 42, wherein performing image staining enhancement comprises performing one or more of the following: modifying staining intensity or staining contrast of one or more tiles of the acquired image; performing color jittering or color normalization of one or more tiles of the acquired image; and performing horizontal or vertical flipping of one or more tiles of the acquired image.

[0268] Embodiment 44: The method of any one of embodiments 40 to 43, wherein pre-processing of the acquired image is performed through an automated process.

[0269] Embodiment 45: The method of embodiment 40, wherein the neural network model includes a convolutional neural network sub-model and an attention network sub-model.

[0270] Embodiment 46: The method described in embodiment 45, wherein the convolutional neural network sub-model generates tile-level predictions for one or more tiles of the preprocessed image.

[0271] Embodiment 47: The method described in embodiment 45 or 46, wherein the attention network sub-model receives tile-level predictions of the convolutional neural network sub-model and generates image-level predictions.

[0272] Embodiment 48: A method according to any one of embodiments 45 to 47, wherein the convolutional neural network sub-model and the attention network sub-model each comprise one or more layers of nodes.

[0273] Embodiment 49: A method according to any one of embodiments 45 to 48, wherein the convolutional neural network sub-model and the attention network sub-model each comprise one or more learned parameters.

[0274] Embodiment 50: The method of embodiment 49, wherein one or more learned parameters of the attention network sub-model guide the aggregation of tile-level predictions from the convolutional neural network sub-model.

[0275] Embodiment 51: The method according to any one of embodiments 40 to 50, wherein the image level prediction is the presence or absence of a genotypic alteration.

[0276] Embodiment 52: The method of embodiment 51, wherein the genotypic alteration comprises an FGFR alteration.

[0277] Embodiment 53: The method of embodiment 52, wherein the FGFR alteration comprises either an FGFR3 mutation, an FGFR3 fusion, or a combination of an FGFR3 mutation and an FGFR3 fusion.

[0278] Embodiment 54: The method of embodiment 52, wherein the FGFR alteration is either a p.R248C mutation, a p.G370C mutation, a p.S249C mutation, or a p.Y373C mutation.

[0279] Embodiment 55: The method of embodiment 52, wherein the FGFR alteration is any of a FGFR3:TACC3V1 fusion, a FGFR3:TACC3V3 fusion, a FGFR3:BAIAP2L1 fusion, a FGFR2:BICC1 fusion, or a FGFR2:CASP7 fusion.

[0280] Embodiment 56: The method of any one of embodiments 52 to 55, wherein the tumor tissue slide from the subject contains bladder cancer.

[0281] Embodiment 57: The method of embodiment 51, wherein the genotypic variation comprises a single nucleotide polymorphism (SNP), a copy number variation (CNV), or a gene fusion involving the BRCA1 or BRCA2 gene.

[0282] Embodiment 58: The method of embodiment 57, wherein the tumor tissue slide from the subject contains prostate cancer.

[0283] Embodiment 59: The method of any one of embodiments 40 to 58, wherein the image is a hematoxylin and eosin (H&E) stained histopathology image.

[0284] Embodiment 60: The method of any one of embodiments 40 to 59, wherein the subject has or is suspected of having cancer.

[0285] Embodiment 61: The method according to any one of embodiments 40 to 60, further comprising determining, based on the determined genotype, whether to perform further molecular testing to confirm the determined genotype.

[0286] Embodiment 62: The method of embodiment 61, wherein determining whether to perform further molecular testing comprises prioritizing subjects for undergoing further molecular testing in response to the determined genotype indicating a genotype change.

[0287] Embodiment 63: The method of embodiment 62, wherein determining whether to perform further molecular testing comprises excluding the subject from undergoing further molecular testing in response to the determined genotype indicating a wildtype genotype.

[0288] Embodiment 64: The method according to any one of embodiments 40 to 61, further comprising determining whether to enroll the subject in a clinical trial at least according to the determined genotype.

[0289] Embodiment 65: The method of embodiment 61, wherein determining whether to enroll the subject comprises determining that the determined genotype comprises a genotypic variation, and determining that the subject is eligible for enrollment in the clinical trial based at least on a determination that the genotype comprises a genotypic variation.

[0290] Embodiment 66: The method of embodiment 65, wherein determining that the subject is eligible to enroll in the clinical trial is further based on a molecular test confirming that the subject exhibits a genotype that includes a genotypic alteration.

[0291] Embodiment 67: The method of embodiment 66, wherein determining whether to enroll the subject comprises determining that the determined genotype does not comprise a genotypic variation, and determining that the subject is ineligible for enrollment in the clinical trial based at least on a determination that the genotype does not comprise a genotypic variation.

[0292] Embodiment 68: The method of embodiment 67, wherein the subject does not undergo further molecular testing based on a determination that the determined genotype does not contain a genotypic alteration.

[0293] 69. The method of any one of embodiments 40-61, further comprising determining whether to administer a treatment regimen according to at least the determined genotype.

[0294] Embodiment 70: The method of embodiment 69, wherein the therapeutic agent is an FGFR kinase inhibitor.

[0295] Embodiment 71: The method of embodiment 70, wherein the FGFR kinase inhibitor is erdafitinib (BALVERSA).

[0296] Embodiment 72: The method of embodiment 69, wherein the therapeutic agent is a PARP inhibitor.

[0297] Embodiment 73: The method of embodiment 69, wherein the PARP inhibitor is niraparib (ZEJULA).

[0298] Embodiment 74: The method described in any one of embodiments 40 to 44 and 51 to 63, wherein the neural network model exhibits an auROC performance metric of at least 0.82 for the bladder cancer model.

[0299] Embodiment 75: The method of any one of embodiments 40 to 44 and 51 to 63, wherein the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.22 with 100% recall against a bladder cancer model in a test dataset having a baseline prevalence of 14% for FGFR.

[0300] Embodiment 76: The method described in any one of embodiments 45 to 73, wherein the neural network model exhibits an auROC performance metric of at least 0.71 for the prostate cancer model.

[0301] Embodiment 77: The method according to any one of embodiments 45 to 73, wherein the neural network model exhibits a positive predictive value (PPV) performance metric of at least 0.14 with 100% recall against a prostate cancer model in a test dataset having a baseline prevalence of 11% for PARP.

[0302] Embodiment 78: The method of any one of embodiments 40 to 77, further comprising reporting one or more tiles of the image that are most strongly associated with the subject's genotype.

[0303] Embodiment 79: A method comprising: acquiring or having acquired training data including training histopathology images; and training a neural network model by analyzing a plurality of tiles of the training histopathology images, wherein the neural network model is configured to generate informative image-level predictions for determining a genotype corresponding to the training histopathology images.

[0304] Embodiment 80: The method of embodiment 79, wherein the neural network model includes a convolutional neural network sub-model and an attention network sub-model.

[0305] Embodiment 81: The method described in embodiment 80, wherein the attention network sub-model is trained using training data including training histopathology images.

[0306] Embodiment 82: The method described in embodiment 81, wherein the convolutional neural network sub-model and the attention network sub-model are trained separately.

[0307] Embodiment 83: The method described in embodiment 81, wherein the convolutional neural network sub-model and the attention network sub-model are jointly trained using training data.

[0308] Embodiment 84: The method of any one of embodiments 79 to 83, wherein the training data further includes reference ground truth labels indicating the genotypes of the training histopathology images.

[0309] Embodiment 85: The method of any one of embodiments 79 to 84, wherein the training histopathology images are obtained from a publicly available database.

[0310] Embodiment 86: The method of embodiment 85, wherein the publicly available database is The Cancer Genome Atlas (TCGA) database.

[0311] Embodiment 87: The method of any one of embodiments 79 to 84, wherein the training histopathology images are obtained from a proprietary database.

[0312] Embodiment 88: A method according to any one of embodiments 80 to 87, wherein the convolutional neural network sub-model generates tile-level predictions for one or more tiles of the preprocessed image.

[0313] Embodiment 89: A method according to any one of embodiments 80 to 88, wherein the attention network sub-model receives the tile-level predictions of the neural network sub-model and generates image-level predictions.

[0314] Embodiment 90: A method according to any one of embodiments 80 to 89, wherein the convolutional neural network sub-model and the attention network sub-model each comprise one or more layers of nodes.

[0315] Embodiment 91: A method according to any one of embodiments 80 to 90, wherein the convolutional neural network sub-model and the attention network sub-model each comprise one or more learned parameters.

[0316] Embodiment 92: A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations including steps in the methods described in any one of embodiments 1 to 91.

[0317] Although the present disclosure has been particularly described with respect to the illustrated embodiments, it will be understood that various changes, modifications, and adaptations may be made based on the present disclosure and are intended to be within the scope of the present disclosure. Although the present disclosure has been described with respect to what are presently considered to be the most practical and preferred embodiments, it is to be understood that the present disclosure is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements that are within the scope of the basic principles of the invention as described by the various embodiments set forth above and referenced below.

Claims

1. 1. A method of generating, via a deep learning pipeline, a deep learning network configured to run on one or more computers for generating predictions related to likelihood of therapeutic response to a new treatment in members of a cohort of interest using histopathology image data from histopathology images corresponding to members of the cohort of interest, wherein the cohort of interest includes candidates for receiving the new treatment in a clinical trial, the method comprising: training a plurality of respective deep learning networks in succession, from a first deep learning network to a last deep learning network, using respective histopathology image datasets having respective degrees of relevance to the cohort of interest; and sequentially transferring learned parameters of one deep learning network of the plurality of respective deep learning networks to another deep learning network of the plurality of respective deep learning networks after training the one deep learning network with one of the respective histopathology image datasets and before training the other deep learning network with another of the respective histopathology image datasets.

2. 2. The method of claim 1, wherein the respective relevance to the cohort of interest increases from the first to the last respective histopathology image dataset used in training the respective deep learning network.

3. The method of claim 2 , wherein the first histopathology image dataset is significantly larger than the last histopathology image dataset.

4. The method of claim 1 , wherein a first one of the respective histopathology image datasets comprises unlabeled histopathology image data.

5. The method of claim 4 , wherein the first deep learning network comprises a feature extraction network and a contrastive learning module.

6. 6. The method of claim 5, wherein the first deep learning network further comprises a projection network configured to receive feature vectors from the feature extraction network and provide feature vectors to the control learning module.

7. 7. The method of claim 1, wherein each deep learning network of the plurality of respective deep learning networks from a second deep learning network to the final deep learning network comprises a feature extraction network and a classification network, and further wherein the second deep learning network to the final deep learning network are trained using supervised learning.

8. 8. The method of claim 7, wherein each of the deep learning networks from the second deep learning network to the last deep learning network further comprises an attention network.

9. 10. The method of claim 8, further comprising, for each of the second through last deep learning networks, combining an output of the feature extraction layer and an output of the attention network and providing a combined output to a classification network.

10. 9. The method of claim 8, wherein the attention network comprises one or more fully connected layers configured to generate an attention value for each feature vector.

11. 11. The method of claim 10, wherein for each feature vector obtained from data corresponding to a particular histopathology image, the method multiplies the feature vector by a corresponding attention value and combines the results to produce a summarized feature vector that summarizes all feature vectors obtained from the data corresponding to the particular histopathology image.

12. The method of claim 10 , wherein the summarized feature vector is obtained by averaging the results of multiplying each feature vector by the corresponding attention value.

13. The method of claim 12 , wherein the summarized feature vector is submitted to a classification network.

14. 14. The method of claim 13, wherein the results from the classification network and the labels corresponding to the current histopathology image dataset are used to calculate an error based on a loss function, and the error is used to adjust weights in a currently trained deep learning network.

15. 7. The method of any one of claims 1 to 6, wherein said prediction of likely therapeutic response comprises a prediction of the presence or absence of genotypic alterations corresponding to tumor tissue in said histopathology image.

16. 16. The method of claim 15, wherein the genotypic alteration comprises a fibroblast growth factor receptor (FGFR) alteration.

17. 17. The method of claim 16, wherein the FGFR alteration comprises either an FGFR3 mutation, an FGFR3 fusion, or a combination of an FGFR3 mutation and an FGFR3 fusion.

18. 17. The method of claim 16, wherein the FGFR alteration is any of a p. R248C mutation, a p. G370C mutation, a p. S249C mutation, or a p. Y373C mutation.

19. 17. The method of claim 16, wherein the FGFR alteration is any of a FGFR3:TACC3V1 fusion, a FGFR3:TACC3V3 fusion, a FGFR3:BAIAP2L1 fusion, a FGFR2:BICC1 fusion, or a FGFR2:CASP7 fusion.

20. 17. The method of claim 16, wherein the tumor tissue comprises bladder cancer.

21. 16. The method of claim 15, wherein the genotypic alteration comprises one or more of a single nucleotide polymorphism (SNP), copy number variation (CNV), gene fusion, or DNA repair deficiency (DRD) involving BRCA1, BRCA2, BRIP1, CDK12, CHEK2, FANCA, PALB2, RAD51B, RAD54L, RAD21, or SPOP.

22. 16. The method of claim 15, wherein the tumor tissue comprises prostate cancer.

23. The method according to any one of claims 1 to 6, wherein the histopathological image is a hematoxylin and eosin (H&E) stained histopathological image.

24. The method of claims 1-6, wherein said members of said cohort of subjects have or are suspected of having cancer.

25. 16. The method of claim 15, further comprising determining, based on the determined genotype, whether to perform further molecular testing to confirm the determined genotype.

26. 26. The method of claim 25, wherein determining whether to perform further molecular testing comprises prioritizing the members of the subject cohort for undergoing further molecular testing in response to the determined genotype indicating a genotypic variation.

27. 27. The method of claim 26, wherein determining whether to perform further molecular testing comprises excluding the member of the subject cohort from undergoing further molecular testing in response to the determined genotype indicating a wild-type genotype.

28. 16. The method of claim 15, further comprising determining whether to enroll said member of said target cohort in a clinical trial according to at least said determined genotype.

29. determining whether to enroll the member of the target cohort, determining that the determined genotype comprises a genotypic variation; and determining that the member of the subject cohort is eligible for enrollment in the clinical trial based at least on the determination that the genotype comprises a genotypic variation.

30. 30. The method of claim 29, wherein determining that the member of the subject cohort is eligible for enrollment in the clinical trial is further based on molecular testing confirming that the member of the subject cohort exhibits a genotype that includes the genotypic variation.

31. 31. The method of claim 30, wherein determining whether to enroll the member of the target cohort comprises determining that the determined genotype does not comprise a genotypic variation, and determining that the member of the target cohort is ineligible for enrollment in the clinical trial based at least on the determination that the genotype does not comprise a genotypic variation.

32. 32. The method of claim 31 , wherein based on the determination that the determined genotype does not comprise the genotypic variation, the member of the subject cohort does not undergo further molecular testing.

33. 16. The method of claim 15, further comprising determining whether to administer a therapeutic agent according to at least the determined genotype.

34. 34. The method of claim 33, wherein the therapeutic agent is an FGFR kinase inhibitor.

35. 35. The method of claim 34, wherein the FGFR kinase inhibitor is erdafitinib.

36. 34. The method of claim 33, wherein the therapeutic agent is a PARP inhibitor.

37. 37. The method of claim 36, wherein the PARP inhibitor is niraparib.

38. 34. The method of claim 33, wherein the therapeutic agent is a monoclonal antibody.

39. 39. The method of claim 38, wherein the monoclonal antibody is amivantamab.

40. The method of claims 1-6, wherein the deep learning network exhibits an auROC performance metric of at least 0.82 on bladder cancer images.

41. 7. The method of claims 1-6, wherein the deep learning network exhibits a positive predictive value (PPV) performance metric of at least 0.22 with 100% recall for bladder cancer images in a test dataset having a baseline prevalence of 14% for FGFR.

42. The method of claims 1-6, wherein the deep learning network exhibits an auROC performance metric of at least 0.71 on prostate cancer images.

43. 7. The method of claims 1-6, wherein the deep learning network exhibits a positive predictive value (PPV) performance metric of at least 0.14 with 100% recall for prostate cancer images in a test dataset having a baseline prevalence of 11% for PARP.

44. 7. The method of claims 1-6, wherein the deep learning network exhibits an auROC performance metric of at least 0.78 + / - 0.03 for predicting MET changes on non-small cell lung cancer images.

45. The method of claim 15 further comprising reporting one or more tiles of the image that are most strongly associated with the genotype of the genotypic variation.

46. A computer program product stored on a non-transitory computer readable medium comprising instructions configured to carry out the method of claims 1 to 6 using one or more computer processors.

47. A computerized pipeline system comprising a series of consecutive computerized deep learning networks configured to execute processes for carrying out the methods of claims 1 to 6.

48. 1. A system comprising: a deep learning pipeline comprising one or more computers coupled to a non-transitory computer-readable medium storing instructions, the instructions being executable by one or more processors of the one or more computers to sequentially train a plurality of respective deep learning networks using respective histopathology image datasets, each respective histopathology image dataset having a respective degree of relevance to a cohort of interest, the training comprising: training a first deep learning network of the plurality of deep learning networks with one histopathology image dataset of the respective histopathology image datasets; sequentially transferring a plurality of learned parameters of the first deep learning network to a second deep learning network of the plurality of respective deep learning networks; training the second deep learning network on another of the respective histopathology image datasets.

49. 16. The method of claim 15, wherein the genotypic variation comprises a variation in the MET gene.

50. 16. The method of claim 15, wherein the tumor tissue comprises lung cancer.

Images