Machine learning techniques for predicting phenotypes in dual digital pathology images

Abstract

Dual immunohistochemistry (IHC) staining of tissue sections allows for simultaneous detection of two biomarkers and their co-expression at the single cell level, without the need for two IHC stains and further alignment to identify co-localization. Dual IHC is often difficult for humans, including pathologists, to reliably score. The methods and systems described here use machine learning models and probability maps to detect and score individual phenotypes ER / PR.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 261,308, filed September 17, 2021, which is incorporated herein by reference in its entirety for all purposes.

[0002] Field The present disclosure relates to digital pathology, and in particular to techniques for predicting two or more phenotypes in cells displayed in dual digital pathology images using machine learning techniques. [Background technology]

[0003] background Digital pathology involves scanning pathology slides (e.g., histopathology or cytopathology glass slides) into digital images. The tissues and / or cells in the digital images may then be examined by digital pathology image analysis and / or interpreted by a pathologist for a variety of reasons, including diagnosing disease, assessing response to treatment, and developing pharmacological agents to combat disease. To examine the tissues and / or cells in the digital images (which are substantially transparent), pathology slides may be prepared using various staining assays (e.g., immunostaining) that selectively bind to tissue and / or cellular components.

[0004] Immunohistochemistry (IHC) staining is used to detect the presence of specific proteins in specimens fixed on slides. Estrogen receptor (ER) and progesterone receptor (PR) proteins are important biomarkers that drive clinical management and treatment decisions for breast cancer. Single or singleplex IHC staining involves staining a single slide for a single biomarker. Single IHC staining requires the use of multiple slides to look for colocalization of biomarkers. To look for colocalization, images from each slide must be registered or mapped to one another and then sometimes overlaid to create a single image showing both biomarkers.

[0005] Dual immunohistochemistry (IHC) staining of tissue sections allows for simultaneous detection of two biomarkers on a single slide. Therefore, instead of preparing two slides, each with a single IHC stain and then further alignment to identify colocalization, dual IHC staining allows for analysis of biomarker coexpression at the single-cell level. However, dual IHC slides face certain challenges. For example, dual IHC slides are often difficult for humans (e.g., pathologists) to reliably score. Pathologists often cannot directly score dual IHC slides (e.g., ER / PR) due to overlapping and clustered biomarkers. Furthermore, because each slide may contain thousands of cells, the scoring process can be time-consuming. Therefore, accurately identifying and counting expression and colocalization levels is practically impossible. Summary of the Invention

[0006] overview In various embodiments, a computer-implemented method for predicting two or more phenotypes in cells displayed in a dual digital pathology image is provided. The computer-implemented method includes accessing digital pathology images displaying at least a portion of a biological sample stained for a first type of biomarker and a second type of biomarker. The computer-implemented method includes unblending the digital pathology images to generate (i) a first composite single image displaying at least a portion of the biological sample in which the first type of biomarker was identified, and (ii) a second composite single image displaying at least a portion of the biological sample in which the second type of biomarker was identified. The computer-implemented method further includes applying a first machine learning model to the first composite single image to (i) detect a first plurality of cells from the first composite single image and, for each cell of the first plurality of cells, determine a classification from a first set of classifications. The classification from the first set indicates whether the cell contains biomarkers with the first type of biomarker.

[0007] The computer-implemented method further includes applying a second machine learning model to the second composite single image to (i) detect a second plurality of cells from the second composite single image and (ii) determine, for each cell of the second plurality of cells, a classification among a second set of classifications. The classification among the second set indicates whether the cell contains a biomarker having a second type of biomarker. In some cases, the first set of classifications differs from the second set of classifications. The computer-implemented method further includes merging the classifications of the first plurality of cells and the classifications of the second plurality of cells to generate a merged classification. The computer-implemented method further includes outputting a digital pathology image having the merged classification.

[0008] In some embodiments, determining a classification for the first plurality of cells includes generating a first set of probability maps. Each probability map in the first set of probability maps includes a plurality of pixels and is associated with a classification from the first set of classifications. Each probability map in the first set of probability maps also identifies, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to that classification. Determining the classification for the first plurality of cells includes, for each cell of the first plurality of cells, (i) identifying a probability map from the first set of probability maps that includes a highest probability value for one or more pixels representing the cell, and (ii) assigning the cell a classification associated with the identified probability map.

[0009] In some embodiments, determining a classification for the second plurality of cells includes generating a second set of probability maps. Each probability map in the second set of probability maps includes a plurality of pixels and is associated with a classification from the second set of classifications. Each probability map in the second set of probability maps also identifies, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to that classification. Determining a classification for the second plurality of cells includes, for each cell of the second plurality of cells, (i) identifying a probability map from the second set of probability maps that includes a highest probability value for one or more pixels representing the cell, and (ii) assigning the cell a classification associated with the identified probability map.

[0010] In some embodiments, the first machine learning model and / or the second machine learning model include a U-Net model.

[0011] In some embodiments, the first type of biomarker is an estrogen receptor protein and the second type of biomarker is a progesterone receptor protein.

[0012] In some embodiments, outputting the digital pathology image with the merged classification comprises overlaying the merged classification onto the digital pathology image.

[0013] In some embodiments, the digital pathology images with the merged classifications are used as training images for training a third machine learning model.

[0014] In some embodiments, determining a classification for the first plurality of cells includes generating a first set of probability maps, each probability map of the first set of probability maps including a plurality of pixels and associated with a classification from the first set of classifications, the probability map identifying, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to that classification.

[0015] In some embodiments, determining a classification of the second plurality of cells includes generating a second set of probability maps, each probability map of the second set of probability maps including a plurality of pixels and associated with a classification from the second set of classifications, the probability map identifying, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to that classification.

[0016] In some embodiments, the first set of probability maps and the second set of probability maps are merged to generate a set of anchor points, each anchor point of the set of anchor points being assigned a first classification of the first set of classifications and a second classification of the second set of classifications.

[0017] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0018] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0019] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0020] The terms and expressions employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be employed by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS Aspects and features of various embodiments will become more apparent by way of example only and with reference to the accompanying drawings. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram illustrating an image processing system 100 that uses machine learning techniques to merge detected phenotypes in a composite single image, according to some embodiments. [Figure 2] FIG. 1 illustrates an exemplary network for generating digital pathology images, according to some embodiments. [Figure 3] FIG. 1 is a block diagram illustrating a computing environment for processing digital pathology images using machine learning models, according to some embodiments. [Figure 4] FIG. 1 shows an example of double ER / PR and adjacent single IHC ER / PR, according to some embodiments. [Figure 5] FIG. 10 shows a field of view (FOV) selected from a dual ER / PR image and registered onto a single IHC ER image and a single IHC PR image, according to some embodiments. [Figure 6A] FIG. 1 illustrates an exemplary technique for generating a composite single image, according to some embodiments. [Figure 6B] FIG. 1 illustrates an exemplary technique for generating a composite single image, according to some embodiments. [Figure 6C] FIG. 1 illustrates an exemplary technique for generating a composite single image, according to some embodiments. [Figure 7] FIG. 10 illustrates exemplary initial seed positions for dabsyl ER and tamra PR obtained by a pre-trained U-Net model, according to some embodiments. [Figure 8] 1 is an exemplary architecture of a machine learning model used to detect phenotypes in dual slide images, according to some embodiments. [Figure 9A] FIG. 2 illustrates a residual block according to some embodiments. [Figure 9B] FIG. 1 illustrates pyramid layers according to some embodiments. [Figure 10]FIG. 1 illustrates a process for training a machine learning model to detect phenotypes in dual images, according to some embodiments. [Figure 11] FIG. 10 illustrates exemplary seed positions and different class labels produced by two models, according to some embodiments. [Figure 12] FIG. 1 illustrates five probability maps generated by an ER model, according to some embodiments. [Figure 13] FIG. 10 illustrates the computation of five merged probability maps between the probability maps generated by two U-Net models in the ER and PR channels, in accordance with some embodiments. [Figure 14] FIG. 10 illustrates an example of anchor points obtained by searching merged probability maps between ER and PR channels, according to some embodiments. [Figure 15] FIG. 10 illustrates an exemplary anchor point and surrounding ER+ and PR+ points at the pixel level, according to some embodiments. [Figure 16] FIG. 10 illustrates an exemplary label graph determined from a probability map, according to some embodiments. [Figure 17] FIG. 10 illustrates an example of assigning anchor points to ER+ / ER- / other, according to some embodiments. [Figure 18] FIG. 1 illustrates a process for assigning anchor points to ER+ / ER- / etc. in a composite ER image, according to some embodiments. [Figure 19] FIG. 10 illustrates an example of assigning anchor points to PR+ / PR- / etc., according to some embodiments. [Figure 20] FIG. 1 illustrates a process for assigning anchor points to PR+ / PR- / etc. in a composite PR image, according to some embodiments. [Figure 21] FIG. 10 shows an example of merged phenotypes overlaid on a double image, according to some embodiments. [Figure 22]FIG. 1 illustrates a process for detecting phenotypes in dual images using a trained machine learning model, according to some embodiments. [Figure 23] FIG. 1 illustrates a training pipeline using merged phenotypes for the dual ER / PR algorithm, according to some embodiments. [Figure 24A] FIG. 1 illustrates the consensus scores of three pathologists and the U-Net and merge phenotype algorithms, according to some embodiments. [Figure 24B] FIG. 1 illustrates the consensus scores of three pathologists and the U-Net and merge phenotype algorithms, according to some embodiments. [Figure 25] FIG. 1 shows an example of phenotypic detection of dual ER / PR results, according to some embodiments. [Figure 26A] FIG. 1 shows an example of dual ER / PR with different types of phenotypes, according to some embodiments. [Figure 26B] FIG. 1 shows an example of dual ER / PR with different types of phenotypes, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0023] Detailed Description Existing techniques generally involve detecting cells in digital pathology images using cell detection algorithms and then applying machine learning models to the detected cells to predict whether a particular type of biomarker is present in one or more of the detected cells. The performance of these machine learning models typically depends on the accuracy of the training dataset. Generating the training dataset can include labeling cells that appear in one or more training images of the training dataset, where the labels may be assigned by one or more pathologists. The labeling process can be challenging for dual images stained to identify two or more types of biomarkers. This is because two or more stains may form unrecognizable clusters in different regions of the dual image and / or may be present in the same image region within the dual image. Such staining characteristics can lead to confusion regarding how to properly label cellular regions. Such difficulties can lead to training images being associated with inaccurate or inconsistent training labels, which further leads to poor performance of machine learning models trained on such training images to predict the presence of two or more biomarkers in cells that appear in other dual images.

[0024] Therefore, certain embodiments described herein can address these and other problems by accurately predicting cellular phenotypes associated with two or more types of biomarkers in cells displayed in a dual image. An image processing system can access dual slide images. The dual image displays at least a portion of a biological sample, such as a section of breast tissue. At least a portion of the biological sample can be stained to identify two or more types of biomarkers. For example, a first type of biomarker can be an estrogen receptor (ER) and a second type of biomarker can be a progesterone receptor (PR).

[0025] The image processing system can process the dual images to generate a set of composite single images. Each composite single image in the set of composite single images can be generated to display cells stained for a single biomarker (e.g., ER). In some examples, the composite single images are generated by (i) generating a pre-processed image displaying cells stained for the corresponding biomarker, and (ii) combining the pre-processed image with a counterstained image. The counterstained image can be used to visually distinguish the biomarker from other cellular structures displayed in the pre-processed image.

[0026] For each synthetic single image in the set of synthetic single images, the image processing system can apply a machine learning model to the synthetic single image to predict a phenotype for each detected cell displayed in the synthetic single image, where the phenotype is associated with a corresponding type of biomarker. In some examples, a machine learning model is trained to process a first synthetic single image displaying cells stained for a first type of biomarker, and a different machine learning model is trained to process a second synthetic single image stained for a second type of biomarker. For example, the image processing system can apply a first trained U-Net model to a synthetic single image stained for an ER biomarker to (i) detect cells in the synthetic single image and (ii) predict, for each detected cell, a cellular phenotype for the ER biomarker (e.g., ER positive, ER negative, artifact). Continuing with this example, the image processing system can apply a second trained U-Net model to another synthetic single image stained for a PR biomarker to (i) detect cells in the other synthetic single image and (ii) predict, for each detected cell, a cellular phenotype for the PR biomarker (e.g., PR positive, PR negative, artifact).

[0027] In some cases, a machine learning model is trained to determine a classification from a set of classifications for each cell in the synthetic single image. Each classification in the set can correspond to a particular phenotype of the cell for a corresponding type of biomarker. For example, the machine learning model can output whether the detected cell (i) contains an ER biomarker (ER+), (ii) does not contain an ER biomarker (ER-), (iii) corresponds to a stromal cell, (iv) corresponds to an immune cell, or (v) corresponds to an artifact or other type of biological structure. To generate the classification, the image processing system can process the synthetic single image using the machine learning model to generate a set of probability maps. Each probability map in the set of probability maps can represent multiple pixels of the synthetic single image and can correspond to a particular classification in the set of classifications. The probability map includes, for each pixel of the multiple pixels, a probability value indicating whether the pixel corresponds to that classification. Continuing with this example, the set of probability maps for the synthetic single image can include a first probability map for ER+, a second probability map for ER-, a third probability map for stromal cells, a fourth probability map for immune cells, and a fifth probability map for artifacts.

[0028] A set of probability maps representing a composite single image can be merged with another set of probability maps representing another composite single image to generate a merged set of probability maps. The merged set of probability maps can be used to determine a set of locations ("anchor points") within the dual image, and the presence of one or more biomarkers can be identified for each anchor point. Continuing with the above example, a first set of probability maps can represent a composite single image stained for an ER biomarker, the first set of probability maps including: (i) a first probability map for an ER+ classification; (ii) a second probability map for an ER- classification; (iii) a third probability map for a stromal cell classification; (iv) a fourth probability map for an immune cell classification; and (v) a fifth probability map for an artifact classification. Additionally, a second set of probability maps can represent a composite single image stained for PR biomarkers, the second set of probability maps including: (i) a first probability map for PR+ classification; (ii) a second probability map for PR- classification; (iii) a third probability map for stromal cell classification; (iv) a fourth probability map for immune cell classification; and (v) a fifth probability map for artifact classification.

[0029] The image processing system can generate a first probability map of the merged set by comparing, for each pixel, the probability value of the ER+ probability map of the first set of probability maps with another probability value of the PR+ probability map of the second set of probability maps. Then, based on the comparison, a higher probability value and its corresponding classification (e.g., ER+) can be assigned to each pixel of the first probability map of the merged set (e.g., ER+ / PR+). The comparing and assigning steps can be repeated for other pixels to generate the first probability map of the merged set. The above steps can also be repeated to generate other probability maps of the merged set (e.g., ER- / PR-, ER+ / PR-, ER- / PR+, etc.).

[0030] The image processing system can use the merged set of probability maps to identify a set of anchor points for the double image. Each anchor point in the set of anchor points can correspond to a region (e.g., a cell center) in the double image that may be predicted to have an individual biomarker or multiple biomarkers. To determine the anchor points, the image processing system can select a region of the double image that includes a set of pixels. The image processing system can obtain a set of probability values ​​for that region from each probability map in the merged set, each probability value indicating whether the corresponding pixel in that region corresponds to the classification associated with the probability map. From the set of pixels across the merged set of probability maps, the image processing system can select the pixel with the highest probability value and assign the selected pixel as the anchor point.

[0031] Each anchor point in the set of anchor points can be labeled with one or more corresponding classifications that predict whether the image region represented by the anchor point indicates the presence of an individual marker or multiple biomarkers. For example, an anchor point can be labeled with an ER+ classification based on the anchor point being within a predetermined distance (e.g., 10 pixels) from a region of the ER+ probability map predicted to contain an ER biomarker. Continuing this example, the same anchor point can also be labeled with a PR+ classification based on the anchor point being within a predetermined distance (e.g., 10 pixels) from a region of the PR+ probability map predicted to contain a PR biomarker. As a result, the region of the dual image corresponding to the anchor point can be labeled as having a phenotype indicative of the presence of two types of biomarkers (e.g., ER+ / PR+). Incorporating the probability values ​​of the probability map into the anchor points in the dual image allows the dual image to include one or more regions that identify the phenotype of a cell for multiple biomarkers. Anchor points with merged phenotypes can be overlaid on the dual image, thereby accurately displaying the phenotypes corresponding to multiple biomarkers for a given image region of the dual image.

[0032] Certain embodiments described herein improve the performance of machine learning models that identify colocalization or coexpression of biomarkers in dual images. The image processing system can improve performance by generating a composite single image from the dual images, applying separate machine learning models to the composite single image, and merging the probability maps generated by the machine learning models to output multiple classifications for each cell displayed in the dual image. The image processing system can perform accurately even when multiple types of biomarkers are mixed and clustered within the same dual image. Thus, embodiments herein reflect improved capabilities of artificial intelligence systems and digital pathology image processing technology.

[0033] While specific embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of protection. The apparatus, methods, and systems described herein may be embodied in various other forms. Furthermore, various omissions, substitutions, and changes in form of the exemplary methods and systems described herein may be made without departing from the scope of protection.

[0034] I. Definition As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a part of the something.

[0035] As used herein, the terms "substantially," "approximately," and "about" are defined as being largely, but not necessarily entirely, specified (and including) what is specified entirely, as would be understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" may be replaced with "within [percentage]" of what is specified, where percentages include 0.1, 1, 5, and 10%.

[0036] As used herein, the terms "sample," "biological sample," "tissue," or "tissue sample" refer to any sample containing biomolecules (e.g., proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (e.g., veterinary animals such as humans, cats, dogs, horses, cows, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and needle biopsies of tissue), cell samples (such as cytological smears, such as Pap smears or blood smears, or samples of cells obtained by microdissection), or cell fractions, fragments, or organelles (such as those obtained by lysing cells and separating their components, such as by centrifugation). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swabs (such as oral swabs), or any material containing biomolecules derived from an initial biological sample. In some embodiments, the term "biological sample" as used herein refers to a sample (such as a homogenized or liquefied sample) prepared from a tumor or portion thereof obtained from a subject.

[0037] As used herein, the terms "biological material," "biological structure," or "cellular structure" refer to naturally occurring materials or structures that comprise all or part of a biological structure (e.g., a cell nucleus, cell membrane, cytoplasm, chromosomes, DNA, cell, cell mass, etc.).

[0038] As used herein, the term "biomarker" refers to a biological molecule (e.g., a protein molecule) found in blood, other body fluids, or tissue that is indicative of a normal or abnormal process, or a condition or disease. A biomarker can be associated with a specific type of biomarker. For example, the biomarker can be an estrogen receptor (ER) or a progesterone receptor (PR).

[0039] As used herein, "digital pathology image" refers to a digital image of a stained sample.

[0040] As used herein, "phenotype" refers to a cell type that expresses a characteristic associated with a corresponding biomarker in a digital pathology image. For example, estrogen receptor phenotypes in a region of a double image can identify the presence of estrogen receptors (ER+), while progesterone receptor phenotypes in the same region can identify the absence of progesterone receptors (PR-). Phenotypes can be merged to define characteristics of multiple biomarkers (e.g., ER+ / PR+).

[0041] As used herein, a "composite single image" refers to a digital pathology image generated from a double image, where the composite single image is generated by combining (a) a single-stain image derived from the double image and (ii) a counterstain image. The composite single image can identify a specific phenotype.

[0042] As used herein, "dual image" refers to a digital pathology image stained to identify two or more biomarkers (e.g., ER, PR).

[0043] As used herein, "anchor point" refers to a region (e.g., a cell center) of a digital pathology image (e.g., a dual image) that may be expected to have an individual biomarker or multiple biomarkers.

[0044] II. Overview Deep learning-based systems and methods can be used to generate a composite single image from dual images, identify cellular phenotypes detected from each of the dual images, and merge the identified phenotypes to determine image regions containing two or more biomarkers. Figure 1 shows a schematic diagram of an image processing system 100 that uses machine learning techniques to merge phenotypes detected in a composite single image, according to some embodiments. The image processing system can be configured to receive images of a pathology slide 102 stained to indicate two or more types of biomarkers (block 104). In some embodiments, the image processing system is configured to operate using images of dual slides stained to indicate the presence of estrogen receptor (ER) and progesterone receptor (PR) proteins. Each cell in the image can be classified as positive or negative for each of the ER and PR markers. Thus, the phenotype of each cell can be identified as ER+PR+, ER+PR-, ER-PR+, ER-PR-, or other (e.g., stromal, immune, necrotic, artifact, etc.)

[0045] In some cases, the image processing system divides the dual slide image into multiple image tiles (block 106). Identification of cellular phenotypes for the multiple biomarkers can be performed for each of the multiple image tiles. In some cases, a set of image tiles is selected from the multiple image tiles, and the set of image tiles is used as training images for training one or more machine learning models to detect cellular phenotypes for the types of biomarkers.

[0046] The image processing system can unmix the double images to generate a set of composite single images (blocks 108 and 110). Each composite single image in the set of composite single images can be generated to display cells stained for a single biomarker (e.g., ER). In some examples, the composite single images are generated by (i) generating a pre-processed image displaying cells stained for the corresponding biomarker, and (ii) combining the pre-processed image with a counterstained image.

[0047] In some embodiments, the image processing system trains a machine learning model (e.g., a U-Net model) using the set of image tiles to predict the phenotype of each detected cell in the composite single image. For example, a first machine learning model can be trained using the set of image tiles to predict the cellular phenotype of an ER biomarker (block 112), and a second machine learning model can be trained using the set of image tiles to predict the cellular phenotype of a PR biomarker (block 114).

[0048] The trained machine learning model can then be used to detect the phenotype of cells in the corresponding composite single image (blocks 116 and 118). For example, the image processing system can use two U-Net models to generate multiple probability maps (e.g., 10 probability maps), and the multiple probability maps can be used to detect ER and PR cellular phenotypes. For example, five of the 10 probability maps can represent composite single images stained for ER biomarkers, including (i) a first probability map for ER+ classification, (ii) a second probability map for ER- classification, (iii) a third probability map for stromal cell classification, (iv) a fourth probability map for immune cell classification, and (v) a fifth probability map for artifact classification. The remaining five probability maps can represent composite single images stained for PR biomarkers.

[0049] The image processing system can then merge the cellular phenotypes predicted by the two machine learning models (block 120). In some cases, a logical "AND" is used to directly calculate which phenotype each cell belongs to. Probabilistic priorities can be used to merge all different co-expressions of biomarkers and identify cellular phenotypes associated with the identification of multiple types of biomarkers. In some embodiments, the image processing system determines anchor points to be the cell centers of each nucleus (including ER+ / PR+ / ER- / PR- / other). The anchor points can be obtained by comparing ten probability maps generated by two machine learning models (e.g., an ER model and a PR model) trained by two deep learning networks. The system can then match each anchor point with the ER+ / ER- / other classification generated by the first machine learning model (the "ER model"). The system can also match each anchor point with the PR+ / PR- / other classification generated by the second machine learning model (the "PR model"). In some cases, a data structure can be used to record all locations of phenotypes / biomarkers ER / PR and associate them with a map that records anchor points / nuclear cell centers. Logical operators (e.g., "AND") and probabilistic priorities can be used to merge all distinct co-expressions. As a result, a merged phenotype representing multiple biomarkers can be identified for the duplicate slide image (block 122). In some cases, the image processing system overlays the merged phenotype on the duplicate image, as shown in image 124. Using two machine learning models and the described mapping method significantly reduces computational and detection time when compared to alternative methods.

[0050] III. Generation of digital pathology images Digital pathology involves the interpretation of digitized images to accurately diagnose a subject and guide treatment decisions. In a digital pathology solution, an image analysis workflow can be established to automatically detect or classify biological objects of interest, such as positive or negative tumor cells. An exemplary digital pathology solution workflow includes acquiring a tissue slide, scanning a preselected area or the entire tissue slide with a digital image scanner (e.g., a whole slide imaging (WSI) scanner) to acquire a digital image, performing image analysis on the digital image using one or more image analysis algorithms, and potentially detecting and quantifying each object of interest (e.g., counting or identifying object-specific or cumulative areas of each object of interest) based on the image analysis (e.g., quantitative or semi-quantitative scoring such as positive, negative, moderate, weak, etc.).

[0051] FIG. 2 illustrates an exemplary network 200 for generating digital pathology images. A fixation / embedding system 205 fixes and / or embeds tissue samples (e.g., samples containing at least a portion of at least one tumor) using a fixative (e.g., a liquid fixative such as formaldehyde solution) and / or an embedding substance (e.g., a histological wax such as paraffin wax and / or one or more resins such as styrene or polyethylene). Each sample may be fixed by exposing the sample to a fixative for a predetermined period of time (e.g., at least 3 hours) and then dehydrating the sample (e.g., via exposure to an ethanol solution and / or a clearing intermediate agent). The embedding substance can be infiltrated when the sample is in a liquid state (e.g., upon heating).

[0052] Fixation and / or embedding of samples is used to preserve samples and slow their degradation. In histology, fixation generally refers to an irreversible process using chemicals to preserve chemical composition, preserve natural sample structure, and protect cellular structures from degradation. Fixation may also harden cells or tissues for sectioning. Fixatives may enhance sample and cell preservation by cross-linking proteins. Fixatives may bind to and cross-link some proteins and denature others through dehydration, which hardens tissue and can inactivate enzymes that might otherwise degrade the sample. Fixatives may also kill bacteria.

[0053] Fixatives may be administered, for example, by perfusion and immersion of the prepared sample. Various fixatives may be used, including methanol, buprenorphine fixatives, and / or formaldehyde fixatives, such as neutral buffered formalin (NBF) or paraffin-formalin (paraformaldehyde-PFA). If the sample is a liquid (e.g., blood sample), the sample may be smeared onto a slide and allowed to dry before fixation. While the fixation process can help preserve the structure of the sample and cells for histological examination, fixation can mask tissue antigens, thereby reducing antigen detection. Therefore, fixation is generally considered a limiting factor in immunohistochemistry, as formalin can crosslink antigens and mask epitopes. In some instances, additional processes are performed to reverse the effects of crosslinking, including treating the fixed sample with citraconic anhydride (a reversible protein crosslinker) and heating.

[0054] Embedding can involve infiltrating a sample (e.g., a fixed tissue sample) with an appropriate histological wax, such as paraffin wax. Histological waxes can be insoluble in water or alcohol but soluble in paraffin solvents such as xylene. Therefore, it may be necessary to replace the water in the tissue with xylene. To do so, the sample can first be dehydrated by gradually replacing the water in the sample with alcohol. This can be achieved by passing the tissue through increasing concentrations of ethyl alcohol (e.g., from 0 to approximately 100%). After replacing the water with alcohol, the alcohol may be replaced with xylene, which is miscible with alcohol. Because histological waxes can be soluble in xylene, the molten wax can be filled with xylene, filling the spaces previously filled with water. The wax-filled sample can be cooled to form a hardened block, which can be clamped to a microtome, vibratome, or compressome for sectioning. In some cases, deviations from the exemplary procedure above can result in infiltration of the paraffin wax, inhibiting penetration of antibodies, chemicals, or other fixatives.

[0055] The tissue slicer 210 may then be used to section the fixed and / or embedded tissue sample (e.g., a tumor sample). Sectioning is the process of cutting thin slices (e.g., 2-5 μm thick) of a sample from a tissue block for mounting on a microscope slide for examination. Sectioning may be performed using a microtome, vibratome, or compressome. In some cases, tissue can be rapidly frozen in dry ice or isopentane and then cut with a cold knife in a refrigerated cabinet (e.g., a cryostat). Other types of coolants, such as liquid nitrogen, can be used to freeze tissue. Sections for use with brightfield and fluorescence microscopy are generally on the order of 2-10 μm thick. In some cases, sections can be embedded in epoxy or acrylic resin, which may allow for cutting thinner sections (e.g., <2 μm). These sections may then be mounted on one or more glass slides. A coverslip may be placed on top to protect the sample sections.

[0056] Because tissue sections and the cells therein are substantially transparent, slide preparation typically further includes staining the tissue sections (e.g., automated staining) to make relevant structures more visible. In some cases, the staining is performed manually. In some cases, the staining is performed semi-automatically or automatically using a staining system 215. The staining process involves exposing sections of the tissue sample or fixed liquid sample to one or more different stains (e.g., sequentially or simultaneously) to reveal different properties of the tissue.

[0057] For example, stains may be used to mark specific types of cells and / or flag specific types of nucleic acids and / or proteins to aid in microscopy. The staining process generally involves adding a dye or stain to a sample to confirm or quantify the presence of a particular compound, structure, molecule, or feature (e.g., subcellular feature). For example, stains can help identify or highlight specific biomarkers in tissue sections. In other examples, stains can be used to identify or highlight biological tissues (e.g., muscle fibers or connective tissue), cell populations (e.g., different blood cells), or organelles within individual cells.

[0058] One exemplary type of tissue stain is histochemical staining, which uses one or more chemical dyes (e.g., acid dyes, basic dyes, chromogens) to stain tissue structures. Histochemical stains may be used to reveal general aspects of tissue morphology and / or cellular microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to reveal lipid droplets, etc.). One example of a histochemical stain is H&E. Other examples of histochemical stains include trichrome stains (e.g., Masson's trichrome), periodic acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of histochemical stains (e.g., dyes) is typically about 500 kilodaltons (kD) or less, although some histochemical stains (e.g., Alcian blue, phosphomolybdic acid (PMA)) can have molecular weights of up to 2 or 3,000 kD. One example of a high-molecular-weight histochemical stain is α-amylase (about 55 kD), which may be used to reveal glycogen.

[0059] Another type of tissue staining is IHC, also known as "immunostaining," which uses a primary antibody that specifically binds to a target antigen of interest (also known as a biomarker). IHC can be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody first binds to the target antigen, and then a secondary antibody conjugated to a label (e.g., a chromophore or fluorophore) binds to the primary antibody. The molecular weight of IHC reagents is much higher than that of histochemical staining reagents, as antibodies have a molecular weight of approximately 150 kD or greater.

[0060] Various types of staining protocols may be used to perform staining. For example, an exemplary IHC staining protocol includes using a hydrophobic barrier around the sample (e.g., tissue section) to prevent leakage of reagents from the slide during incubation, treating the tissue section with reagents to block endogenous sources of nonspecific staining (e.g., enzymes, free aldehyde groups, immunoglobulins, and other unrelated molecules that may mimic specific staining), incubating the sample with a permeabilization buffer to promote penetration of antibodies and other staining reagents into the tissue, incubating the tissue section with a primary antibody at a specific temperature (e.g., room temperature, 6-8 °C) for a certain period of time (e.g., 1-24 hours), rinsing the sample using a wash buffer, and then incubating the sample (tissue section) with a secondary antibody at another specific temperature (e.g., room temperature) for another period of time, rinsing the sample again using a water buffer, incubating the rinsed sample with a chromogen (e.g., DAB: 3,3'-diaminobenzidine), and washing off the chromogen to stop the reaction. In some cases, a counterstain is then used to distinguish the entire "landscape" of the sample and serve as the primary color reference used to detect tissue targets. Counterstains can include, for example, hematoxylin (a blue-to-purple stain), methylene blue (a blue stain), toluidine blue (a stain that renders nuclei deep blue and polysaccharides pink to red), nuclear fast red (also known as Kern Echtrot dye, a red stain), methyl green (a green stain), and non-nucleogenic stains such as eosin (a pink stain). Those skilled in the art will recognize that other immunohistochemical staining techniques can be implemented to perform the staining.

[0061] In another example, an H&E staining protocol can be performed to stain tissue sections. The H&E staining protocol involves applying hematoxylin stain mixed with a metal salt or mordant to the sample. The sample can then be rinsed in a weak acid solution to remove excess stain (differentiation), followed by bluing in weak alkaline water. After application of hematoxylin, the sample can be counterstained with eosin. It will be understood that other H&E staining techniques can be performed.

[0062] In some embodiments, staining can be performed using various types of staining agents depending on the feature of interest. For example, DAB can be used on various tissue sections for IHC staining, producing a brown color that displays the feature of interest in the stained image. In another example, alkaline phosphatase (AP) can be used on skin tissue sections for IHC staining, as the DAB color can be masked by melanin pigments. Regarding primary staining techniques, applicable staining agents can include, for example, basophilic and eosinophilic stains, hematin and hematoxylin, silver nitrate, trichrome stains, etc. Acidic dyes can react with cationic or basic components in tissues or cells, such as proteins and other components in the cytoplasm. Basic dyes can react with anionic or acidic components in tissues or cells, such as nucleic acids. As mentioned above, one example of a staining system is H&E. Eosin can be a negatively charged pink acidic dye, and hematoxylin can be a purple or blue basic dye containing hematein and aluminum ions. Other examples of stains may include Periodic Acid-Schiff (PAS) stain, Masson's trichrome, Alcian blue, Van Gieson, reticulin stain, etc. In some embodiments, different types of stains may be used in combination.

[0063] The sections may then be mounted on corresponding slides, which the imaging system 220 can then scan or image to generate raw digital pathology images 225a-n. A microscope (e.g., an electron microscope or an optical microscope) can be used to magnify the stained sample. For example, an optical microscope may have a resolution of less than 1 μm, such as on the order of several hundred nanometers. To observe finer details in the nanometer or sub-nanometer range, an electron microscope may be used. An imaging device (combined with or separate from the microscope) images the magnified biological sample to obtain image data, such as multichannel images (e.g., multichannel fluorescence) with several (e.g., 10-16) channels. The imaging device may include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), optical elements (e.g., one or more lenses, a sensor-focusing lens group, a microscope objective, etc.), an imaging sensor (e.g., a charge-coupled device (CCD), a complementary metal-oxide semiconductor (CMOS) image sensor, etc.), photographic film, etc. In digital embodiments, the imaging device may include multiple lenses that cooperate to provide on-the-fly focusing. An image sensor, e.g., a CCD sensor, may capture digital images of the biological sample. In some embodiments, the imaging device is a bright-field imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. The imaging device may utilize invisible electromagnetic radiation (e.g., UV light) or other imaging techniques to capture images. For example, the imaging device may include a microscope and a camera configured to capture images magnified by the microscope. The image data received by the analysis system may be identical to and / or derived from the raw image data captured by the imaging device.

[0064] Images of the stained sections may then be stored in a storage device 225, such as a server. The images may be stored locally, remotely, and / or on a cloud server. Each image may be stored in association with a subject identifier and a date (e.g., the date the sample was collected and / or the date the image was captured). The images may also be transmitted to another system (e.g., a system associated with a pathologist, an automated or semi-automated image analysis system, or a machine learning training and deployment system, as described in further detail herein).

[0065] It will be understood that modifications to the process described with respect to network 200 are contemplated. For example, if the sample is a liquid sample, embedding and / or sectioning may be omitted from the process.

[0066] IV. Exemplary Systems for Digital Pathology Image Conversion 3 is a block diagram illustrating a computing environment 300 for processing digital pathology images using a machine learning model, according to some embodiments. As further described herein, processing the digital pathology images may include training a machine learning algorithm using the digital pathology images and / or converting some or all of the digital pathology images into one or more results using a trained (or partially trained) version of the machine learning algorithm (i.e., a machine learning model).

[0067] As shown in FIG. 3, the computing environment 300 includes several stages: an image storage stage 305, a preprocessing stage 310, a labeling stage 315, a data augmentation stage 317, a training stage 320, and a result generation stage 325.

[0068] A. Image memory stage The image storage stage 305 includes one or more image data stores 330 (e.g., storage device 230 described in connection with FIG. 2 ) that are accessed (e.g., by the pre-processing stage 310) to provide a set of digital images 335 of pre-selected regions from a biological specimen slide (e.g., a histology slide) or the entire biological specimen slide. Each digital image 335 stored in each image data store 330 and accessed by the image storage stage 310 may include a digital pathology image generated according to some or all of the processes described with respect to the network 200 displayed in FIG. 2 . In some embodiments, each digital image 335 includes image data from one or more scanned slides. Each of the digital images 335 may correspond to image data from a single specimen and / or from a single day on which the underlying image data corresponding to the image was collected.

[0069] The image data may include the image, as well as any information about the color or wavelength channels, as well as details about the imaging platform on which the image was generated. For example, a tissue section may need to be stained by applying a staining assay containing one or more different biomarkers associated with a chromogenic stain for brightfield imaging or a fluorophore for fluorescent imaging. The staining assay may use a chromogenic stain for brightfield imaging, an organic fluorophore for fluorescent imaging, quantum dots, or a combination of organic fluorophores and quantum dots, or any other combination of stains, biomarkers, and observation or imaging devices. Examples of biomarkers include estrogen receptor (ER), human epidermal growth factor receptor 2 (HER2), human Ki-67 protein, progesterone receptor (PR), programmed cell death protein 1 (PD1), etc., and the tissue section may be detectably labeled with a binding agent (e.g., an antibody) for ER, HER2, Ki-67, PR, PD1, etc. In some embodiments, digital image and data analysis operations, such as classification, scoring, Cox modeling, and risk stratification, depend on the type of biomarker used and the field of view (FOV) selection and annotation. Furthermore, typical tissue sections are processed in an automated staining / assay platform that applies a staining assay to the tissue section, resulting in a stained sample. Various commercially available products suitable for use as staining / assay platforms exist on the market, one example being the VENTANA® SYMPHONY® product from the assignee, Ventana Medical Systems, Inc. The stained tissue section may be fed into an imaging system, such as a microscope or a whole slide scanner with a microscope and / or imaging components, one example being the VENTANA® iScan Coreo® / VENTANA® DP200 product from the assignee, Ventana Medical Systems, Inc. Multiple tissue slides can be scanned with an equivalent multiple slide scanner system.Additional information provided by the imaging system may include any information regarding the staining platform, including the concentration of chemicals used for staining, the reaction time of the chemicals applied to the tissue in the staining, and / or the pre-analysis conditions of the tissue, such as the age of the tissue, fixation method, duration, embedding method for sections, and cutting method.

[0070] B. Image pre-processing stage In the preprocessing stage 310, one, more, or all of the set of digital images 335 are each preprocessed using one or more techniques to generate corresponding preprocessed images 340. Preprocessing may include cropping the image. In some cases, preprocessing may further include standardizing or rescaling (e.g., normalizing) all features to the same scale (e.g., the same size scale or the same color or saturation scale). In certain cases, the image is resized to a minimum size (width or height) of a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels), optionally maintaining the original aspect ratio. Preprocessing may further include removing noise. For example, the image may be smoothed, such as by applying a Gaussian function or a Gaussian blur, to remove unwanted noise.

[0071] The preprocessed images 340 may include one or more training images, validation images, test images, and unlabeled images. It should be understood that the preprocessed images 340 corresponding to the training, validation, and unlabeled groups do not need to be accessed simultaneously. For example, an initial set of training and validation preprocessed images 340 may be initially accessed and used to train the machine learning algorithm 355, and subsequently, unlabeled input images may be accessed or received (e.g., one or more times thereafter) and used by the trained machine learning model 360 to provide a desired output (e.g., cell classification).

[0072] C. Labeling stage In some cases, the machine learning algorithm 355 is trained using supervised training, and some or all of the preprocessed image 340 is partially or fully labeled in the labeling stage 315, manually, semi-automatically, or automatically, with labels 345 that identify the “correct” interpretation (i.e., “ground truth”) of various biological materials and structures within the preprocessed image 340. For example, the labels 345 may identify features of interest (for example), cellular classifications, binary indications as to whether a given cell is a particular type of cell, binary indications as to whether the preprocessed image 340 (or a particular region having the preprocessed image 340) contains a particular type of indication (e.g., necrosis or artifact), categorical features of slide-level or region-specific indications (e.g., identifying a particular type of cell), counts (e.g., identifying the amount of a particular type of cell in a region, the amount of displayed artifact, or the amount of necrotic area), the presence or absence of one or more biomarkers, etc. In some cases, the labels 345 include location. For example, the label 345 may identify the point location of the nucleus of a particular type of cell, or the point location of a particular type of cell (e.g., a raw dot label). As another example, the label 345 may include the rim or boundary of a displayed tumor, blood vessel, necrotic region, etc. As another example, the label 345 may include one or more biomarkers identified based on biomarker patterns observed using one or more stains. For example, a tissue slide stained for a biomarker, such as programmed cell death protein 1 (“PD1”), may be observed and / or processed to label cells as either positive or negative cells, taking into account the expression level and pattern of PD1 in the tissue. Depending on the features of interest, a given labeled preprocessed image 340 may be associated with a single label 345 or multiple labels 345. In the latter case, each label 345 may be associated with (for example) an indication as to which location or portion within the preprocessed image 345 the label corresponds to.

[0073] The labels 345 assigned in the labeling stage 315 may be identified based on input from a human user (e.g., a pathologist or image scientist) and / or an algorithm (e.g., an annotation tool) configured to define the labels 345. In some cases, the labeling stage 315 may include transmitting and / or presenting some or all of the one or more preprocessed images 340 to a computing device operated by a user. In some cases, the labeling stage 315 includes utilizing an interface presented by the labeling controller 350 (e.g., using an API) at the computing device operated by the user, the interface including an input component for accepting input identifying labels 345 for features of interest. For example, a user interface may be provided by the labeling controller 350 that enables selection of an image or region of an image (e.g., FOV) for labeling. A user operating a terminal may select an image or FOV using the user interface. Several image or FOV selection mechanisms may be provided, such as specifying a known or irregular shape or defining an anatomical region of interest (e.g., tumor region). In one example, the image or FOV is an entire tumor region selected on an IHC slide stained with a combination of H&E stains. The selection of the image or FOV may be performed by a user or by an automated image analysis algorithm, such as tumor region segmentation on an H&E tissue slide. For example, a user may select the image or FOV as the entire slide or the entire tumor, or the entire slide or the entire tumor region may be automatically designated as the image or FOV using a segmentation algorithm. The user operating the terminal may then select one or more labels 345 to apply to the selected image or FOV, such as point locations on cells, positive markers for biomarkers expressed by the cells, negative biomarkers for biomarkers not expressed by the cells, or a boundary around the cells.

[0074] In some cases, the interface may identify which specific labels 345 are requested and / or to what extent, which may be communicated to the user via (e.g.) text instructions and / or visualization. For example, a particular color, size, and / or symbol may indicate that a label 345 is requested for a particular representation within an image (e.g., a particular cell or region or staining pattern) relative to other representations. If labels 345 corresponding to multiple representations are requested, the interface may identify each of the representations simultaneously, or may identify each representation sequentially (such that providing a label for one identified representation triggers the identification of the next representation for labeling). In some cases, each image is presented until the user identifies a particular number of labels 345 (e.g., of a particular type). For example, a given full slide image or a given patch of a full slide image may be presented until the user identifies the presence or absence of three different biomarkers, at which point the interface may present a different full slide image or image of a different patch (e.g., until a threshold number of images or patches have been labeled). Thus, in some cases, the interface is configured to request and / or accept labels 345 of an incomplete subset of features of interest, and the user may decide which of the potentially many representations will be labeled.

[0075] In some cases, the labeling stage 315 includes a labeling controller 350 that implements an annotation algorithm to semi-automatically or automatically label various features of an image or a region of interest within the image. The labeling controller 350 annotates the image or FOV on the first slide according to user input or an annotation algorithm and maps the annotations across the remaining slides. Depending on the defined FOV, several methods for annotation and alignment are possible. For example, the entire tumor region annotated on the H&E slide from multiple consecutive slides may be selected automatically or by the user on an interface such as VIRTUOSO / VERSO™. Because the other tissue slides correspond to serial sections from the same tissue block, the labeling controller 350 performs a marker-to-marker alignment operation to map and transfer the entire tumor annotation from the H&E slide to each of the remaining IHC slides in the series. Exemplary methods for marker-to-marker registration are described in further detail in commonly assigned International Publication No. WO 2014140070, filed March 12, 2014, entitled "Whole Slide Image Registration and Cross-Image Annotation Devices, Systems, and Methods," which is incorporated herein by reference in its entirety for all purposes. In some embodiments, any other method for image registration and generation of whole-tumor annotations may be used. For example, a qualified radiologist, such as a pathologist, may annotate whole-tumor regions on any other IHC slides and execute the labeling controller 350 to map the whole-tumor annotations on other digitized slides. For example, a pathologist (or an automated detection algorithm) may annotate whole-tumor regions on an H&E slide and trigger analysis of all adjacent serially sectioned IHC slides to determine a whole-slide tumor score for the annotated regions on all slides.

[0076] In some cases, the labeling stage 315 further includes an annotation processing system 351 that implements an annotation algorithm to identify annotation position and annotation label conflicts within a set of annotations associated with the image (or the FOV of the image). The annotation processing system 351 can determine a consensus position for a set of annotations located at different positions within a region of the training image. In some cases, the annotation processing system 351 determines that an annotation position conflict exists for a region in the training image by determining that two or more annotations from the same annotator are present within the region. The annotation processing system 351 can resolve such position conflicts by retaining the annotation that has the closest distance to other annotations within the region while discarding other annotations from the same annotator. At the determined consensus position, a consensus label can be determined for the set of annotations that identify different target types of biological structures. The consensus labels across different positions can be used to generate ground truth labels for the image. The ground truth labels can be used to train, validate, and / or test machine learning models configured to predict different types of biological structures within digital pathology images.

[0077] D. Augmentation Phase In the augmentation stage 317, a training set of labeled or unlabeled images (original images) from the preprocessed images 340 is augmented with synthetic images 352 generated using an augmentation control 354 that runs one or more augmentation algorithms. Augmentation techniques are used to artificially increase the amount and / or type of training data by adding slightly modified synthetic copies of existing training data or newly created synthetic data from existing training data. As described herein, variations between scanners and laboratories can cause variations in intensity and color within digital images. Additionally, poor scanning can result in gradient shifts and blurring effects, assay staining can result in staining artifacts such as background washout, and different tissue / patient samples can result in variations in cell size. These variations and fluctuations can adversely affect the quality and reliability of deep learning and artificial intelligence networks. The augmentation techniques implemented in the augmentation stage 317 act as regularizers of these variations and fluctuations, helping to reduce overfitting when training machine learning models. It should be understood that the enhancement techniques described herein can be used as regularizers for any number and type of variations and fluctuations and are not limited to the various specific examples described herein.

[0078] E. Training Phase During the training phase 320, the labels 345 and corresponding preprocessed images 340 can be used by a training controller 365 to train a machine learning algorithm 355 according to various workflows described herein. For example, to train the algorithm 355, the preprocessed images 340 may be partitioned into a subset of images 340a (e.g., 90%) for training and a subset of images 340b (e.g., 10%) for validation. The partitioning may be performed randomly (e.g., 90 / 10% or 70 / 30%), or according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc., to minimize sampling bias and overfitting. The partitioning may also be performed based on the inclusion of augmented or synthetic images 352 within the preprocessed images 340. For example, it may be beneficial to limit the number or proportion of synthetic images 352 included in the subset of training images 340a. In some cases, the ratio of original image 335 to composite image 352 is maintained at 1:1, 1:2, 2:1, 1:3, 3:1, 1:4, or 4:1.

[0079] In some cases, the machine learning algorithm 355 includes a CNN, a modified CNN with the encoding layer replaced by a residual neural network ("Resnet"), or a modified CNN with the encoding and decoding layers replaced by a Resnet. In other examples, the machine learning algorithm 355 may be any suitable machine learning algorithm configured to localize, classify, and / or analyze the preprocessed image 340, such as a two-dimensional CNN ("2DCNN"), Mask R-CNN, U-Net, feature pyramid network (FPN), dynamic time warping ("DTW") technique, hidden Markov model ("HMM"), pure attention-based model, or one or more combinations of such techniques, such as a visual transformer, CNN-HMM, or MCNN (multiscale convolutional neural network). The computing environment 300 may employ the same type of machine learning algorithm or different types of machine learning algorithms trained to detect and classify different cells. For example, the computing environment 300 may include a first machine learning algorithm (e.g., U-Net) for detecting and classifying PD1. The computing environment 500 may also include a second machine learning algorithm (e.g., 2DCNN) for detecting and classifying cluster of differentiation 68 ("CD68"). The computing environment 300 may also include a third machine learning algorithm (e.g., U-Net) for detecting and classifying PD1 and CD68 in combination. The computing environment 300 may also include a fourth machine learning algorithm (e.g., HMM) for diagnosing disease for treatment or prognosis of a subject, such as a patient. In other examples according to the present disclosure, still other types of machine learning algorithms may be implemented.

[0080] The training process of the machine learning algorithm 355 includes selecting hyperparameters for the machine learning algorithm 355 from the parameter data store 363, inputting a subset of images 340a (e.g., labels 345 and corresponding preprocessed images 340) into the machine learning algorithm 355, and performing iterative operations to learn a set of parameters (e.g., one or more coefficients and / or weights) for the machine learning algorithm 355. Hyperparameters are settings that can be adjusted or optimized to control the behavior of the machine learning algorithm 355. Most algorithms explicitly define hyperparameters that control different aspects of the algorithm, such as memory or execution cost. However, additional hyperparameters may be defined to adapt the algorithm to specific scenarios. For example, hyperparameters may include the number of hidden units of the algorithm, the learning rate of the algorithm (e.g., 1e-4), the convolution kernel width, or the number of kernels of the algorithm. In some cases, the number of model parameters is reduced for each convolutional and deconvolutional layer, and / or the number of kernels is reduced by half for each convolutional and deconvolutional layer compared to a typical CNN.

[0081] The subset of images 340a may be input to the machine learning algorithm 355 in batches of a predetermined size. The batch size limits the number of images shown to the machine learning algorithm 355 before parameter updates can be performed. Alternatively, the subset of images 340a may be input to the machine learning algorithm 355 as a time series or sequentially. In either case, if augmented or composite images 352 are included in the preprocessed images 340a, the number of original images 335 versus the number of composite images 352 included in each batch, or the manner in which the original images 335 and phenotypic images 352 are fed to the algorithm (e.g., every other batch or image is the original batch of images or original image), can be defined as a hyperparameter.

[0082] Each parameter is a tunable variable such that values ​​for the parameter are adjusted during training. For example, a cost function or objective function may be configured to optimize accurate classification of displayed expressions, optimize characterization of a given type of feature (e.g., characterization of shape, size, uniformity, etc.), optimize detection of a given type of feature, and / or optimize accurate localization of a given type of feature. Each iteration may include learning a set of parameters for the machine learning algorithm 355 that minimizes or maximizes a cost function of the machine learning algorithm 355, such that the value of the cost function using a set of parameters may be smaller or larger than the value of the cost function using a different set of parameters in a previous iteration. The cost function may be constructed to measure the difference between the predicted output using the machine learning algorithm 355 and the labels 345 contained in the training data. For example, in the case of a model based on supervised learning, the goal of training is to learn a function “h()” (sometimes called a hypothesis function) that maps a training input space X to a target value space Y, h: X → Y, where h(x) is a good predictor for the corresponding value of y. A variety of different techniques may be used to learn this hypothesis function. In some techniques, as part of deriving the hypothesis function, a cost or loss function may be defined that measures the difference between the ground truth value for an input and the predicted value for that input. As part of training, techniques such as backpropagation, random feedback, direct feedback alignment (DFA), indirect feedback alignment (IFA), and Hebbian learning are used to minimize this cost or loss function.

[0083] The training iterations continue until a stopping condition is met. The training completion condition may be configured to be met (for example) when a predetermined number of training iterations are completed, when a statistical value generated based on testing or validation exceeds a predetermined threshold (e.g., a classification accuracy threshold), when a statistical value generated based on a confidence metric (e.g., the mean or median of the confidence metric or the percentage of the confidence metric above a particular value) exceeds a predetermined confidence threshold, and / or when a user device involved in the training review closes the training application executed by the training controller 365. Once a set of model parameters is identified through training, the machine learning algorithm 355 is trained, and the training controller 365 performs an additional process of testing or validation using the subset of images 340b (the test or validation dataset). The validation process may include iterative operations of inputting images from the subset of images 340b into the machine learning algorithm 355 using validation techniques such as k-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, and nested cross-validation to adjust the hyperparameters and ultimately find an optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set of images from the subset of images 340b is input into the machine learning algorithm 355 to obtain an output, which is evaluated against the ground truth by calculating performance metrics such as error, accuracy, precision, recall, receiver operating characteristic curves (ROC), etc. using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient. In some cases, a new training iteration may be initiated (e.g., initial model development, model update / adaptation, continuous learning, drift determined within the trained machine learning model 360, etc.) in response to receiving a corresponding request or trigger condition from a user device.

[0084] As will be appreciated, other training / validation mechanisms are contemplated and may be implemented within computing environment 300. For example, machine learning algorithm 355 may be trained and hyperparameters may be tuned on images from subset of images 340a, while images from subset of images 340b may be used solely to test and evaluate the performance of machine learning algorithm 355. Furthermore, the training mechanisms described herein focus on training new machine learning algorithms 355. These training mechanisms may also be utilized for initial model development, model updating / adaptation, and continuous learning of existing machine learning models 360 trained from other datasets, as described in detail herein. For example, in some cases, machine learning model 360 may have been pre-trained using images of other subjects or biological structures, or from sections from other subjects or studies (e.g., human or mouse studies). In those cases, machine learning model 360 may be used for initial model development, model updating / adaptation, and continuous learning using preprocessed images 340.

[0085] F. Result Generation Phase The trained machine learning model 360 can then be used (in the result generation stage 325) to process the new preprocessed image 340 to generate predictions or inferences, such as predicting cell center and / or location probabilities, classifying cell types, generating cell masks (e.g., segmentation masks for each pixel of the image), predicting a disease diagnosis or prognosis for a subject, such as a patient, or any combination thereof. In some cases, the masks identify the locations of displayed cells associated with one or more biomarkers. For example, given tissue stained for a single biomarker, the trained machine learning model 360 can be configured to (i) infer cell centers and / or locations, (ii) classify the cells based on features of the staining pattern associated with the biomarker, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cells. As another example, given tissue stained for two biomarkers, the trained machine learning model 360 may be configured to (i) infer cell centers and / or locations, (ii) classify the cells based on features of the staining patterns associated with the two biomarkers, and (iii) output a cell detection mask for cells positive for the first biomarker, a cell detection mask for cells negative for the first biomarker, a cell detection mask for cells positive for the second biomarker, and a cell detection mask for cells negative for the second biomarker. As another example, given tissue stained for a single biomarker, the trained machine learning model 360 may be configured to (i) infer cell centers and / or locations, (ii) classify the cells based on features of the cells and the staining patterns associated with the biomarkers, and (iii) output a cell detection mask for positive cells and a cell detection mask for negative cell codes, as well as mask cells classified as tissue cells.

[0086] In some cases, analysis controller 380 generates analysis results 385 that are utilized by the entity that requested processing of the underlying image. Analysis results 385 may include a mask output from trained machine learning model 360 overlaid on new preprocessed image 340. Additionally or alternatively, analysis results 385 may include information calculated or determined from the output of the trained machine learning model, such as a whole-slide tumor score. In an exemplary embodiment, the automated analysis of the tissue slide uses Assignee VENTANA's FDA-cleared 510(k) algorithm. Alternatively or additionally, any other automated algorithm may be used to analyze selected regions of the image (e.g., masked images) and generate a score. In some embodiments, analysis controller 380 may further respond to instructions received from a computing device from a pathologist, physician, investigator (e.g., associated with a clinical trial), patient, medical professional, etc. In some cases, the communication from the computing device includes an identifier for each of a particular set of subjects and corresponds to a request to perform an iteration of the analysis for each subject represented in the set. The computing device may further perform analysis based on the machine learning model and / or the output of analysis controller 380 and / or may provide a recommended diagnosis / treatment to the subject.

[0087] It will be understood that computing environment 300 is exemplary and that computing environments 300 having different stages and / or using different components are contemplated. For example, in some cases, a network may omit preprocessing stage 310, such that the images used to train an algorithm and / or images processed by a model are raw images (e.g., from an image data store). As another example, it will be understood that each of preprocessing stage 310 and training stage 320 can include a controller for performing one or more operations described herein. Similarly, while labeling stage 315 is shown in relation to labeling controller 350 and result generation stage 325 is shown in relation to analysis controller 380, the controllers associated with each stage may additionally or alternatively facilitate other operations described herein other than generating labels and / or generating analysis results. 3 lacks displayed representations of devices associated with a programmer (e.g., who selected the architecture of the machine learning algorithm 355, which defined how various interfaces would function, etc.), devices associated with a user providing an initial label or label review (e.g., in the labeling stage 315), and devices associated with a user requesting model processing for a given image (which may be the same user as the user who provided the initial label or label review or a different user). Despite the lack of representations of these devices, the computing environment 300 may include use of one, more, or all of the devices, and may in fact include use of multiple devices associated with a corresponding number of users providing initial labels or label reviews and / or a corresponding number of users requesting model processing for various images.

[0088] V. Generate a single composite image from the double images FIG. 4 shows an example of a dual ER / PR and adjacent single IHC ER / PR image 400, according to some embodiments. The dual and single images 400 can be obtained using the process performed by the network 200 of FIG. 2. In FIG. 4, the dual ER / PR image 402 can be stained for both estrogen receptor and progesterone receptor, the single ER image 404 can be stained for estrogen receptor only, and the single PR image 406 can be stained for progesterone receptor only. Dual IHC staining of tissue sections enables simultaneous detection of two or more biomarkers. Dual IHC staining further enables co-expression of both biomarkers at the single-cell level. Dual IHC staining does not require two IHC stains and additional registration to identify co-localized biomarkers. Despite these advantages, biomarkers in dual IHC images are often difficult for pathologists to distinguish because the biomarkers often overlap and cluster with each other. In practice, the manual scoring process of dual IHC images can be challenging. In the exemplary image 400 of FIG. 4, Tamra (purple) was used to stain PR, Quinone Methide Dabsyl (yellow) was used to stain ER, and hematoxylin (blue) was used as a counterstain.

[0089] 5 illustrates a field of view (FOV) 500 selected on a dual ER / PR image and registered on a single IHC ER image and a single IHC PR image, according to some embodiments. In FIG. 5, an FOV is selected on the dual ER / PR image 502, and then the FOV is registered and overlaid on a single IHC ER image 504 and a single IHC PR image 506. In some embodiments, the FOV is selected by one or more pathologists and presented directly on the dual ER / PR image 502. An automatic registration algorithm can be utilized to register the dual ER / PR image 502 on the single IHC ER image 504 and the single IHC PR image 506.

[0090] The double images can be unmixed to generate a composite single image, where each composite single image can display cells stained for a single biomarker (e.g., ER). In some cases, each double image can be processed to generate three monochrome images, two corresponding to each of the two stains (e.g., ER, PR), and a third corresponding to a counterstain (e.g., hematoxylin). Each single-stain image can be combined with a counterstain ("CS") image to generate a composite single image. Each composite single image can be processed using one or more machine learning models to predict the phenotype of each cell, which can be associated with a corresponding type of biomarker. Other types of image preprocessing can be applied to generate and modify the composite single images, and are described in the preprocessing stage 310 of FIG. 3 . In some cases, the composite single images can be securely annotated or reviewed by a pathologist to generate a set of training images.

[0091] Additionally or alternatively, the composite single image may be processed using a color correction method to reduce the range of colors displayed on the display device. FIGS. 6A-6C illustrate an exemplary technique for generating a composite single image, according to some embodiments. In FIG. 6A, the display device 604a has a limited color gamut and may not be able to display all possible colors captured by the slide scanning device 602a. Therefore, a color correction operation 606a may be performed on the raw image. The color correction 606a may be used to ensure that the depicted color space falls within the color gamut of the display device 604a. The corrected image may then be displayed on the display device 604a. The color-corrected image may include colors similar to those visible under a microscope. In some cases, a color deconvolution technique is used to perform the color correction operation 606a.

[0092] FIG. 6B is a schematic diagram illustrating a process 600B for generating a composite Dabsyl ER image and a composite PR image from a dual ER / PR image, according to some embodiments. In FIG. 6B, dual ER / PR image 602b can be unblended to generate a raw composite single image 604b, which includes a composite ER image and a composite PR image. Each raw composite single image 604b can be processed using a color correction operation to generate a color-normalized composite single image 606b. In some cases, composite ER / PR image 606b can be provided to a pathologist for scoring, and the score can be compared with a registered single ER / PR image 608b to determine the presence of system / color unblending or staining errors. A specific slide cutting sequence can be used to enable accurate comparison between the composite image and the single-stained image (e.g., to avoid tissue gaps during registration). For example, a cutting sequence of "H&E, single ER, dual ER / PR, single PR" can be used to reduce tissue gaps during the registration process.

[0093] FIG. 6C shows an exemplary set of color-corrected composite single images 600C generated from dual images, according to some embodiments. The first row of images includes processing an exemplary dual image stained with ER / PR 602c to generate a color-corrected composite image displaying the presence of PR biomarker 604c. As shown in FIG. 6C, the color-corrected composite image displays a similar biomarker pattern as that of the single PR image displaying adjacent tissue portion 606c. Similarly, the second row of images includes processing another exemplary dual image stained with ER / PR 608c to generate another color-corrected composite image displaying the presence of ER biomarker 610c. As shown in FIG. 6C, the color-corrected composite image displays a similar biomarker pattern as that of the single ER image displaying adjacent tissue portion 612c.

[0094] VI. Training a Machine Learning Model to Detect Phenotypes in Dual Images To train the machine learning model, the image processing system can generate training data including a set of training images. The training images can be labeled with ground truth labels, including ER+, PR+, ER-, PR+, and others (e.g., stromal cells, immune cells, artifacts). In some cases, the image processing system performs a training process that includes: (i) a pathologist selecting one or more fields of view (FOVs) within the double image; (ii) aligning the FOVs to adjacent single images based on the FOVs selected from the double image; (iii) extracting FOVs from both the double ER / PR and the single ER and single PR; (iv) generating a composite Tamra PR / composite Dabsyl ER image from the double image, thereby generating four images for each FOV; (v) a pathologist scoring all FOVs, including the composite single image; and (vi) using the scores to train an ER-specific machine learning model and a PR-specific machine learning model.

[0095] A. Training Data FIG. 7 shows an exemplary set of training images 700 for training a machine learning model to detect cellular phenotypes in a composite single image, according to some embodiments. The training images may be generated from double images 702 displaying at least a portion of a biological sample (e.g., a tissue section) and stained with two or more biomarkers (e.g., ER, PR). The double images 702 may be unblended into composite single images 704. Each of the composite single images 704 may represent at least a portion of the biological sample stained with a single corresponding biomarker. In some cases, the composite single images 704 are compared to a ground truth single image 706 to remove staining errors. The ground truth single image 706 may correspond to a single image representing an adjacent tissue portion of the tissue portion of the double image 702. To compare the composite single image 704 with the ground truth single image 706, a registration operation may be performed to align the ground truth single image 706 and the composite single image 704 to a single coordinate system.

[0096] Once staining errors have been removed, training image 708 can be generated by adding one or more training labels to composite single image 704. Thus, training image 708 can correspond to a composite single image that includes multiple training labels. Each training label of the multiple training labels can include (i) the location of a cell identified by the training label and (ii) a phenotype of the cell for the corresponding biomarker. For example, a red training label can indicate a "tumor-positive" cancer cell, a green training label can indicate a "tumor-negative" non-cancerous cell, a blue training label can indicate a stromal cell, a yellow training label can indicate an immune cell, and a black training label can indicate an artifact. The above steps can be repeated to generate a set of training images for training each machine learning model.

[0097] B. Model Selection FIG. 8 illustrates an exemplary architecture of a machine learning model used to detect phenotypes in dual-slide images, according to some embodiments. As shown in FIG. 8, U-Net 800 may include a reduction path 805 and an expansion path 810, resulting in a U-shaped architecture. The reduction path 805 is a CNN network that includes repeated application of convolutions (e.g., 3×3 convolutions (unpadded convolutions)), each followed by a rectified linear unit (ReLU) and max-pooling operation for downsampling (e.g., max 2×2 pooling with stride 2). At each downsampling step or pooling operation, the number of feature channels may be doubled. During reduction, the spatial information of the image data decreases, but the feature information increases. The expansion path 810 is a CNN network that combines the features from the reduction path 805 with the spatial information (upsampling the feature maps from the reduction path 805). The upsampling of the feature maps is followed by a series of upconvolutions (upsampling operators) that halve the number of channels, concatenation with the corresponding cropped feature maps from the reduction path 805, repeated application of convolutions (e.g., two 3x3 convolutions), each followed by a rectified linear unit (ReLU), and a final convolution (e.g., a 1x1 convolution) to generate a 2D tumor mask. For localization, the high-resolution features from the reduction path 805 are combined with the upsampled output from the augmentation path 810. The U-Net 800 uses the effective portion of each convolution without fully connected layers; i.e., the segmentation map contains only pixels for which the full context is available in the input image, and uses skip connections to link the context features learned during the reduction block with the localization features learned in the augmentation block.

[0098] In a conventional U-Net architecture, a convolutional block consists of convolutional layers (e.g., typically two or three layers) for performing convolutions. However, according to various embodiments, the convolutional block and convolutional layers are replaced with a residual block 815 having separable convolutions performed in pyramid layers 820 at one or more augmentation levels (e.g., stacked filtered images). (A single convolutional layer may be replaced with two or more pyramid layers 820). Figure 9A shows the layer structure of the residual block 815 shown in Figure 8. As shown, the residual block 900 may include multiple pyramid layers 905. In a network (e.g., a ResNet) including the residual block 900, each pyramid layer 905 feeds the next layer (A, B, C, etc.) and then directly feeds layers (D, E, etc.) approximately two to three layers away. The use of the residual block 900 in the network helps overcome the degradation problem that arises from increasing the number of pyramid layers (if the number of layers continues to increase, accuracy will initially increase but at some point begin to saturate and eventually degrade). The residual block 900 uses skip or residual connections to skip some of these additional pyramid layers, ultimately propagating large gradients to the initial pyramid layers. Skipping effectively simplifies the network by using fewer pyramid layers in the initial training phase. This speeds up learning by reducing the impact of vanishing gradients because there are fewer layers to propagate (i.e., multi-speed residual learning). The network then gradually restores the skipped layers as it learns the feature space.

[0099] FIG. 9B illustrates a single pyramid layer 910 of FIG. 9A in accordance with various embodiments. As shown in FIG. 9B, the pyramid layer 910 may use attrausive separable convolutions at multiple different scales (“attrausive blocks”) (four levels in this example). The pyramid layer 910 includes the same image at multiple different scales to improve the accuracy of object (e.g., tumor) detection. Attrausive convolutions refer to filters with “widened” receptive fields that increase the size of the receptive field relative to the kernel size. In some embodiments, the one or more attrausive levels are four attrausive levels. In other embodiments, more or fewer attrausive levels, e.g., six attrausive levels, may be used. The convolution layer output 915 is the output of attrausive blocks 920 (here labeled as attrausive levels 1, 2, 4, and 8). The illustrated example of FIG. 9B assumes four attrausive blocks, each outputting two channels (of the same color), for a total of eight channels. The number of channels output by each attrausive block may vary depending on the residual block in question. The example of Figure 9B shows the upper left or upper right residual block 815 of Figure 8. In some embodiments, the number of channels output by each enhancement block 915 in the pyramid layer 910 of the residual block 905 is equal to the number of k filters on the residual block 905 divided by four.

[0100] Empirical evidence shows that the residual block allows for improved accuracy and easier optimization. Separable convolution, depthwise convolution followed by pointwise convolution, also shows a significant increase in convergence speed and a significant reduction in model size. Dilated convolutions expand the receptive field without losing resolution, thus enabling multiscale contextual information to be aggregated with downsampling. Redesigning the convolution block allows for the extraction of highly localized and rare information within an image.

[0101] C. Methods for training machine learning models to detect phenotypes in dual images 10 illustrates a process 1000 for training a machine learning model to detect phenotypes in dual images, according to some embodiments. For illustrative purposes, the process 1000 is described with reference to the image processing system 100 of FIG. 1 and / or the components shown in FIG. 3, although other implementations are possible. For example, program code for the computing environment 300 of FIG. 3, stored on a non-transitory computer-readable medium, may be executed by one or more processing devices to cause a server system to perform one or more operations described herein.

[0102] In step 1002, the training subsystem accesses a digital pathology image displaying at least a portion of a biological sample (e.g., a tissue section). The digital pathology image may be a dual image stained for a first type of biomarker and a second type of biomarker. In some cases, the digital pathology image corresponds to a portion (e.g., an image tile) of a larger digital image.

[0103] In step 1004, the training subsystem unmixes the digital pathology images to generate (i) a first composite single image displaying at least a portion of the biological sample in which a first type of biomarker is identified, and (ii) a second composite single image displaying at least a portion of the biological sample in which a second type of biomarker is identified. In some cases, the composite single image is generated by (i) generating preprocessed images displaying cells stained for the corresponding biomarkers, and (ii) combining the preprocessed images with a counterstain image. The counterstain image can be used to visually distinguish the biomarkers from other cellular structures shown in the preprocessed images.

[0104] Further, the first type of biomarker can correspond to an ER biomarker, and the second type of biomarker can correspond to a PR biomarker. Additionally or alternatively, the first and second composite single images can be further processed using a color correction operation to facilitate the addition of training labels. In some cases, the first and second composite single images are compared with respective single images displaying another portion of the biological sample (e.g., an adjacent tissue section) to address any possible errors, such as staining errors.

[0105] In step 1006, the training subsystem adds a first set of training labels to the first composite single image to generate a first training image. Each training label can identify a cell's location and a cell's phenotype for the corresponding biomarker identified by the training label. For example, the first set of training labels can include (i) an ER+ classification, (ii) an ER- classification, (iii) a stromal cell classification, (iv) an immune cell classification, and (v) an artifact classification. In some cases, the training labels are added to the first composite single image by one or more pathologists.

[0106] In step 1008, the training subsystem trains a first machine learning model using the first training images. Continuing with the above example, the first machine learning model can be trained to predict the phenotype of cells in images stained for an ER biomarker, which includes adjusting parameters of the first machine learning model based on a loss calculated between the output of the first machine learning model and a ground truth corresponding to the first synthetic single image. The first machine learning model can be trained to generate a set of probability maps for the first training images. Each probability map in the set of probability maps can represent multiple pixels of the synthetic single image and can correspond to a particular classification among the set of classifications. For example, the first machine learning model can be trained to generate a first set of probability maps for the first training images, the first set of probability maps including: (i) a first probability map for an ER+ classification; (ii) a second probability map for an ER− classification; (iii) a third probability map for a stromal cell classification; (iv) a fourth probability map for an immune cell classification; and (v) a fifth probability map for an artifact classification. In some cases, the first machine learning model is a U-Net model.

[0107] In step 1010, the training subsystem adds a second set of training labels to the second composite single image to generate a second training image. Each training label can identify the location of a cell identified by the training label and a phenotype of the cell for the corresponding biomarker. For example, the first set of training labels can include (i) a PR+ classification, (ii) a PR- classification, (iii) a stromal cell classification, (iv) an immune cell classification, and (v) an artifact classification. In some cases, the training labels are added to the second composite single image by one or more pathologists.

[0108] In step 1012, the training subsystem trains a second machine learning model using the second training images. Continuing with the above example, the second machine learning model can be trained to predict the phenotype of cells in images stained for PR biomarkers, which includes adjusting the parameters of the second machine learning model based on a loss calculated between the output of the second machine learning model and a ground truth corresponding to the second synthetic single image. The second machine learning model can be trained to generate a set of probability maps for the second training images. For example, the first machine learning model can be trained to generate a first set of probability maps for the first training images, the first set of probability maps including: (i) a first probability map for PR+ classification; (ii) a second probability map for PR− classification; (iii) a third probability map for stromal cell classification; (iv) a fourth probability map for immune cell classification; and (v) a fifth probability map for artifact classification. In some cases, the second machine learning model is a U-Net model. Thus, the first machine learning model and the second machine learning model can be trained using separate training images, such that the trained machine learning models can accurately predict the phenotype of the cells shown in each composite single image.

[0109] In step 1014, the training subsystem provides the first and second machine learning models. For example, the first and second machine learning models can be accessed by another computing system (e.g., image processing system 100 of FIG. 1) via a communications network to predict the phenotype of cells in another dual image. In some cases, the first and second machine learning models are accessed by the other system when the losses determined for each of the first and second machine learning models fall below a predetermined threshold. Process 1000 then ends.

[0110] VII. Merging Cellular Phenotypes for Multiple Biomarkers The two trained machine learning models can be used to generate initial seed locations and predict phenotypes at the seed locations. In particular, a first machine learning model (e.g., an ER model) can be used to generate phenotypes of cells in a synthetic single ER image, and a second machine learning model (e.g., a PR model) can be used to generate phenotypes of cells in a synthetic single PR image.

[0111] FIG. 11 shows exemplary seed locations and different class labels generated by two models, according to some embodiments. In FIG. 11, image 1102 identifies red "diamonds" generated by the ER model, which detected 1,762 cells (including ER+, ER-, immune, matrix, and background). The first image 1102 also identifies green "Xs" generated by the PR model, which detected 1,034 cells. A magnified portion 1104 of the image further shows that there are cells in the dual image whose phenotypes are difficult to identify. This difficulty may be due to multiple green and red marks clustered in the same image. A different region 1106 of the dual image shows an example of a cell whose phenotype may be difficult to identify based on multiple biomarkers. For example, a cell may be predicted by the ER model to be an ER-positive cell, while the PR model may predict the same cell to correspond to other cells, such as immune and stromal cells.

[0112] A. Generating probability maps from synthetic single images using trained machine learning models To address the above-mentioned challenges, the image processing system can merge phenotypes generated by the first and second machine learning models. The phenotype merging can be performed using a probability map. In particular, the probability maps generated by the machine learning models can be compared for a given image region, and an output for the given region can be determined based on the probability map. The use of the probability map provides advantages over existing techniques, such as tuning heuristic parameters to assign classes.

[0113] FIG. 12 illustrates five probability maps 1200 generated by an ER model, according to some embodiments. Each probability map in the probability maps 1200 can represent multiple pixels of the composite ER image and can correspond to a particular classification among a set of classifications. The probability map includes, for each pixel of the multiple pixels, a probability value indicating whether the pixel corresponds to that classification. As shown in FIG. 12, the probability map 1200 representing the composite ER image can include: (i) a first probability map 1204 for an ER+ classification; (ii) a second probability map 1206 for an ER− classification; (iii) a third probability map 1208 for an immune cell classification; (iv) a fourth probability map 1210 for a stromal cell classification; and (v) a fifth probability map 1212 for an artifact / noise classification. In addition to the probability maps, the ER model can also generate a background / object map 1214. For each pixel in the ER channel, the image processing system can compare the probability values ​​among the five probability maps to determine whether the pixel should be assigned as ER-positive, ER-negative, an immune cell, a stromal cell, or other cells. The PR model can also generate five probability maps by processing the synthetic PR image. The probability maps generated by the PR model can be used to determine whether the pixel will be PR-positive, PR-negative, an immune cell, a stromal cell, or other cells.

[0114] B. Generating a merged probability map to define the set of anchor points The probability maps generated by each of the ER and PR models can be merged to generate a merged set of probability maps. A set of anchor points can be determined based on the merged set of probability maps. The set of anchor points can correspond to locations within the dual image that can identify the presence of one or more biomarkers. Figure 13 illustrates the computation of five merged probability maps 1300 between the probability maps generated by two U-Net models in the ER and PR channels, according to some embodiments. In Figure 13, the ER model can be used on a synthetic ER image to generate object maps and five probability maps 1302 for ER tumor positive, ER tumor negative, immune cells, stromal cells, and other cells. The PR model can be used on a synthetic PR image to generate object maps and five probability maps 1302 for PR tumor positive, PR tumor negative, immune cells, stromal cells, and other cells. The image processing system can compare both probability maps 1302 and 1304 to determine the maximum probability for each pixel in the ER and PR tumor positive probability maps. The image processing system can also determine the maximum probability for each pixel in the ER and PR tumor negative probability maps. The image processing system can continue to perform the above process for immune probability maps, stromal probability maps and other cellular probability maps between the ER and PR channels.

[0115] After calculating the five merged probability maps 1306 from the ER and PR channel images, the image processing system can compare the maximum probability of each pixel in the merged probability maps 1306 within a specific distance or region, and predict the pixel with the maximum probability as the potential center of each cell or the anchor point of each cell.

[0116] Figure 14 shows an example of anchor points 1400 obtained by exploring merged probability maps between the ER and PR channels, according to some embodiments. In Figure 14, anchor points 1400 are overlaid on each of a composite PR image 1402 and a composite ER image 1404. Blue squares in images 1402 and 1404 identify anchor points, red diamonds correspond to the first detected cells in the Tamra PR image 1402 and the Dabsyl ER image 1404, with anchor points at the centers of each cell, and all red dots combining five types of labels in the ER / PR channels. In some embodiments, the anchor points correspond to different parts of the cells (e.g., the nucleus).

[0117] C. Association of anchor points with corresponding phenotypes 15 illustrates an exemplary anchor point and surrounding ER+ and PR+ points at the pixel level, according to some embodiments. Shown in FIG. 15 is an exemplary anchor point 1502 at the pixel level, and surrounding ER+ and PR+ points 1504 and 1506. In some embodiments, a k-nearest neighbor or distance algorithm is applied to calculate the distance between the anchor point and the ER+, PR+ points 1504 and 1506. For example, if both the ER+ and PR+ locations are within a predetermined distance from the anchor point, the anchor point can be assigned both ER+ and PR+ labels.

[0118] Additionally or alternatively, labeling techniques can be used. FIG. 16 shows an exemplary label graph 1600 determined from the probability map, according to some embodiments. FIG. 16 further illustrates a logical AND operation applied to the five label graphs and anchor points. For each point in the five probability maps in the ER channel (ER+ "1," ER- "2," stromal- "3," immune "4," and other "5"), the image processing system can calculate a surrounding label graph 1602. In some cases, the label graph can correspond to a 10×10 pixel region surrounding the point classified as exhibiting a phenotype (e.g., ER+) for the corresponding type of biomarker. In the label graph 1604, the anchor point can also be marked as a "red" circle, and a logical "&" operation and the five label graphs are used to determine whether the anchor point belongs to the ER+, ER-, immune, stromal, or other classification. For example, anchor point 1606 can be assigned to the ER+ classification because it is within a 10×10 pixel region corresponding to a point classified as ER+. Another ER-labeled graph overlaps with the ER+ labeled graph, but the image processing system can assign a higher weight to the ER+ labeled graph and assign the anchor point an ER+ classification over other classifications.

[0119] i. Association of anchor points with ER classification FIG. 17 illustrates an example of assigning anchor points to ER+ / ER- / other, according to some embodiments. In FIG. 17 , an image processing system generates a pixel-distance map 1702 for a composite single image 1704 (e.g., an ER channel image). The pixel-distance map 1702 includes a set of points classified as having a phenotype for a corresponding type of biomarker, with each point in the set of points defined by a predetermined pixel area (e.g., 10×10 pixels). The image processing system can overlay the pixel-distance map 1702 on the set of anchor points corresponding to the composite single image 1704, thereby generating a mapped image 1706 including the classification of the set of anchor points. A magnified portion 1708 of the mapped image 1706 is shown, showing one or more anchor points (defined as “red” dots) associated with corresponding pixel areas (defined as “white” squares). The mapped image 1706 can be combined with another mapped image (not shown) generated for the composite PR image to generate an output image 1710 including colocalization of the ER and PR biomarkers. The above techniques may have advantages over other existing algorithms because they may perform faster, more efficiently, and more accurately.

[0120] FIG. 18 illustrates a process 1800 for assigning anchor points to ER+ / ER- / other in a composite ER image, according to some embodiments. In FIG. 18, process 1800 includes a process for assigning classification labels (e.g., ER+ / ER- / other) to anchor points in the ER channel using the “&” operator, where (i) the portion of the “ER+” label map that overlaps with the anchor point generates an ER+ classification for the anchor point (block 1802), (ii) the portion of the “ER-” label map that overlaps with the anchor point generates an ER- classification for the anchor point (block 1804), and (iii) the portion of the “immune / stromal / artifact” label map that overlaps with the anchor point generates an “other” classification for the anchor point (block 1806). In some cases, when the image processing system calculates the maximum probability of classification for pixels in the composite single image, weakly stained ER-positive cells may not be considered and may be overlooked. To address the above issue, the image processing system can add weakly stained ER-positive cells in this step (block 1808).

[0121] ii. Association of anchor points with PR classification Similarly, the image processing system can repeat the equivalent process on the composite PR image shown in FIGS. 19 and 20. FIG. 19 is a diagram illustrating an example of assigning anchor points to PR+ / PR- / etc., according to some embodiments. In FIG. 19, the image processing system generates a pixel-distance map 1902 for a composite single image 1904 (e.g., a PR channel image). The pixel-distance map 1902 includes a set of points classified as having a PR biomarker phenotype, with each point in the set defined by a predetermined pixel area (e.g., 10×10 pixels). The image processing system can overlay the pixel-distance map 1902 on the set of anchor points corresponding to the composite single image 1904, thereby generating a mapped image 1906 including the classification of the set of anchor points. The mapped image 1906 can be combined with the mapped image 1806 generated for the composite ER image to generate an output image 1910 including colocalization of the ER and PR biomarkers.

[0122] FIG. 20 illustrates a process 2000 for assigning anchor points to PR+ / PR- / other in a composite PR image, according to some embodiments. In FIG. 20, process 2000 includes a process for assigning classification labels (e.g., PR+ / PR- / other) to anchor points in the PR channel using the “&” operator, where (i) the portion of the “PR+” label map that overlaps with the anchor point generates a PR+ classification for the anchor point (block 2002), (ii) the portion of the “PR-” label map that overlaps with the anchor point generates a PR- classification for the anchor point (block 2004), and (iii) the portion of the “immune / stromal / artifact” label map that overlaps with the anchor point generates an “other” classification for the anchor point (block 2006). In some cases, when the image processing system calculates the maximum probability of classification for pixels in the composite single-plane image, weakly stained PR-positive cells may not be considered and may be overlooked. To address the above issue, the image processing system can add weakly stained PR-positive cells in this step (block 2008).

[0123] D. Merging phenotypes into a double image In some embodiments, the system is configured to combine and merge ER+ / PR+ / ER- / PR- / etc. for all different types of anchor points. Classifications can be determined according to different prior probabilities in the composite ER and PR images. For example, equal weights can be assigned to ER+ / PR+ / ER- / PR- in the composite ER and PR images, and the image processing system can combine the above phenotypes detected in the two composite ER and PR images. Other cells detected in the composite ER and PR images can be assigned smaller weights. For example, if a pixel is detected as ER+ in the ER channel and PR+ in the PR channel, the image processing system can identify equal weights and assign the pixel to the ER+PR+ co-expression phenotype. In another example, if a pixel is detected as ER+ in the ER channel and stromal cells are detected in the PR channel, the image processing system can identify two different weights and assign the pixel to the ER+ classification only. Figure 21 shows a set of exemplary images showing merged phenotypes overlaid on a double image in some embodiments.

[0124] E. How to use machine learning techniques to detect phenotypes in dual images 22 illustrates a process 2200 for detecting phenotypes in dual images using a trained machine learning model, according to some embodiments. For illustrative purposes, the process 2200 is described with reference to the image processing system 100 of FIG. 1 and / or the components shown in FIG. 3, although other implementations are possible. For example, program code for the computing environment 300 of FIG. 3, stored on a non-transitory computer-readable medium, may be executed by one or more processing devices to cause a server system to perform one or more operations described herein.

[0125] In step 2202, the image processing system accesses a digital pathology image displaying at least a portion of a biological sample (e.g., a tissue section). The digital pathology image may be a dual image stained for a first type of biomarker and a second type of biomarker. In some cases, the digital pathology image corresponds to a portion (e.g., an image tile) of a larger digital image.

[0126] In step 2204, the image processing system unmixes the digital pathology images to generate (i) a first composite single image displaying at least a portion of the biological sample in which a first type of biomarker is identified, and (ii) a second composite single image displaying at least a portion of the biological sample in which a second type of biomarker is identified. In some cases, the first type of biomarker corresponds to an ER biomarker, and the second type of biomarker corresponds to a PR biomarker. In some cases, the composite single image is generated by (i) generating a preprocessed image displaying cells stained for the corresponding biomarker, and (ii) combining the preprocessed image with a counterstain image. The counterstain image can be used to visually distinguish the biomarkers from other cellular structures shown in the preprocessed image.

[0127] In step 2206, the image processing system applies a first machine learning model to the first composite single image to (i) detect a first plurality of cells from the first composite single image and (ii) determine a classification from a first set of classifications for each cell of the first plurality of cells. The first machine learning model may be a first trained U-Net model. The classification from the first set indicates whether the cell contains a biomarker having a first type of biomarker. The first machine learning model may be an ER model trained using process 1000 of FIG. 10. The first set of classifications may identify a cellular phenotype for the ER biomarker, the first set of classifications including: (i) an ER-positive classification (ER+); (ii) an ER-negative classification biomarker (ER-); (iii) a stromal cell classification; (iv) an immune cell classification; and (v) an artifact or other biological structure classification. In some cases, the image processing system applies the first machine learning model to generate a first set of probability maps representing the first composite single image. Each probability map in the first set of probability maps can represent a plurality of pixels in the first composite single image and can correspond to a particular classification in the first set of classifications, and the probability map includes, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to that classification.

[0128] In step 2208, the image processing system applies a second machine learning model to the second composite single image to (i) detect a second plurality of cells from the second composite single image and (ii) determine a classification among a second set of classifications for each cell of the second plurality of cells. The second machine learning model may be a second trained U-Net model trained separately from the first trained U-Net model used to generate the first set of classifications. The classification among the second set indicates whether the cell contains a biomarker having a second type of biomarker. The second machine learning model may be a PR model trained using process 1000 of FIG. 10. The second set of classifications may identify a cellular phenotype for the PR biomarker, the second set of classifications including: (i) a PR-positive classification (PR+), (ii) a PR-negative classification biomarker (PR-), (iii) a stromal cell classification, (iv) an immune cell classification, and (v) an artifact or other biological structure classification. In some cases, the image processing system applies a second machine learning model to generate a second set of probability maps representing the second composite single image. Each probability map in the second set of probability maps can represent a plurality of pixels in the second composite single image and can correspond to a particular classification in the second set of classifications. For each pixel in the plurality of pixels, the probability map includes a probability value indicating whether the pixel corresponds to that classification. The first set of classifications can include classifications that are different from the second set of classifications.

[0129] In step 2210, the image processing system merges the classifications of the first plurality of cells with the classifications of the second plurality of cells to generate merged classifications. Merging the first set of classifications with the second set of classifications facilitates identification of image regions of the digital pathology image displaying cells associated with multiple biomarker phenotypes (e.g., ER / PR). In some cases, the image processing system merges the first set of classifications with the second set of classifications by merging the first set of classifications with the second set of classifications. The set of anchor points correspond to locations in the digital pathology image, and for each anchor point, the presence of one or more biomarkers can be identified. To merge the classifications, the image processing system assigns a corresponding classification from the first set of classifications and a corresponding classification from the second set of classifications to each anchor point in the set of anchor points. Corresponding classifications from the first and second sets can be identified based on determining whether the anchor point is within a predetermined distance (e.g., 10 pixels) from the respective locations of the corresponding classifications.

[0130] In step 2212, the image processing system outputs the digital pathology image with the merged classification. The merged classification can identify a phenotype of the detected cells, which can indicate the presence of multiple biomarkers stained for the digital pathology image. In some cases, the image processing system outputs the digital pathology image by overlaying the merged classification on the digital pathology image.

[0131] VII. Training an Enhanced Machine Learning Model Using Merged Phenotypes In some cases, the merged phenotype can be used to train another machine learning model to predict the phenotype of cells in the dual image (referred to as an "enhanced machine learning model"), where the enhanced machine learning model does not require unmixing of the dual image. Figure 23 shows a training pipeline 2300 using merged phenotypes for the dual ER / PR algorithm, according to some embodiments.

[0132] In Figure 23, an image processing system may be configured to receive an image of a pathology slide stained to indicate two or more types of biomarkers (block 2302). In some embodiments, the image processing system is configured to operate using images of a double slide stained to indicate the presence of estrogen receptor (ER) and progesterone receptor (PR) proteins. Each cell in the image may be classified as positive or negative for each of the ER and PR markers. Thus, the phenotype of each cell may be identified as ER+PR+, ER+PR-, ER-PR+, ER-PR-, or other (e.g., stromal, immune, necrotic, artifact, etc.).

[0133] In some cases, the image processing system divides the dual slide image into multiple image tiles (block 2304). Identification of cellular phenotypes for the multiple biomarkers may be performed for each of the multiple image tiles. In some cases, a set of image tiles is selected from the multiple image tiles, and the set of image tiles is used as training images for training one or more machine learning models to detect cellular phenotypes for the types of biomarkers.

[0134] The image processing system can identify a merged phenotype of the dual images to generate a training dual image (block 2306). The training dual image can include a set of training labels, each of which identifies a corresponding cellular phenotype for multiple biomarkers (e.g., ER+ / PR+). The merged phenotype of the training image can be generated using the steps of process 2200 described in FIG. 22. In some cases, the image processing system identifies a merged phenotype for each image tile of the set of image tiles to generate a set of training images for training a machine learning model. In some cases, the merged phenotype of the set of training images is compared to ground truth training labels manually generated by a pathologist (block 2308).

[0135] In some embodiments, the image processing system uses the set of image tiles with the merged phenotypes (i.e., training images) to train an enhanced machine learning model (e.g., a U-Net model) such that the enhanced machine learning model can predict the phenotype of each detected cell for multiple biomarkers. For example, the enhanced machine learning model can be trained using the set of training images to predict the phenotype of a cell for both the ER / PR biomarkers (block 2310).

[0136] The trained enhanced machine learning model can then be used to detect the phenotype of cells in the other corresponding dual image (block 2312). As a result, the image processing system can predict the phenotype of multiple biomarkers using a single enhanced machine learning model (block 2314). In some cases, the image processing system overlays the merged phenotype on the dual image. Training the enhanced machine learning model using the merged phenotype can improve the accuracy of detecting multiple biomarkers in the dual image without performing image preprocessing steps such as color unmixing operations.

[0137] VIII. Experimental Results The performance of the present machine learning technique is comparable to or better than manual annotation performed by a pathologist, thereby demonstrating improved accuracy of image processing systems in identifying the presence of multiple biomarkers in dual images.

[0138] A. Comparison of consensus scores Figures 24A-24B show the consensus scores of three pathologists and the U-Net and merged phenotype algorithms, according to some embodiments. Figure 24A shows the consensus scores (e.g., average scores) of three pathologists for ER biomarkers compared with the scores corresponding to the merged phenotype predicted by the machine learning technology described in aspects of the present disclosure. Furthermore, Figure 24B shows the consensus scores of three pathologists for PR biomarkers compared with the scores corresponding to the merged phenotype predicted by the machine learning technology. As shown in Figures 24A-24B, the merged phenotypes (green dots) generated by the machine learning technology were within the range of the scores of the three pathologists (red bars) and very close to the consensus score (yellow dots). Thus, the graphs in Figures 24A-24B demonstrate that the machine learning technology can perform comparably to annotations manually performed by pathologists.

[0139] Table 1 further lists the correlation between the "phenotype merging + U-Net" algorithm and the consensus pathologist scores (median of the scores of three pathologists). [Table 1]

[0140] U-Net Table 1 shows that the U-Net algorithm and phenotype merging perform comparably to pathologists in detecting ER and PR biomarkers.

[0141] B. Accuracy evaluation of the enhanced machine learning model In another evaluation, a pathologist selected 50 FOVs from six slides with a range of ER and PR biomarker diversity and intensity. The designed merged phenotype trained a single U-Net model on the dual images to detect co-expression of ER+PR+, ER+PR-, ER-PR+, ER-PR-, and other cells. The enhanced U-Net model can be trained using the steps described in process 2300 of Figure 23. Furthermore, the following configuration was used to train the enhanced U-Net model: (i) the optimizer included the Adam Optimizer; (ii) the learning rate was set to 1e-4; (iii) the epochs were set to 100 / 200; (iv) the training / validation ratio was set to 80 / 20; (v) the patch size was set to 256 × 256; and (vi) the loss function was set to the cross-entropy function. Table 2 provides the training dataset used to train the U-Net model. The accuracy results of the trained U-Net model are listed in Table 3. [Table 2] [Table 3]

[0142] C. Qualitative results In addition to quantitative results, qualitative data was also generated to evaluate the performance of the machine learning model. Figure 25 shows an example of phenotypic detection of a dual ER / PR result, according to some embodiments. Figure 25 shows an example of phenotypic detection of a dual ER / PR result, including a merged phenotype overlaid on an ER / PR dual image 2502, a merged phenotype overlaid on a composite ER image 2504, and a merged phenotype overlaid on a composite PR image 2506.

[0143] Figures 26A-B display additional examples of dual ER / PR with different types of merged phenotypes. Figure 26A shows an example of ER+ / PR+ coexpression in dual ER / PR2602 and an example of PR+ dominance in dual ER / PR2604. Figure 26B shows an example of ER+ dominance in dual ER / PR2606 and an example of tumor-negative dominance in dual ER / PR2608.

[0144] X. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0145] The terms and expressions employed are used as terms of description and not of limitation, and the use of such terms and expressions is not intended to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the claimed invention. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be employed by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0146] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0147] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. accessing a digital pathology image displaying at least a portion of a biological sample stained for a first type of biomarker and a second type of biomarker; unmixing the digital pathology image; a first composite single image displaying the at least a portion of the biological sample in which the first type of biomarker is identified; and a second composite single image displaying said at least a portion of said biological sample in which said second type of biomarker is identified; and applying a first machine learning model to the first synthetic single image; Detecting a first plurality of cells from the first composite single image; determining a classification among a first set of classifications for each cell of the first plurality of cells, the classification among the first set indicating whether the cell contains biomarkers having the first type of biomarker; And, applying a second machine learning model to the second synthetic single image; Detecting a second plurality of cells from the second composite single image; determining a classification among a second set of classifications for each cell of the second plurality of cells, the classification among the second set indicating whether the cell contains a biomarker having the second type of biomarker, and the first set of classifications is different from the second set of classifications; And, merging the classification of the first plurality of cells with the classification of the second plurality of cells to generate a merged classification; outputting the digital pathology image with the merged classification; Including, determining the classification for the first plurality of cells; generating a first set of probability maps, each probability map of the first set of probability maps comprising a plurality of pixels and associated with a classification of the first set of classifications, the probability map comprising, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to the classification; determining a classification for the second plurality of cells; generating a second set of probability maps, each probability map of the second set of probability maps comprising a plurality of pixels and associated with a classification of the second set of classifications, the probability map comprising, for each pixel of the plurality of pixels, a probability value indicating whether the pixel corresponds to the classification; the first set of probability maps and the second set of probability maps are merged to generate a set of anchor points, each anchor point in the set of anchor points being assigned a first classification from the first set of classifications and a second classification from the second set of classifications.

2. A non-transitory computer readable medium storing a plurality of instructions that, when executed, control a computer system to perform the method of claim 1.

3. The non-transitory computer-readable medium of claim 2; one or more processors for executing instructions stored on the computer-readable medium; A system comprising:

4. A system comprising means for performing the method of claim 1.

5. 10. A system comprising one or more processors configured to perform the method of claim 1.

6. A system comprising modules for performing each of the steps of the method of claim 1.

Images