Systems and methods for cell classification

Abstract

The present disclosure relates to automated systems and methods suitable for quickly and accurately training a neural network to detect and / or classify cells and / or nuclei. The present disclosure also relates to automated systems and methods using a trained cell detection and classification engine, such as a neural network, to classify cells within unlabeled images.

Description

[0001] Cross Reference to Related Patent Applications

[0002] This application claims the benefit of the filing date of U.S. Patent Application No. 62 / 830,823 filed April 8, 2019; and also claims the benefit of the filing date of U.S. Patent Application No. 62 / 745,953 filed October 15, 2018, the entire disclosure of which is hereby incorporated by reference in its entirety. BACKGROUND

[0003] Digital pathology is the scanning of entire histopathology or cytology slides into digital images that can be interpreted on a computer screen. Subsequently, these images are processed using imaging algorithms or interpreted by pathologists. To examine the tissue sections (which are almost transparent), the tissue sections are prepared using color histochemical stains that selectively bind to cellular components. The color enhancement or staining of cellular structures is utilized by clinicians or computer-aided diagnostic (CAD) algorithms to identify morphological markers of disease and proceed with treatment accordingly. Various processes, including disease diagnosis, assessment of response to treatment, and development of new drugs to combat disease, can be achieved by observation at the time of determination.

[0004] Immunohistochemistry (IHC) slide staining can be used to identify proteins in cells of a tissue section and is therefore widely used to study different types of cells, such as cancer cells and immune cells in biological tissues. Thus, IHC staining can be used to study the distribution and localization of biomarkers differentially expressed by immune cells (such as T cells or B cells) in cancerous tissues for immune response studies. For example, tumors often contain infiltrates of immune cells, which can either prevent the development of the tumor or promote tumor growth.

[0005] In situ hybridization (ISH) can be used to determine whether there is specific amplification of a genetic aberration or oncogene in cells that morphologically appear malignant when viewed under a microscope. In situ hybridization (ISH) uses labeled DNA or RNA probe molecules that are antisense to a target gene sequence or transcript to detect or localize a targeted nucleic acid target gene within a cell or tissue sample. ISH is accomplished by exposing a cell or tissue sample fixed on a glass slide to a labeled nucleic acid probe that is capable of specifically hybridizing to a given target gene in the cell or tissue sample. Multiple target genes can be analyzed simultaneously by exposing a cell or tissue sample to multiple nucleic acid probes that have been labeled by multiple different nucleic acid tags. With labels having different emission wavelengths, simultaneous multicolor analysis can be performed on a single target cell or tissue sample in a single step. For example, the INFORM HER2 Dual ISH DNA Probe Mix assay from Ventana Medical Systems, Inc. is designed to determine the status of the HER2 gene by calculating the ratio of the HER2 gene to chromosome 17. Dual-color chromogenic ISH detects HER2 and chromosome 17 probes in formalin-fixed, paraffin-embedded human breast cancer tissue specimens. SUMMARY

[0006] Cell and / or nucleus detection is a core task of digital pathology and generally refers to the identification and localization of individual cells in stained cancer tissue images. Cancer histology images vary in size, resolution, stain type, and are cluttered with overlapping cells that are visually heterogeneous. Challenges commonly faced when detecting cells and / or nuclei from these images are the difficulty and sometimes impossibility of: obtaining ground truth annotations; and applying models tuned for a particular resolution or stain to datasets with different resolutions or different stains. In fact, the process of ground truth labeling is often cumbersome and requires a significant investment of time. Therefore, it is desirable to develop systems and methods that facilitate ground truth labeling for cell and / or nucleus classification.

[0007] In view of the foregoing, the Applicant has developed systems and methods that facilitate the generation of ground truth data in a fast and robust manner, thereby addressing the problem of time-consuming and labor-intensive ground truth annotation. To achieve this, and as described in detail herein, the systems and methods facilitate the generation of homogenous clusters of cells and / or nuclei in a sample image, wherein the generated homogenous clusters themselves can be annotated (rather than individual cells and / or nuclei). This alleviates the burden of ground truth labeling for ordering from the number of cells in a dataset to the number of clusters identified in the dataset. The generation of homogenous clusters of cells and / or nuclei is unconventional, and thus transforms the system input into an output that enables fast and accurate ground truth generation. As such, the technical problem of ground truth annotation is addressed by using a system that is adapted for fast and accurate generation of homogenous clusters, such as a system that is pre-trained using image datasets that are not related to histology images and / or cytologist images, and / or a system that is minimally adjusted using only histology and / or cytology images. The systems and methods described herein also enable fast and accurate detection of cells and / or nuclei in test images using a cell detection and classification engine, thereby facilitating accurate diagnosis of diseases such as cancer, and thus facilitating strengthening of treatment of diseased subjects (e.g., human patients). The systems and methods described herein rely on tangible data sources, namely image data corresponding to biological specimens, such as biological specimens derived from patients in need of treatment.

[0008] One aspect of the present disclosure is a system for identifying cell clusters within a sample image of a biological specimen that is stained for the presence of one or more biomarkers, the system comprising: (i) one or more processors, and (ii) one or more memories coupled with the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: (a) detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine comprises a convolutional neural network adapted to detect features of cells within the sample image; (b) extracting the features of the cells from one or more layers of the convolutional neural network; and (c) clustering the detected cells in the sample image based on the extracted features of the cells to provide one or more homogenous clusters of the detected cells.

[0009] In certain embodiments, the trained object detection engine comprises a region-based convolutional neural network. In certain embodiments, the region-based convolutional neural network is adapted from Fast-RCNN. In certain embodiments, the trained object detection engine is adapted from Faster-RCNN. In certain embodiments, the trained object detection engine further comprises a region proposal network.

[0010] In certain embodiments, the convolutional neural network of the trained object detection engine is adapted from one of LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, and DenseNet, or any variant thereof. In certain embodiments, the convolutional neural network of the trained object detection engine is a deep residual network. In certain embodiments, the deep residual network is adapted from ResNet. In certain embodiments, the neural network is a self-encoding neural network.

[0011] In certain embodiments, the convolutional neural network includes at least one scaling layer. In certain embodiments, the at least one scaling layer scales the sample image by a constant factor. In certain embodiments, the clustering of the detected cells includes performing agglomerative hierarchical clustering.

[0012] In certain embodiments, the convolutional neural network is trained using a dataset including a plurality of training images, wherein each training image of the plurality of training images is derived from a biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers (e.g., a plurality of training images stained for the presence of HER2). In certain embodiments, each training image of the plurality of training images includes at least one class label. In certain embodiments, each training image of the plurality of training images does not include any class label. In certain embodiments, each training image of the plurality of training images does not include any pathologist annotation. In those embodiments where the object detection engine is trained using training images including at least one class label, the system can further include instructions for classifying the detected cells. In certain embodiments, the sample image includes a different stain than the stain in any one of the plurality of training images used to train the object detection engine.

[0013] In certain embodiments, the system is adapted to receive an annotation of at least one of the one or more provided homogeneous clusters corresponding to a cell in the sample image. In certain embodiments, the annotation is provided by a pathologist.

[0014] In certain embodiments, the system further includes instructions for training a cell detection and classification engine using the received annotations. In certain embodiments, the system further includes instructions for detecting cells in a test image using the trained cell detection and classification engine. In certain embodiments, the system further includes instructions for classifying cells in a test image using the trained cell detection and classification engine. In certain embodiments, the test image includes at least one stain. In certain embodiments, the at least one stain of the test image is a cell stain. In certain embodiments, the at least one stain of the test image is a membrane stain. In certain embodiments, the at least one stain in the test image is different from the stains used to train the cell detection and classification engine.

[0015] In certain embodiments, the cell detection and classification engine includes a neural network. In certain embodiments, the neural network is adapted from LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, DenseNet, or ResNet. In certain embodiments, the neural network is a self-encoding neural network. In other embodiments, the cell detection and classification engine includes a classifier. In certain embodiments, the classifier is a support vector machine or an ensemble learning method, such as a random forest supervised learning algorithm. In certain embodiments, the cell detection and classification module includes a neural network and a classifier. In certain embodiments, the neural network of the cell detection and classification module is adapted to extract features from an input test image.

[0016] In certain embodiments, the system further includes instructions for fine-tuning the trained object detection engine using the received annotations to provide a fine-tuned object detection engine. In certain embodiments, the system further includes instructions for detecting cells in a test image using the fine-tuned object detection engine. In certain embodiments, the system further includes instructions for classifying cells in a test image using the fine-tuned object detection engine.

[0017] Another aspect of the present disclosure is a non-transitory computer readable medium storing instructions for identifying cell clusters within a sample image of a biological specimen stained for presence of one or more biomarkers, comprising: (a) detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine includes a first portion and a second portion, wherein the first portion is configured to identify cell features within the sample image, and wherein the second portion is configured to detect cells based on the identified cell features; (b) extracting at least a portion of the identified cell features from one or more layers of the trained object detection engine; and (c) clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of the detected cells.

[0018] In certain embodiments, the first portion comprises a neural network. In certain embodiments, the neural network is a convolutional neural network. In certain embodiments, the convolutional neural network is adapted from LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, and DenseNet. In certain embodiments, the convolutional neural network is a deep residual network. In certain embodiments, the deep residual network is adapted from ResNet. In certain embodiments, the second portion comprises a region proposal network. In certain embodiments, the object detection engine is trained using a dataset comprising a plurality of training images of biological specimens stained with a primary stain or stained for the presence of one or more biomarkers. In certain embodiments, the object detection engine is trained using a dataset comprising a plurality of training images, wherein each training image of the plurality of training images is stained with a primary stain or stained for the presence of one or more biomarkers, and each training image of the plurality of training images does not contain any class label or pathologist annotation.

[0019] Another aspect of the present disclosure is a non-transitory computer-readable medium storing instructions for identifying cell clusters within a sample image of a biological specimen stained for the presence of one or more biomarkers, comprising: (a) detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine comprises a convolutional neural network adapted to detect cell features within the sample image; (b) extracting cell features from one or more layers of the convolutional neural network; and (c) clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of the detected cells.

[0020] In certain embodiments, the object detection engine is trained using a dataset comprising a plurality of training images of biological specimens stained with a primary stain or stained for the presence of one or more biomarkers. In certain embodiments, the object detection engine is trained using a dataset comprising a plurality of training images, wherein each training image of the plurality of training images is stained with a primary stain or stained for the presence of one or more biomarkers, and each training image of the plurality of training images does not contain any class label or pathologist annotation.

[0021] In certain embodiments, the convolutional neural network is adapted from LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, and DenseNet. In certain embodiments, the convolutional neural network is a deep residual network. In certain embodiments, the deep residual network is adapted from ResNet. In certain embodiments, the trained object detection engine includes a network architecture adapted from Faster-RCNN. In certain embodiments, the neural network is a self-encoding neural network.

[0022] In certain embodiments, the non-transitory computer-readable medium further includes instructions for training a cell detection and classification engine using pathologist annotations corresponding to at least one of the one or more homogeneous clusters of cells in the sample image. In certain embodiments, the non-transitory computer-readable medium further includes instructions for detecting cells in a test image using the trained cell detection and classification engine. In certain embodiments, the non-transitory computer-readable medium further includes instructions for classifying detected cells in the test image using the trained cell detection and classification engine. In certain embodiments, the cell detection and classification engine includes a neural network. In certain embodiments, the cell detection and classification engine includes a support vector machine.

[0023] In certain embodiments, the non-transitory computer-readable medium further includes instructions for fine-tuning the trained object detection engine using annotations received for at least one of the one or more homogeneous clusters of cells in the sample image to provide a fine-tuned object detection engine. In certain embodiments, the non-transitory computer-readable medium further includes instructions for detecting cells in a test image using the fine-tuned object detection engine.

[0024] Another aspect of the present disclosure is a method of identifying a cluster of cells within a sample image of a biological specimen stained for presence of one or more biomarkers, the method comprising: detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine includes a convolutional neural network and a region proposal network, wherein the convolutional neural network is configured to identify cell features, and wherein the region proposal network is configured to detect cells based on the identified cell features; extracting the identified cell features from one or more layers of the convolutional neural network; and, clustering the detected cells in the sample image based on the extracted cell features to identify one or more homogeneous clusters of the detected cells.

[0025] In certain embodiments, the trained object detection engine includes a region-based convolutional neural network. In certain embodiments, the region-based convolutional neural network is adapted from Fast-RCNN. In certain embodiments, the trained object detection engine is adapted from Faster-RCNN. In certain embodiments, the convolutional neural network is adapted from LeNet, AlexNet, ZFNet, DetectNet, GoogleNet, VGGNet, VGG16, and DenseNet. In certain embodiments, the convolutional neural network is a deep residual network. In certain embodiments, the deep residual network is adapted from ResNet. In certain embodiments, the convolutional neural network is a self-encoding neural network.

[0026] In certain embodiments, the object detection engine is trained using a dataset including a plurality of training images of a biological specimen stained with a primary stain or stained for the presence of one or more biomarkers. In certain embodiments, the object detection engine is trained using a dataset including a plurality of training images, wherein each training image of the plurality of training images is stained with a primary stain or stained for the presence of one or more biomarkers, and each training image of the plurality of training images does not include any class label or pathologist annotation.

[0027] Another aspect of the present disclosure is a method of detecting cells in a test image of a biological specimen stained for the presence of one or more biomarkers, the method comprising: determining cell features within the test image of the biological specimen by inputting the test image into a first portion of a trained cell detection and classification engine, wherein the first portion includes a convolutional neural network configured to detect cell features; and detecting cells in the test image based on the determined cell features by inputting the determined features of the test image into a second portion of the trained cell detection and classification engine, wherein the second portion includes at least a region proposal network. In certain embodiments, the convolutional neural network is a deep residual network. In certain embodiments, the deep residual network is adapted from ResNet.

[0028] In certain embodiments, the method further comprises classifying the detected cells in the test image. In certain embodiments, the cell detection and classification engine is trained using ground truth data, wherein the ground truth data comprises pathologist annotations corresponding to one or more identified cell homogenous clusters originating from one or more sample images. In certain embodiments, the cell homogenous clusters originating from one or more sample images are identified by: (i) providing the sample images to the trained object detection engine; (ii) detecting cells in the sample images using the trained object detection engine; (iii) extracting a plurality of cell features from at least one convolutional layer within the trained object detection engine; and (iv) clustering the detected cells in the test image based on the extracted cell features. In certain embodiments, the object detection engine is trained using a dataset comprising a plurality of training images, each training image of the plurality of training images originating from a biological specimen stained with a hematoxylin stain or stained for the presence of one or more biomarkers.

[0029] Another aspect of the present disclosure is a method of detecting cells in a test image of a biological specimen stained for the presence of one or more biomarkers, the method comprising: determining cell features within the test image of the biological specimen by inputting the test image into a first portion of a trained cell detection and classification engine, wherein the first portion comprises a neural network configured to detect the cell features; and detecting cells in the test image based on the determined cell features by inputting the determined features of the test image into a second portion of the trained cell detection and classification engine, wherein the second portion comprises a trained classifier. In certain embodiments, the trained classifier comprises a support vector machine or a random forest ensemble learning method. BRIEF DESCRIPTION OF DRAWINGS

[0030] The features of the present disclosure are best understood from the attached drawings. In the drawings, like reference numerals are used to identify like elements.

[0031] FIG. 1 A representative digital pathology system is shown that includes an image acquisition device and a computer system, according to one embodiment of the present disclosure.

[0032] FIG. 2A Various modules are listed that can be used in a digital pathology system or in a digital pathology workflow to detect cells in sample images, according to one embodiment of the present disclosure.

[0033] FIG. 2B Various modules are listed that can be used in a digital pathology system or in a digital pathology workflow to detect cells in sample images, according to one embodiment of the present disclosure.

[0034] FIG. 2C Various modules are listed that can be used in a digital pathology system or in a digital pathology workflow to detect and / or classify cells in test images according to one embodiment of the present disclosure.

[0035] FIG. 3A A flowchart showing steps of generating cell clusters according to one embodiment of the present disclosure is provided.

[0036] FIG. 3B A flowchart showing steps of generating cell clusters according to one embodiment of the present disclosure is provided.

[0037] FIG. 4 A flowchart showing steps of adjusting a pre-trained neural network according to one embodiment of the present disclosure is provided.

[0038] FIG. 5 A flowchart showing steps of detecting and / or classifying cells in test images according to one embodiment of the present disclosure is provided.

[0039] FIG. 6A An overview of the Region-based Convolutional Neural Network (RCNN) methodology according to one embodiment of the present disclosure is provided.

[0040] FIG. 6B An overview of the Fast-RCNN methodology according to one embodiment of the present disclosure is provided.

[0041] FIG. 6C An overview of the Faster-RCNN methodology according to one embodiment of the present disclosure is provided.

[0042] FIG. 7 Results of cell detection on a sample image followed by identification of homogenous clusters of the detected cells are shown. In certain embodiments, a neural network is trained using a minimal labeled dataset. In certain embodiments, the trained neural network is then used to detect cells in large images and to extract relevant features of these cells. In certain embodiments, the cells are then clustered based on their features, thereby providing homogenous clusters of cells to a pathologist that can be labeled in batches rather than one by one, thereby significantly reducing the workload of the ground truth related task. In certain embodiments, the identified homogenous clusters can be labeled by a pathologist and the labeled clusters can be used as ground truth data. It is believed that employing Faster-RCNN enables a pathologist to provide labels to only the detected bounding boxes of cell candidates in the detected clusters, which can be distinguished by different colors.

[0043] FIG. 8An example of results of cell detection using Model A on Her2 Dual ISH images is provided (top). The figure also provides ground truth annotations for the corresponding image patches (bottom).

[0044] FIG. 9A Results of clustering using a trained object detection engine based on Faster-RCNN architecture are shown.

[0045] FIG. 9B Clustering of cells using RGB values of detected cells is shown. The bounding boxes can be generated by the Matlab image Labeler APP. Methods such as K-means are used to compute the clusters. In certain embodiments, as further described herein, a agglomerative clustering is implemented in scikit-learn.

[0046] FIG. 10A A plot showing clustering evaluation based on Model A representative BB data (i.e., two class annotations of 1822 hematoxylin and DAB stained cells) is provided.

[0047] FIG. 10B A plot showing clustering evaluation based on Model B representative BBYP data (i.e., 1340 annotated images of IHC Duplex staining containing 5 imbalanced classes with member counts ranging from 10 to 1828) is provided.

[0048] FIG. 11 Examples of T cells (right) and tumor cells (left) in the BBYP dataset are provided. Ki67+ and Ki67- tumor cells have the same color as Ki68+ and Ki67- T cells, respectively. These cells can be distinguished based on their size, shape, and environment. DETAILED DESCRIPTION

[0049] It should also be understood that, unless clearly indicated to the contrary, in any methods claimed herein that include more than one step or act, the order of the steps or acts of the method is not necessarily limited to the order in which the steps or acts are recited.

[0050] As used herein, the singular terms "a," "an," and "the" include plural referents unless the context clearly indicates otherwise. Similarly, the word "or" is intended to include "and" unless the context clearly indicates otherwise. The term "includes" is defined inclusively, such that "includes A or B" means A, B, or both A and B.

[0051] As used herein in the specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly

[0052] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," and the like are used synonymously to mean and intended to convey the full scope of each term, including that a statement that "comprises," "comprising," "includes," "including," "has," "having," and the like are used inclusively, in an open-ended way, and that additional elements can be included within the scope of these terms. Similarly, the terms "consisting essentially of and "consisting of are used as defined by statute in the United States. Specifically, the definition of each of these terms is consistent with the definition of "consisting of in the United States Patent Law, and thus each of these terms can be construed to be an open term, meaning that additional elements can be present beyond those recited, but that the additional elements do not materially change the basic and novel characteristics of the claimed application. Accordingly, for example, a device "comprising components a, b, and c" means that the device includes at least components a, b, and c. Likewise, the phrase "a method comprising steps a, b, and c" means that the method includes at least steps a, b, and c. Furthermore, to the extent that the terms "comprising," "including," containing," and the like are used in the detailed description and

[0053] As used herein in the specification and claims, the phrase "at least one," in reference to a list of one or more elements, should be understood to mean that at least one element from among that one or more elements is present in the list, but does not necessarily include every element specifically listed in the list of elements and does not preclude additional elements not listed. The additional elements can be present in the list of elements, whether or not specifically listed. As a non-limiting example, in the phrase "at least one of A and B" (or, equivalently, "at least one of A or B"), every instance of A will not necessarily be present in some embodiments; similarly, every instance of B will not necessarily be present in some embodiments. In one embodiment, "at least one of A and B" will include only A, exclusive of B; in another embodiment, "at least one of A and B" will include only B, exclusive of A; in yet another embodiment, "at least one of A and B" will include both A and B; in still another embodiment, "at least one of A and B" will include virtually any combination of A and B, including any number of instances of A and any number of instances of B. In various embodiments, the number of instances of A included in "at least one of A and B" can be greater than the number of instances of B included in "at least one of A and B"; in other embodiments, the number of instances of A included in "at least one of A and B" can be less than the number of instances of B included in "at least one of A and B"; in other embodiments, the number of instances of A included in "at least one of A and B" can equal the number of instances of B included in "at least one of A and B."

[0054] As used herein, the term "biological specimen," "sample," or "tissue sample" refers to any sample obtained from any organism (including viruses) that includes biological molecules (e.g., proteins, peptides, cellular acids, lipids, carbohydrates, or combinations thereof). Examples of other organisms include mammals (e.g., humans; veterinary animals such as cats, dogs, horses, cows, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological specimens include tissue samples (such as tissue sections and punch biopsies of tissue), cell samples (such as cytological smears, such as a Pap smear or a blood smear, or cell samples obtained by microdissection), or cell fractions, fragments, or organelles (such as obtained by lysing a cell and separating its components by centrifugation or otherwise). Other examples of biological specimens include blood, serum, urine, semen, fecal matter, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or punch biopsy), nipple aspirate, cerumen, milk, vaginal secretions, saliva, swabs (e.g., buccal swabs), or any material containing biological molecules derived from a first biological specimen. In certain embodiments, the term "biological specimen" as used herein refers to a sample (such as a homogenized or liquefied sample) prepared from a tumor or a portion thereof obtained from a subject.

[0055] As used herein, the term "biomarker" or "marker" refers to a measurable indicator of certain biological state or condition. In particular, a biomarker can be a protein or peptide, e.g. a surface protein, that can be specifically stained and that is indicative of a biological characteristic of a cell, e.g. cell type or physiological state of a cell. An immune cell marker is a biomarker that selectively indicates a feature related to the immune response of a mammal. Biomarkers can be used to determine the degree of response of the body to a treatment of a disease or condition or whether a subject is susceptible to a disease or condition. In the case of cancer, a biomarker refers to a biological substance that is indicative of the presence of cancer in the body. A biomarker can be a molecule secreted by a tumor or a specific response of the body to the presence of cancer. Genetic, epigenetic, proteomic, glycomic, and imaging biomarkers can be used for the diagnosis, prognosis, and epidemiology of cancer. Such biomarkers can be assayed in non-invasively collected biological fluids, such as blood or serum. Several gene- and protein-based biomarkers have been used in patient care, including but not limited to AFP (liver cancer), BCR-ABL (chronic myeloid leukemia), BRCA1 / BRCA2 (breast / ovarian cancer), BRAF V600E (melanoma / colorectal cancer), CA-125 (ovarian cancer), CA19.9 (pancreatic cancer), CEA (colorectal cancer), EGFR (non-small cell lung cancer), HER-2 (breast cancer), KIT (gastrointestinal stromal tumor), PSA (prostate specific antigen), S100 (melanoma), etc. Biomarkers can be used as diagnostic (identifying early stage cancer) and / or prognostic (predicting the aggressiveness of a cancer and / or predicting how a subject will respond to a particular treatment and / or the likelihood of recurrence of a cancer).

[0056] As used herein, the term "image", "image scan" or "scanned image" encompasses raw image data acquired from a biological tissue sample, e.g. by an optical sensor or sensor array, or pre-processed image data. In particular, the image data can comprise a matrix of pixels.

[0057] As used herein, the term "image data" encompasses raw image data acquired from a biological tissue sample, e.g. by an optical sensor or sensor array, or pre-processed image data. In particular, the image data can comprise a matrix of pixels.

[0058] As used herein, the term "immunohistochemistry" refers to a method for determining the presence or distribution of an antigen in a sample by detecting the interaction of the antigen with a specific binding agent, such as an antibody. The sample is contacted with the antibody under conditions that allow for antibody-antigen binding. The antibody-antigen binding can be detected by means of a detectable label conjugated to the antibody (direct detection) or by means of a secondary antibody conjugated to a detectable label that specifically binds to the primary antibody (indirect detection).

[0059] As used herein, the term "multi-channel image" or "multi-path image" encompasses digital images obtained from biological tissue samples in which different biological structures of cell nuclei, cells, and tissue structures are simultaneously stained using specific fluorescent dyes, quantum dots, chromogens, and the like, each of which emits fluorescence or can otherwise be detected in different spectral bands, constituting one of the channels of the multi-channel image.

[0060] As used herein, the term "slide" refers to any substrate (e.g., a substrate made entirely or in part of glass, quartz, plastic, silicon, and the like) of any suitable size on which a biological specimen can be placed for analysis, and more particularly refers to a "microscope slide" such as a standard 3 x 1 inch microscope slide or a standard 75 mm x 25 mm microscope slide. Examples of biological specimens that can be placed on a slide include, but are not limited to, cytological smears, thin tissue sections (e.g., from a biopsy), and biological specimen arrays such as tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof. Thus, in one embodiment, a tissue section, a DNA sample, an RNA sample, and / or a protein is placed on a slide at a particular location. In some embodiments, the term "slide" can refer to SELDI and MALDI chips, as well as silicon wafers.

[0061] SUMMARY

[0062] Despite recent successes, modern computer vision techniques (e.g., convolutional neural networks (CNNs)) applied to cell-level prediction problems in cancer histology images are costly due to the difficulty of providing cell-level supervision. The present disclosure provides systems and methods that facilitate true value labeling of cell classifications by detecting cells in large images of biological specimens in certain embodiments using deep learning, and using learned representations of the detected cells to cluster them into homogeneous subsets, reducing the burden of true value labeling in order from the number of cells in a dataset to the number of clusters identified in a dataset. In certain embodiments, the present disclosure provides systems and methods that use models trained on small labeled image patches to detect cells in images of biological specimens (e.g., histology images, cytology images), and wherein the systems and methods further utilize neural networks representing the detected cells to cluster and / or classify them. Applicants surprisingly found that the proposed systems and methods can achieve superior cell detection and classification performance even using small training image sets, and can even generalize to unseen stain types. In fact, Applicants demonstrated the ability of the developed models to detect cells in datasets of mixed sizes and resolutions, and demonstrated the ability of the developed models to generalize to stain types not seen during training.

[0063] Accordingly, embodiments of the unified deep learning classification and detection framework described herein have a number of advantages over previously used methods. For example, embodiments described herein have a number of "ease-of-use" and cost advantages. In one such example, embodiments described herein significantly reduce the burden on users to label cells at the pixel level for detection network learning. In other words, embodiments described herein enable bounding box labeling for detection, which significantly reduces the user labeling burden. In this way, embodiments provide "ease-of-use" for labeling, training, and testing.

[0064] In certain embodiments, the present disclosure describes systems and methods for adjusting a pre-trained neural network (e.g., a convolutional neural network) using a plurality of training images to provide a tuned convolutional neural network. In certain embodiments, the training images used to adjust the pre-trained network are images of biological specimens stained for the presence of one or more biomarkers. In certain embodiments, the training images contain one or more class labels. In other embodiments, the training images do not contain class labels. The present disclosure also provides a trained object detection engine that contains a tuned neural network and a region proposal network. As described herein, the trained object detection engine can be used to detect cells.

[0065] In certain embodiments, the present disclosure relates to systems and methods for detecting cells (and optionally classifying cells) in sample images using a trained object detection engine. In certain embodiments, the systems and methods described herein facilitate clustering of detected cells in sample images, thereby providing one or more homogeneous clusters of detected cells in each sample image. The homogeneous clusters of detected cells identified in each sample image can be further annotated, for example, by a pathologist, and in certain embodiments, such annotations can be used to train a classifier or neural network. In other words, the annotated homogeneous clusters can be used as ground truth data.

[0066] In certain embodiments, the present disclosure provides automated systems and methods for detecting and / or classifying cells within test images (e.g., unlabeled images derived from biological specimens originating from a subject (e.g., a human patient)) using a cell detection and classification engine, wherein the cell detection and classification engine is trained using ground truth data. In certain embodiments, the ground truth data is derived from pathologist annotations of homogeneous clusters of detected cells in one or more sample images. In certain embodiments, the cell detection and classification engine contains a neural network, e.g., a different neural network than the neural network contained in the object detection engine described above.

[0067] At least certain embodiments of the present disclosure relate to computer systems and methods for analyzing digital images captured from biological specimens, including tissue samples, that are stained with one or more primary stains, such as hematoxylin and eosin (H&E), and one or more detection probes, such as probes containing specific binding entities that facilitate labeling of targets within the sample. Although examples herein can refer to specific tissues and / or apply specific stains or detection probes for detecting specific markers, one skilled in the art will recognize that different tissues and different stains / detection probes can be applied to detect different markers.

[0068] In FIG. 1 and FIGS. 2A-2C A digital pathology system 200 for imaging and analyzing a sample is shown in FIG. 1. The digital pathology system 200 can include an imaging device 12, such as a device having a means for scanning a microscope slide bearing a specimen, and a computer 14, whereby the imaging device 12 and computer can be communicatively coupled together, such as directly, or indirectly through a network 20. The computer system 14 can include a desktop computer, a laptop, a tablet or the like, digital electronic circuitry, firmware, hardware, memory 201, computer storage media (240), a computer program or set of instructions (such as the program stored within the memory or storage medium), one or more processors (209) including a programmed processor, and any other hardware, software, or firmware modules or combinations thereof (as further described herein). For example, the FIG. 1 The computing system 14 shown in FIG. 1 can include a computer having a display device 16 and a housing 18. The computer system can store digital images in binary form (stored locally, such as in memory, on a server or another network-connected device). The digital images can also be divided into a matrix of pixels. The pixels can include one or more bits of digital value defined by a bit depth. One skilled in the art will recognize that other computer devices or systems can be utilized, and that the computer system described herein can be communicatively coupled with additional components, such as a specimen analyzer, a microscope, other imaging systems, automated slide preparation devices, and the like. Further description of some of these additional components, as well as various available computers, networks, and the like, are described herein.

[0069] In general, the imaging device 12 (or other image source including pre-scanned images stored in memory) can include, but is not limited to, one or more image capture devices. The image capture device can include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), optics (e.g., one or more lenses, a sensor focusing lens group, a microscope objective, etc.), an imaging sensor (e.g., a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.), photographic film, etc. In digital embodiments, the image capture device can include multiple lenses that can cooperate to prove instant focus capabilities. An image sensor, e.g., a CCD sensor, can capture a digital image of the specimen. In some embodiments, the imaging device 12 is a brightfield imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. The digitized tissue data can be generated, for example, by an image scanning system, such as a VENTANA DP200 scanner by VENTANA MEDICAL SYSTEMS, Inc. (Tucson, Arizona) or other suitable imaging device. Other imaging devices and systems will be further described herein. Those skilled in the art will recognize that a digital color image acquired by the imaging device 12 is generally composed of elemental color pixels. Each color pixel can be encoded on three digital components, each component containing the same number of bits, and each component corresponding to one of the primary colors, typically red, green, or blue, also referred to by the term "RGB" components.

[0070] FIGS. 2A-2C An overview of the systems 200 and 250 of the present disclosure and the various modules used within each system is provided. In certain embodiments, both digital pathology systems 200 and 250 employ a computer device or computer-implemented method having one or more processors 209 and one or more memories 201 storing non-transitory computer-readable instructions for execution by the one or more processors to cause the one or more processors to perform certain instructions as described herein.

[0071] FIG. 2A And 2B The systems 200 depicted in FIGS. 1-2 each include an object detection engine 210 that can be trained to detect cells (and optionally classify cells) in sample images. The systems 200 also include components that enable homogenous clusters of detected cells to be identified. In certain embodiments, the object detection engine includes a neural network 212. In certain embodiments, the object detection engine 210 includes a first portion that includes a neural network 212 and a second portion that includes a region candidate network 214 (see FIG. 2B). In certain embodiments, the object detection engine 210 comprises an architecture derived from a region-based convolutional neural network. In certain embodiments, the neural network 212 can operate independently of the region proposal network 214. For example, the neural network 212 can be trained as described herein (see also FIG. 4 ). After training the neural network 212, the trained neural network 212 can be used in conjunction with the region proposal network 214 for detecting and / or classifying cells in an input image.

[0072] In certain embodiments, the system is adapted to detect and optionally classify cells in sample images. With reference to FIG. 2A , 2B , 3A and FIG. 3B , in certain embodiments, one or more sample images are received from the imaging device 12 or the storage module 240. In certain embodiments, the sample images are stained with a primary stain or stained for the presence of one or more biomarkers, such as hematoxylin, DAPI, DAB, etc. In certain embodiments, the object detection engine 210 is used to detect and optionally classify cells within the received sample images (step 310). In certain embodiments, the object detection engine comprises a neural network 212, such as a convolutional neural network and a region proposal network 214. In certain embodiments, the neural network 212 is used to identify cell features in the received sample images (step 340). In certain embodiments, the region proposal network 214 is used to detect cells in the received sample images based on the cell features identified by the neural network 212 (step 341). After the cells are detected, the feature extraction module 202 is used to extract at least a portion of the identified cell features from at least one layer of the network 212 (step 311 or 341). Based on the extracted cell features, the clustering module 204 can then be used to generate homogenous clusters of cells (step 312 or 342). The generated homogenous clusters can then be annotated, for example, by a pathologist using the annotation module 206. In certain embodiments, the pathologist annotations can be used as ground truth data, where the ground truth data can be used to train a classifier (such as a support vector machine), or another separate neural network (where the separate neural network can be the same or different type as in the object detection engine 212); or can be used to fine-tune the trained neural network 212 of the object detection engine 210.

[0073] FIG. 2CAn overview of modules for detecting and / or classifying cells in test images in system 250 is provided. In certain embodiments, system 250 includes a cell detection and classification module 255. In certain embodiments, the cell detection and classification module comprises a neural network. In certain embodiments, the cell detection and classification module comprises a classifier, e.g., a support vector machine. In embodiments where the cell detection and classification module comprises a neural network, the neural network can be the same as or different from neural network 212 of object detection engine 210. In certain embodiments, system 250 further includes a scoring module to score the detected and / or classified cells, e.g., to determine a H-score or a percent positivity.

[0074] A skilled artisan will also appreciate that additional modules can be incorporated into the workflow or into systems 200 and 250. In certain embodiments, an image processing or pre-processing module can be run to apply certain filters to the acquired images or to identify certain histological and / or morphological structures or features within the tissue sample. Likewise, a region of interest selection module can be utilized to select particular portions of the sample image or test for analysis. Additionally, a de-mixing module can be run to de-mix the multi-channel images prior to further processing.

[0075] Image acquisition module

[0076] With reference to FIGS. 2A-2C In certain embodiments, digital pathology systems 200 and 250 run an image acquisition module 202 to capture images of a biological specimen (e.g., using imaging device 12), including a biological specimen having one or more stains, e.g., a primary stain or a stain indicating the presence of a biomarker. In certain embodiments, the images are single-channel images. In other embodiments, the images are multi-channel images. In certain embodiments, the received or acquired images are RGB images or multi-spectral images. In certain embodiments, the captured images are stored in memory 201 (or within storage module 240).

[0077] In certain embodiments, the images acquired or stored in the storage module 240 are“training images.” In certain embodiments, the training images are images of biological specimens, and can be histological specimens or cytological specimens, or any combination thereof. In certain embodiments, the training images are used to adjust a pre-trained neural network. In this regard, the training images are used for training purposes. In certain embodiments, the training images include at least one stain. In certain embodiments, the training images include multiple stains. In certain embodiments, the training images include one or more class labels. In certain embodiments, multiple training images are used to adjust a pre-trained neural network. In certain embodiments, the training images can be part of a dataset that includes multiple training images, where each of the multiple training images can include the same or different stains, and where each of the multiple training images can include class labels.

[0078] In certain embodiments, the images acquired or stored in the storage module 240 are“sample images.” In certain embodiments, the sample images are images of biological specimens, and can be histological specimens or cytological specimens. In certain embodiments, the sample images are provided to a trained neural network, such that cells can be detected (and optionally classified), and such that detected cells can be clustered (see, e.g., FIG. 7 ). Ground truth data can ultimately be derived from the sample images, i.e., the ground truth data includes pathologist annotations of homogenous clusters of detected cells generated in the sample images. The generated ground truth data can be used to train a classifier or neural network.

[0079] In certain embodiments, the images acquired or stored in the storage module 240 are“test images.” In certain embodiments, the test images are images of biological specimens, and can be histological specimens or cytological specimens. The test images are provided to a trained classifier or neural network, e.g., a classifier or neural network that has been trained using generated ground truth data, including pathologist annotations, as described above. In this manner, cells within the test images can be detected and / or classified. The test images can be considered to be images of a sample derived from a subject (e.g., a human patient), where a pathological impression or diagnosis can be reached based on the detection and / or classification of cells (or other cellular features) within the test images.

[0080] Any of the training images, sample images, or test images (collectively referred to herein as "input images") can be an entire slide image or any portion thereof (e.g., a predetermined field of view). Further, and regardless of whether the image is a training image, a sample image, or a test image, the input image can be acquired by using any imaging device. The input images can be acquired using imaging device 12, such as in real-time. In certain embodiments, the input images are acquired from a microscope or other instrument capable of capturing image data of a microscope slide bearing a specimen, as described herein. In certain embodiments, the input images are acquired using a two-dimensional scanner, such as a scanner capable of scanning an image tile, or a line-type scanner, such as a VENTANA DP200 scanner, capable of scanning an image line-by-line.

[0081] The camera platform can also include a brightfield microscope, one example of which is the VENTANA iScan HT product by Ventana Medical Systems, Inc., or any microscope having one or more objectives and a digital imager and a set of spectral filters. Other techniques can be used to capture images at different wavelengths. Further, camera platforms suitable for imaging stained biological specimens are known in the art and are commercially available from companies such as Zeiss, Canon, Applied Spectral Imaging, and the like, and such platforms can be readily adapted for use in the systems, methods, and apparatus of the present subject disclosure.

[0082] In general, the input images each contain signals corresponding to stains, including chromogens, fluorophores, quantum dots, and the like. In certain embodiments, the input images have been stained with at least one primary stain (hematoxylin or eosin). In other embodiments, at least one image has been stained in at least one of an IHC assay or an ISH assay for identifying specific biomarkers (referred to herein as "biomarker" images). In certain embodiments, multiple single-channel images are used as input, and those images can be derived from consecutive tissue sections, i.e., from consecutive sections in the same xenograft tissue block. In certain embodiments, the input images can be multi-channel images, i.e., stained for multiple different markers in a multi-channel assay according to methods known to those of ordinary skill in the art.

[0083] Typical biological specimens are processed in a staining / assay platform that stains the sample. In certain embodiments, the specimen processing apparatus is an automated apparatus, such as the BENCHMARK XT instrument, the SYMPHONY instrument, and the BENCHMARK ULTRA instrument sold by Ventana Medical Systems, Inc. Ventana Medical Systems, Inc. is the assignee of a number of U.S. patents that disclose systems and methods for performing automated analysis, including U.S. Patent Nos. 5,650,327; 5,654,200; 6,296,809; 6,352,861; 6,827,901; and 6,943,029, as well as U.S. Published Patent Application Nos. 20030211630 and 20040052685, the disclosure of each of which is incorporated herein by reference in its entirety. Alternatively, the specimens can be processed manually.

[0084] Examples of commercially available H&E stainers include the VENTANA SYMPHONY (single slide stainer) and VENTANA HE 600 (single slide stainer) series H&E stainers from Roche; the Dako Cover Stainer (batch stainer) from Agilent Technologies; the Leica ST4020 compact linear stainer (batch stainer), the Leica ST5020 multifunctional stainer (batch stainer), and the Leica ST5010 Autostainer XL series (batch stainer) H&E stainers from Leica Biosystems Nussloch GmbH. Other commercial products on the market suitable for use as a staining / assay platform are the Discovery ULTRA (batch stainer) and the Discovery ULTRA LT (batch stainer) from Ventana Medical Systems, Inc. (Tucson, AZ). The Discovery ULTRA and the Discovery ULTRA LT are batch stainers that are capable of processing up to 48 slides at a time. The Discovery ULTRA LT is a smaller version of the Discovery ULTRA that is capable of processing up to 24 slides at a time. TM The Ventana Medical Systems, Inc. product is one example.

[0085] As will be appreciated by the skilled artisan, staining of biological specimens can be performed for different types of and / or cellular membrane biomarkers. Methods of staining for tissue architecture and guidelines for selecting stains appropriate for various purposes are discussed, for example, in "Sambrook et al., Molecular Cloning: A Laboratory Manual, Cold Spring Harbor Laboratory Press (1989)" and "Ausubel et al., Current Protocols in Molecular Biology, Greene Publishing Associates and Wiley-Intersciences (1987)," the disclosures of which are incorporated herein by reference.

[0086] As one non-limiting example, and in the case of detecting breast cancer, in certain embodiments, a tissue sample is stained for the presence of one or more biomarkers, including an estrogen receptor marker, a progesterone receptor marker, a Ki-67 marker, or a HER2 marker, in an IHC assay. As such, in certain embodiments, the biomarker images used as input are IHC images that include signals corresponding to the presence of at least one of an estrogen receptor (ER) marker, a progesterone receptor (PR) marker, a Ki-67 marker, or a HER2 marker (corresponding to signals of a stain that can be, for example, chromogenic or fluorescent). In certain embodiments, a sample can be analyzed to detect or measure the presence of ER, HER2, Ki-67, and PR proteins in the sample, for example, qualitatively or quantitatively. As another non-limiting example, and in the case of detecting non-small cell lung cancer, in certain embodiments, a tissue sample is stained for the presence of one or more biomarkers, including a PD-L1 biomarker, in an IHC assay. As such, in certain embodiments, the biomarker images used as input are IHC images that include signals corresponding to the presence of a PD-L1 marker, a CD3 marker, and a CD8 marker.

[0087] Whether the collected images are used as training images or for ground truth sample images; or as test images for cell detection and / or classification, the input images can include information that needs to be masked. In certain embodiments, the input images are optionally masked by the tissue masking module described herein. In some embodiments, the input images are masked such that only tissue regions are present in the images. In some embodiments, a tissue region mask is generated to mask non-tissue regions from the tissue regions. In some embodiments, the tissue region mask can be created by identifying the tissue regions and automatically or semi-automatically (i.e. with minimal user input) excluding background regions (such as the entire section image area corresponding to the sample-free glass, e.g. only regions from the white light of the imaging source) from the tissue regions.

[0088] Those skilled in the art will recognize that in addition to masking non-tissue regions from the tissue regions, the tissue masking module can also mask other regions of interest as needed, such as tissue identified as belonging to a certain tissue type or as part of a suspected tumor region. In some embodiments, a tissue region mask image is generated by masking tissue regions from non-tissue regions in the input images using segmentation techniques. Again, suitable segmentation techniques are also known in the art, (see Digital Image Processing, Third Edition, Rafael C. Gonzalez, Richard E. Woods, Chapter 10, page 689 and Handbook of Medical Imaging, Processing and Analysis, Isaac N. Bankman Academic Press, 2000, Chapter 2). Additional information and examples relating to the generation of tissue region masks are disclosed in PCT / EP / 2015 / 062015 entitled "An Image Processing Method and System for Analyzing a Multi-Channel Image Obtained from a Biological Tissue Sample Being Stained by Multiple Stains", the disclosure of which is hereby incorporated by reference in its entirety.

[0089] Object detection engine

[0090] The system 200 of the present disclosure includes an object detection engine 210 that includes at least a neural network 212. In certain embodiments, the object detection engine 210 includes a first portion that includes a neural network 212 (including any of the neural networks described below) and a region proposal network 214. In certain embodiments, the trained object detection engine 210, i.e., the engine containing the trained neural network 212, is used to detect and / or classify cells in a sample image such that homogenous clusters of cells can be identified in the sample image. In certain embodiments, the neural network 212 is configured to determine features and generate a feature map for a sample image. In certain embodiments, the object detection engine 210 includes an architecture derived from a region-based convolutional neural network as described below.

[0091] Object detection is the process of finding and classifying objects in an image. One deep learning method, the region-based convolutional neural network (RCNN), combines rectangular region proposals with convolutional neural network features (described herein). RCNN is a two-stage detection algorithm. The first stage identifies a subset of regions in an image that can contain objects. The second stage classifies the objects in each region. Models that use regions with CNNs for object detection are based on three processes: (a) finding regions in an image that can contain objects. These regions are called region proposals; (b) extracting CNN features from the region proposals; and (c) classifying the objects using the extracted features.

[0092] There are several variants of RCNN. Each variant attempts to optimize, speed up, or enhance the results of one or more of these processes. The RCNN detector (Girshick, R., J. Donahue, T. Darrell, and J. Malik. "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation." In CVPR'14 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Pages 580-587. 2014) first generates region proposals using an algorithm such as Edge Box (Zitnick, C. Lawrence, and P. Dollar). "Edge boxes: Locating object proposals from edges." In Computer Vision-ECCV. Springer International Publishing. Pages 391-4050. 2014). The proposed regions are cropped from the image and resized. Then, a CNN classifies the cropped and resized regions. Finally, the region proposal bounding boxes are refined by a support vector machine (SVM) trained using the CNN features. FIG. 6AThe process of region detection, computation of CNN features, and region classification is shown.

[0093] Like in the RCNN detector, the Fast RCNN detector (Girshick, Ross. "Fast RCNN." Proceedings of the IEEE International Conference on Computer Vision. 2015) also uses an algorithm like "Edge Box" to generate region proposals. Unlike the RCNN detector, which can crop region proposals and resize them, the Fast RCNN detector processes the entire image. While the RCNN detector must classify each region, the Fast RCNN instead pools the CNN features corresponding to each region proposal. Fast RCNN is more efficient than RCNN because the computation of overlapping regions is shared in the Fast RCNN detector. FIG. 6B Detection using convolutional feature maps and region of interest (ROI) feature vectors is shown.

[0094] The Faster RCNN detector is described by Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun. "Faster RCNN: Towards Real-Time Object Detection with Region Proposal Networks." Advances in Neural Information Processing Systems, vol. 28, 2015. Instead of using an external algorithm like Edge Box, the Faster RCNN adds a region proposal network 214 to generate region proposals directly in the network. The region proposal network uses Anchor Boxes for object detection. It is believed that generating region proposals in the network can more quickly and efficiently detect cells in sample images, especially when considering the size of the input image provided to the object detection engine 210. In certain embodiments, the region proposal network and the classification network share the same feature extraction layers. FIG. 6C An object detection engine 210 based on the Faster-RCNN architecture is shown, which includes multiple convolutional layers and a region proposal network.

[0095] Mask-RCNN extends Faster R-CNN by adding a branch for predicting object masks in parallel with the existing branch for bounding box detection. Mask-RCNN refines the step of ROI pooling. In the ROI, the warping is digitized and the cell boundaries of the target feature map are forced to re-align with the boundaries of the input feature map. The ROI Align used by Mask-RCNN does not digitize the cell boundaries, but instead makes each target cell have the same size. It also applies interpolation to compute the feature map values.

[0096] Particular implementations of RCNNs, including Fast-RCNN and Faster-RCNN, are described in U.S. Publication Nos. 2018 / 0268292, 2018 / 0018535, and 2016 / 034288, the disclosures of each of which are incorporated herein by reference in their entirety. Other particular implementations of RCNNs, including Fast-RCNN and Faster-RCNN, are described in U.S. Patent No. 9,971,953, the disclosure of which is incorporated by reference herein in its entirety. Any of these implementations can be adapted for detecting and / or classifying cells.

[0097] A region candidate network 214 can generally be defined as a fully convolutional network that detects objects in an image and proposes regions corresponding to those detected objects. Candidate networks are attractive for use in the embodiments described herein because they are relatively fast and are able to detect objects of different sizes. A candidate network can cover a sliding window over a feature map generated by one or more convolutional layers that generate k anchor boxes. The result of covering a sliding window over a feature map can be input to an intermediate layer of the candidate network, which can generate 2k scores via a cls layer and 4k coordinates via a reg layer (see Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.” Advances in Neural Information Processing Systems 28, 2015).

[0098] In certain embodiments, the region candidate network 214 is configured for cell detection. In other words, the region candidate network 214 uses features from the generated feature map based on determined features to detect cells in a sample image. In certain embodiments, the region candidate network 214 is configured to generate a bounding box detection result.

[0099] Neural network

[0100] In certain embodiments, the present disclosure is directed to training a neural network 212, such as a convolutional neural network, and then utilizing the trained neural network to detect and / or classify cells in an input image. Suitable neural networks that can be trained and / or used in the detection and / or classification of cells are described below.

[0101] In certain embodiments, the neural network 212 is configured as a deep learning network. Generally, "deep learning" is a branch of machine learning that is based on a set of algorithms that attempt to model high-level abstractions in data. Deep learning is part of a broader family of machine learning methods based on learning representations of data. An observation (e.g., an input image) can be represented in many ways such as a vector of intensity values for each pixel, or in a more abstract way as a set of edges, regions of particular shapes, etc. Certain representations are better than others at simplifying the learning task. One of the prospects of deep learning is to replace handcrafted features with efficient algorithms to enable unsupervised or semi-supervised feature learning and hierarchical feature extraction.

[0102] In certain embodiments, the neural network 212 is a machine learning network. Machine learning can generally be defined as a type of artificial intelligence (AI) that gives computers the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data. In other words, machine learning can be defined as a subfield of computer science that gives computers the ability to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms that can learn from data to make predictions or decisions, overcoming the problems of strictly following static program instructions by establishing models from sample inputs. Machine learning described herein can be further configured as described in "Introduction to Statistics Machine Learning," by Sugiyama, Morgan Kaufmann, 2016, 534 pages; "Discriminative, Generative, and Imitative Learning," Jebara, MIT Thesis, 2002, 212 pages; and "Principles of Data Mining (Adaptive Computation and Machine Learning)," Hand et al., MIT Press, 2001, 578 pages; which are incorporated by reference as if fully set forth herein. Embodiments described herein can be further configured as described in these references.

[0103] In certain embodiments, the neural network 212 is a generative network. A "generative" network can generally be defined as a model that is probabilistic in nature. In other words, a "generative" network is not a network that performs forward simulation or rule-based methods. Rather, a generative network can be learned (in that its parameters can be learned) based on a suitable training dataset (e.g., from a plurality of training images, such as training images of biological specimens stained for the presence of one or more biomarkers). In certain embodiments, the neural network is configured as a deep generative network. For example, the network can be configured to have a deep learning architecture in that the network can include multiple layers that perform a number of algorithms or transformations.

[0104] In certain embodiments, the neural network 212 can be a deep neural network having a set of weights that model the world according to data that has been fed to train the world, as described herein. Neural networks are generally composed of multiple layers, and the signal path traverses from front to back between layers. Any neural network can be implemented for this purpose. Suitable neural networks include LeNet, proposed by Yann LeCun; AlexNet, proposed by Alex Krizhevsky et al.; ZF Net, proposed by Matthew Zeiler et al.; GoogLeNet, proposed by Szegedy et al.; VGGNet, proposed by Karen Simonyan et al.; and ResNet, proposed by Kaiming He et al. In certain embodiments, the neural network is VGG16 (Simonyan, 2014). In other embodiments, the multi-layer neural network is a DenseNet (see Huang et al., "Densely Connected Convolutional Networks," arXiv: 1608.06993). In certain embodiments, a fully convolutional neural network is utilized, such as described by Long et al., "Fully Convolutional Networks for Semantic Segmentation," Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference, June 2015 (INSPEC accession number: 15524435), the disclosure of which is hereby incorporated by reference herein.

[0105] In certain embodiments, the neural network 212 is configured as AlexNet. For example, the classification network structure can be AlexNet. The term "classification network" is used herein to refer to a CNN that includes one or more fully connected layers. Generally, AlexNet includes multiple convolutional layers (e.g., 5), followed by multiple fully connected layers (e.g., 3) configured and trained in a combined manner for classifying images. An example of a neural network configured as AlexNet is described in "ImageNet Classification with Deep Convolutional Neural Networks," by Krizhevsky et al., NIPS 2012, which is incorporated by reference as if fully set forth herein. The neural networks described herein can be further configured as described in that reference.

[0106] In other embodiments, the neural network 212 is configured as GoogleNet. For example, the classification network structure can be GoogleNet. GoogleNet can include layers such as convolutional layers, pooling layers, and fully connected layers, such as those described further herein, configured and trained for classifying images. Although the GoogleNet architecture can include a relatively large number of layers (particularly as compared to some other neural networks described herein), some of the layers can operate in parallel, and groups of layers operating in parallel to each other are generally referred to as inception modules. Other layers can operate sequentially. Thus, GoogleNet differs from some other neural networks described herein in that not all of the layers are arranged in a sequential structure. An example of a neural network configured as GoogleNet is described in "Going Deeper with Convolutions," by Szegedy et al., CVPR 2015, which is incorporated by reference as if fully set forth herein. The neural networks described herein can be further configured as described in that reference.

[0107] In other embodiments, the neural network 212 is configured as a VGG network. For example, the classification network structure can be VGG. VGG networks were created by increasing the number of convolutional layers while fixing other parameters of the architecture. By using a basic small convolutional filter in all layers, convolutional layers can be added to increase the depth. Like other neural networks described herein, VGG networks have been created and trained to classify images. VGG networks also contain convolutional layers followed by fully connected layers. An example of a neural network configured as VGG is described in "Very Deep Convolutional Networks for Large-Scale Image Recognition," Simonyan et al., ICLR 2015, the disclosure of which is incorporated by reference as if fully set forth herein. The neural networks described herein can be further configured as described in this reference.

[0108] In certain embodiments, the neural network 212 contains an autoencoder. An autoencoder neural network is an unsupervised learning algorithm that applies backpropagation with the target values set equal to the input values. The purpose of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction, by training the network to ignore signal "noise". Along with the simplification side, a reconstruction side is learned, where the autoencoder attempts to generate from a simplified encoding a representation as close as possible to its original input. Additional information regarding autoencoders can be found at http: / / ufldl.stanford.edu / tutorial / unsupervised / Autoencoders / , the disclosure of which is hereby incorporated by reference in its entirety.

[0109] In other embodiments, the neural network 212 is configured as a deep residual network. For example, the classification network structure can be a deep residual network or ResNet. Like some other networks described herein, a deep residual network can include convolutional layers followed by fully connected layers that are configured and trained in a combined manner for image classification. In a deep residual network, the layers are configured to learn a residual function with respect to the layer inputs, rather than learning an unreferenced function. In particular, rather than expecting each few-stacked layer to fit directly to the desired base mapping, these layers are explicitly allowed to fit to a residual mapping, which is achieved through a feed-forward neural network with shortcut connections. A shortcut connection is a connection that skips one or more layers. A deep residual network can be created by taking a normal neural network structure that includes convolutional layers and inserting shortcut connections, which thereby takes the normal neural network and converts it into a residual learning replica. An example of a deep residual network is described in He et al., Deep Residual Learning for Image Recognition, NIPS 2015, which is incorporated by reference as if fully set forth herein. The neural networks described herein can be further configured as described in this reference.

[0110] Adjusting a pre-trained neural network

[0111] In certain embodiments, the neural network 212 of the object detection engine 210 is first trained, such as using a plurality of training images or using one or more data sets containing a plurality of training images. The neural network is typically trained on large and publicly available image data archives of natural images (such as ImageNet, Russakovsky et al., 2015, ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015). Given the small number of training images typically available for biological specimens (e.g., tumor samples), a transfer learning approach is employed.

[0112] In certain embodiments, a "transfer learning" approach can be used, in which a pre-trained network such as those described above, along with its learned weights, is used to initialize the training process on a new set of training images, and trained to adapt to the weights of the specific application at hand (see Pan, S. J., & Yang, Q. (2010). A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345-1359. doi:10.1109 / tkde.2009.191, the disclosure of which is hereby incorporated by reference in its entirety). In certain embodiments, the pre-trained network is trained on an existing public or private dataset that can or can not be directly related to the target application and can or can not be biological in nature. In certain embodiments, the weights of the final layers are randomly initialized to avoid getting stuck in suboptimal local minima, while in other embodiments, the weights of the final layers are imported unaltered.

[0113] In certain embodiments, a pre-trained neural network is obtained (step 410). In certain embodiments, the pre-trained neural network is a neural network that has been trained using a database comprising a dataset containing a large number of images (e.g., more than 100,000 images, more than 200,000 images, or more than 300,000 images). In certain embodiments, the pre-trained neural network is a neural network that has been pre-trained using the Microsoft Coco dataset (Tsung-Yi Lin, et al., “Microsoft coco: Common objects in context,” in European conference on computer vision, Springer, 2014, pp. 740-755). In other embodiments, the pre-trained neural network is a neural network that has been pre-trained using the ImageNet dataset (J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR, 2009), the PASCAL VOC (M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The PASCAL visual object classes (VOC) challenge,” IJCV, vol. 88, no. 2, pp. 303-338, June 2010), or the SUN dataset (J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “SUN database: Large-scale scene recognition from abbey to zoo,” in CVPR, 2010).

[0114] The Microsoft Common Objects in Context (MS COCO) dataset contains 91 classes of common objects, 82 of which have more than 5,000 labeled instances. The dataset has a total of 2,500,000 labeled instances across 328,000 images. Compared to the ImageNet dataset, COCO has fewer classes but more instances per class. It is believed that this can help learn detailed object models that are capable of precise 2D localization. Compared to the PASCAL VOC and SUN datasets, the number of instances per class of the dataset is also significantly larger. It is believed that MS COCO contains comparable number of object instances per image (7.7) compared to ImageNet (3.0) and PASCAL (2.3). In contrast, the SUN dataset, which contains significant contextual information, has over 17 objects and is “padded” per image, but overall has significantly fewer object instances. U.S. Pub. No. 2019 / 0073568 further describes methods of training neural networks, the disclosure of which is hereby incorporated by reference in its entirety.

[0115] After obtaining the pre-trained neural network, a plurality of training images are obtained (step 411), which can be used to adjust the pre-trained neural network (step 412). To familiarize the pre-trained model with unique neural networks having unique characteristics of the biological specimen images, in certain embodiments, the pre-trained neural network is adjusted using training images of biological specimens that are stained with a primary stain or stained for the presence of one or more biomarkers. By way of example only, five different training image datasets are listed in Table 1. In certain embodiments, the training images of biological specimens are annotated. In other embodiments, the training images of biological specimens are not annotated, i.e., they do not contain any class labels. In certain embodiments, each training image can have the same resolution or different resolutions. In certain embodiments, each training image can have the same magnification or different magnifications (e.g., 10x training images, 20x training images, 40x training images, etc.). In certain embodiments, the training images are thousands of pixels in each dimension. In certain embodiments, to enable annotation, image patches are extracted randomly, e.g., image patches having a size of 64x64 pixels or 128x128 pixels.

[0116] Table 1: Examples of suitable training image datasets.

[0117] Dataset ]]> ​ <![CDATA[ Number of image tiles ]]> Number of cells ]]> ​ Image tile size ]]> ​ Resolution ]]> ​ Her2 118 560 64x64 pixels 20x Dual ISH 143 1296 128x128 pixels 40x Combined 268 1822 Mixed Mixed BB 200 7162 128x128 pixels 20x BBYP 104 1340 128x128 pixels 20x

[0118] Additional information about each of the datasets listed in Table 1 is as follows:

[0119] Her2 : 560 annotations of cells with Her2 membrane staining.

[0120] Dual ISH : 1269 annotations of cells double in situ hybridization stained.

[0121] Combined : Combination of Her2 and Dual ISH datasets.

[0122] BB : 1822 two-class annotations of cells stained with hematoxylin and DAB.

[0123] BBYP : Images of IHC Duplex stained contain 1340 annotations divided in 5 unbalanced classes with number of members ranging from 10 to 1828.

[0124] For the "Her2", "Dual ISH" and "Combined" datasets, the annotations are bounding boxes of class-agnostic cells; for the BB dataset, the annotations are bounding boxes with "blue" or "brown" cell labels; for the BBYP dataset, the annotations are bounding boxes with "blue tumor cell", "purple tumor cell", "yellow T cell", "purple T cell" and "yellow and purple T cell" labels, which correspond to "ki67- tumor", "ki67+ tumor", "cd8+ T cell", "ki67+ T cell" and "cd8+ ki67+ T cell", which are mutually exclusive

[0125] The pre-trained neural networks were adjusted (step 412) using the datasets listed in Table 1, providing the following trained neural networks. These different trained neural networks were used in specific experiments as detailed in the Examples section of this document.

[0126] Model Zero : FasterRCNN model with Resnet-50 feature extractor pre-trained on COCO dataset.

[0127] HER2 : Model Zero adjusted for cell detection on Her2 data.

[0128] DUALISH : Model Zero adjusted for cell detection on Dual ISH data.

[0129] Model A : Model Zero adjusted for cell detection on Combined data.

[0130] Model B : Model Zero adjusted for detection and classification on BB data.

[0131] Model CModel A, which is adjusted with respect to BB data for detection and classification.

[0132] Generating homogenous clusters of detected cells using a trained object detection engine

[0133] After adjusting the pre-trained neural network, such as by using a plurality of training images of biological specimens stained for the presence of one or more biomarkers, the adjusted neural network 212 (used in conjunction with other modules) is used to detect and / or classify cells in sample images such that homogenous clusters of detected cells can be identified in the sample images (see FIG. 3A ). In other embodiments, the adjusted neural network 212 is used in conjunction with the region proposal network 214 (collectively providing the trained object detection engine 210) to detect and / or classify cells in sample images such that homogenous clusters of detected cells can be identified in the sample images (see FIG. 3B ). These sample images can then be annotated, such as by a pathologist. Those skilled in the art will appreciate that if the object detection engine 210 is one that has been trained with training images including one or more class labels, the trained object detection engine 210 can be adapted to classify detected cells.

[0134] Referring to FIG. 3A , in certain embodiments, one or more sample images are obtained and provided to the trained object detection engine 210 such that cells can be detected (step 310). In certain embodiments, the trained object detection engine 210 is derived from a region-based convolutional neural network. In certain embodiments, the RCNN architecture is based on the Fast-RCNN architecture.

[0135] In other embodiments, and referring to FIG. 3B , the neural network 212 is used to identify cell features within one or more sample images (step 340), and then the region proposal network 214 is used to detect cells in the one or more sample images based on the identified cell features (step 341). In other embodiments, the trained object detection engine 210 is derived from Faster-RCNN. In yet other embodiments, the trained object detection engine 210 is based on the Faster-RCNN architecture, and wherein the neural network 212 comprises a deep residual network, such as ResNet.

[0136] In some embodiments, the object detection engine 210 includes a scaling layer. Although the trained object detection engine is trained on small, human-annotated images (64x64 pixels, 128x128 pixels, etc.), the sample images are much larger, for example, 100 times larger. In some embodiments, and instead of the image scaler typically used in one of the region-based convolutional neural network architectures, a scaling layer is introduced into the trained object detection engine 210 that scales the image by a constant factor, independent of size.

[0137] In some embodiments, the object detection engine is trained for a fixed number of steps (e.g., approximately 30,000) using hyperparameters proposed in the Tensorflow Object Detection API. In other embodiments, the maximum number of proposed hyperparameters is increased during the inference process to accommodate larger images. Detection performance is measured using an average precision gauge at a threshold of 0.50 (mAP@50).

[0138] Table 2 provides a summary of the detection (and classification of class labels present in the training images) performance. mAP@50 refers to mean precision, which is the average of the maximum precision at different recall values. When measuring mAP@50, a detection is considered correct if it matches the ground truth labels with an IoU greater than 50%. Here, precision refers to true positive / completely positive detections. IoU measurement is the overlap between the two regions. FIG. 8 Table 2 shows the cell detection results on Her2 and Dual ISHM images using Model A, compared to the ground truth annotations for the corresponding image patches. The models listed in Table 2 are used in the examples below.

[0139] Table 2: Cell detection and classification performance using a trained object detection engine.

[0140]

[0141] Table 2 shows the Faster-RCNN (as described in the examples above) fine-tuned for different staining agent types, performing well for each staining agent type, even in the following cases: for Her2, the staining agent type is a membrane staining agent; or for Dual ISH, the staining agent type is a DNA staining agent; or for BB, the staining agent type is a cell staining agent. It is believed that the performance differences depend on the quality of the ground truth annotations during training and testing.

[0142] As described herein, the neural network 212 can include multiple layers, such as convolutional layers. In certain embodiments, the neural network is designed to determine particular features within one or more sample images. The features determined by the neural network can include any suitable features described further herein or known in the art that can be inferred from the input and used to generate the output described further herein. For example, the features can include a vector of intensity values for each pixel. These features can also include any other type of feature described herein, such as a vector of scalar values, a vector of independent distributions, or any other suitable feature type known in the art.

[0143] In the case of biological specimens, the neural network can determine particular cells or cell features within one or more sample images of the provided biological specimen that are stained for the presence of one or more biomarkers. After cell feature recognition (step 340) and / or detection of cells based on the recognized cell features (step 341), at least a portion of the cell features can be extracted from one or more layers of the neural network 212 using the feature extractor module 202 (steps 311 or 342). In certain embodiments, a deep residual network, such as ResNet or ResNet-50, is used as a feature extractor for the first stage of the region-based convolutional neural network. As a non-limiting example, features from block 1, block 2, block 3, and the convolutional layers preceding block 1 (referred to as block 0) of the ResNet-50 architecture can be extracted and used in downstream clustering operations.

[0144] After extracting at least a portion of the cell features from one or more layers of the neural network, homogenous clusters of the detected cells are generated (steps 312 or 343). Any clustering method can be used for this task. For example, clustering can be performed using a agglomerative clustering technique. In certain embodiments, agglomerative hierarchical clustering is utilized. Agglomerative hierarchical clustering is a bottom-up clustering method in which clusters have sub-clusters, which in turn have sub-clusters, and so on. Agglomerative hierarchical clustering starts with each single object in a single cluster. Then, in each successive iteration, it agglomerates (merges) the closest pair of clusters by satisfying some similarity criterion until all data is in one cluster.

[0145] In certain embodiments, cell features are extracted from different intermediate layers of the trained region-based convolutional neural network and agglomerative clustering is performed as implemented in scikit-learn (see F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011).

[0146] In other embodiments, coalescent clustering can be performed according to any of the methods described in U.S. Patent Nos. 10,162,878; 9,411,632; and 6,625,585; or as in any of U.S. Publication Nos. 2016 / 0342677; 2018 / 0329989; or 2014 / 0359626, the disclosures of which are hereby incorporated by reference in their entireties.

[0147] A pathologist can then use the annotation module 206 so that one or more of the homogeneous clusters of cells detected in each sample image can be labeled. An example of a suitable annotation module is described in U.S. Patent No. 9,818,190, the disclosure of which is hereby incorporated by reference in its entirety.

[0148] The annotation of one or more homogeneous clusters in each sample image can then be used as ground truth data, e.g., for training a cell detection and classification engine (see FIG. 7 ).

[0149] System for detecting and / or classifying cells in a test image

[0150] In certain embodiments, the present disclosure provides a system 250 for detecting and / or classifying cells in test images. FIG. 2C and FIG. 5 An overview of systems and methods for detecting and / or classifying cells in test images is provided. In certain embodiments, the system 250 includes a cell detection and classification engine 255 that has been trained using the ground truth data derived herein (i.e., the pathologist annotation of the homogeneous clusters of cells identified in the sample images). In certain embodiments, the cell detection and classification engine 255 includes a classifier, e.g., a support vector machine or other similar ensemble learning method. Examples of suitable classifiers are further described herein. In embodiments in which the cell detection and classification engine 255 includes a classifier, the cell detection and classification engine 255 can include an automatic cell detection module and / or a neural network (e.g., a neural network adapted for feature extraction).

[0151] In certain embodiments, the cell detection and classification engine 255 comprises a neural network including any of the types of neural networks described herein. In certain embodiments, the cell detection and classification engine 255 includes a region-based convolutional neural network. In those embodiments, the cell detection and classification engine 255 can include an object detection engine 210 that has been further fine-tuned using the generated ground truth data.

[0152] FIG. 5A flowchart showing the various steps of detecting and / or classifying cells in a test image is set forth. In certain embodiments, a trained cell detection and classification engine 5 is first obtained (step 510). In addition, one or more test images are also obtained (step 511). The cell detection and classification engine 255 is then used to detect and / or classify cells in the one or more obtained test images (step 512). After detecting and / or classifying cells in the one or more obtained test images, the test images can be scored (step 513).

[0153] Automated cell detection module

[0154] The programs and automated algorithms described herein can be adapted to identify and classify various types of cells or nuclei based on features within the input images, including identifying and classifying tumor cells, non-tumor cells, stromal cells, and lymphocytes. Those skilled in the art will recognize that the nuclei, cytoplasm, and cell membranes have different features, and that different stained tissue samples can exhibit different biological features. In fact, those skilled in the art will recognize that certain cell surface receptors can have staining patterns that are localized to the cell membrane or cytoplasm. Thus, a "cell membrane" staining pattern is analytically different from a "cytoplasm" staining pattern. Likewise, a "cytoplasm" staining pattern is analytically different from a "nucleus" staining pattern. Each of these different staining patterns can be used as a feature to identify cells and / or nuclei.

[0155] U.S. Patent No. 7,760,927 ("the '927 patent") describes methods of identifying, classifying, and / or scoring nuclei, cell membranes, and cytoplasm in images of biological samples having one or more stains, the disclosure of which is hereby incorporated by reference in its entirety. For example, US 7,760,927 describes an automated method for simultaneously identifying a plurality of pixels in an input image of a biological tissue stained with a biomarker, comprising considering a first color plane of a plurality of pixels in a foreground of the input image, thereby simultaneously identifying cytoplasm and cell membrane pixels, wherein the input image has been processed to remove its background portion and counterstain components; determining a threshold level between cytoplasm and cell membrane pixels in the foreground of the digital image; and using the determined threshold level to simultaneously determine whether a selected pixel is a cytoplasm pixel, a cell membrane pixel, or a transition pixel in the digital image by the selected pixel and its eight neighboring pixels in the foreground.

[0156] US Patent Publication No. 2017 / 0103521 also describes suitable systems and methods for automatically identifying biomarker-positive cells in a biological sample image, the disclosure of which is incorporated by reference herein in its entirety. For example, US 2017 / 0103521 describes (i) reading a first digital image and a second digital image into one or more memories, the first digital image and the second digital image depicting a same region of a first slide, the first slide including a plurality of tumor cells that have been stained with a first stain and a second stain; (ii) identifying a plurality of nuclei and location information of the nuclei by analyzing light intensity in the first digital image; (iii) identifying cell membranes containing the biomarker by analyzing light intensity in the second digital image and analyzing the identified location information of the nuclei; and (iv) identifying biomarker-positive tumor cells in the region, wherein a biomarker-positive tumor cell is a combination of an identified nucleus and an identified cell membrane surrounding the identified nucleus. Methods of detecting staining using a HER2 protein biomarker or an EGFR protein biomarker are disclosed within US 2017 / 0103521.

[0157] In some embodiments, tumor cell nuclei are automatically identified by first identifying candidate nuclei, and then automatically distinguishing tumor cell nuclei from non-tumor cell nuclei. A variety of methods of identifying candidate nuclei in a tissue image are known in the art. For example, candidate nuclei are automatically detected by applying a radial-symmetry-based method, such as on a post-deconvolved hematoxylin image channel or a biomarker image channel (see Parvin, Bahram, et al. "Iterative voting for inference of structural saliency and characterization of subcellular events." Image Processing, IEEE Transactions on 16.3 (2007): 615-623, the disclosure of which is incorporated by reference herein in its entirety).

[0158] More specifically, in some embodiments, the received image as input is processed, e.g., to detect nucleus centers (seeds) and / or segment nuclei. For example, instructions can be provided to use the technique of Parvin (described above) to detect nucleus centers based on radial symmetry voting. In some embodiments, nuclei are detected using radial symmetry to detect the center of a nucleus, and then the nucleus is classified based on the staining intensity around the cell center. In some embodiments, the radial symmetry based nucleus detection operation is performed as described in commonly assigned and co-pending patent application WO / 2014 / 140085 Al, which is incorporated by reference herein in its entirety. For example, image size can be computed within the image, and one or more votes at each pixel are accumulated by adding the sum of sizes within a selected region. Mean shift clustering can be used to find local centers of the region, which represent the actual nucleus locations. Radial symmetry voting based nucleus detection can be performed on color image intensity data, and explicitly uses the prior domain knowledge that nuclei are differently sized, eccentric, elliptical-like blobs. To accomplish the above, in addition to the color intensity in the input image, image gradient information is used for radial symmetry voting, and combined with an adaptive segmentation process to accurately detect and localize nuclei. For example, as used herein, "gradient" refers to the intensity gradient of a particular pixel computed in consideration of the gradient of intensity values of a set of pixels around the particular pixel. Each gradient can have a particular "direction" with respect to a coordinate system whose x and y axes are defined by two orthogonal edges of the digital image. For example, nucleus seed detection includes defining a seed as a point that is assumed to be located within a nucleus, and serves as a starting point for localizing the nucleus. The first step is to detect seed points associated with each nucleus using a very stable method based on radial symmetry to detect elliptical-like blob structures similar to nuclei. In the radial symmetry method, the gradient image can be processed using a kernel-based voting procedure. Each pixel that passes the voting kernel accumulates a vote, thereby creating a voting response matrix. The kernel is based on the gradient direction computed at that particular pixel, the expected range of minimum and maximum nucleus sizes, and the voting kernel angle (typically in the range of [π / 4, π / 8]). In the resulting voting space, local maxima locations with voting values above a predetermined threshold are saved as seed points. Irrelevant seeds are discarded in the subsequent segmentation or classification process. Other methods are discussed in U.S. Patent Publication No. 2017 / 0140246, the disclosure of which is incorporated by reference herein in its entirety.

[0159] Nuclei can be identified using other techniques known to those of ordinary skill in the art. For example, the image size can be computed from a particular image channel of either the H&E or IHC images, and a plurality of votes can be assigned to each pixel around a specified size with the sum of the sizes in the region around the pixel. Alternatively, a mean shift clustering operation can also be performed to locate the local centers within the voting image that represent the actual locations of the nuclei. In other embodiments, a nuclei segmentation can be used to segment the entire nuclei based on the currently known nuclei centers, by morphological operations and local thresholding. In yet other embodiments, model based segmentation can be utilized to detect the nuclei (i.e., learn a shape model of the nuclei from a training data set and use it as prior knowledge to segment the nuclei in the test image).

[0160] In some embodiments, the nuclei are then segmented using a threshold value that is individually computed for each nucleus. For example, since it is believed that the pixel intensities in the nuclei region can vary, Otsu's method can be used to perform a segmentation operation in the region around the identified nuclei. As will be appreciated by those of ordinary skill in the art, Otsu's method is used to determine the best threshold by minimizing the intra-class variance, and the method is known to those of skill in the art. More specifically, Otsu's method is used to perform automatic clustering-based image thresholding, or reducing a grayscale image to a binary image. The algorithm assumes that the image contains two classes of pixels following a bimodal histogram (foreground and background pixels). Then, the best threshold that separates the two classes of pixels is computed such that the combined spread (intra-class variance) is minimized or equal (since the sum of squared distances is constant), maximizing their inter-class variance.

[0161] In some embodiments, the system and method further comprises automatically analyzing spectral and / or shape features of the nuclei identified in the image to identify nuclei of non-tumor cells. For example, blobs can be identified in the first digital image of the first step. As used herein, a "blob" can be, for example, a region of a digital image in which some attribute (such as intensity or grayscale value) remains constant or varies within a specified range of values. In a sense, all pixels in a blob can be considered similar to each other. Blobs can be identified, for example, using differential methods based on derivatives of position functions on the digital image and methods based on local extrema. A nucleus blob is a blob in which the pixel and / or contour shape suggests that it can have been produced by a nucleus stained with the first stain. For example, a blob can be evaluated for radial symmetry to determine whether it should be identified as a nucleus blob or any other structure, such as a staining artifact. For example, a blob can not be identified as a nucleus blob if it is long and thin and does not have radial symmetry, but rather can be identified as a staining artifact. According to embodiments, blobs identified as "nucleus blobs" can represent a set of pixels identified as candidate nuclei and can be further analyzed to determine whether the nucleus blob represents a nucleus. In some embodiments, any kind of nucleus blob is used directly as an "identified nucleus." In some embodiments, a filtering operation is performed on the identified nuclei or nucleus blobs to identify nuclei that do not belong to a tumor cell that is positive for the biomarker and to remove the identified non-tumor nuclei from the list of identified nuclei or to not add the nuclei to the list of identified nuclei from the start. For example, additional spectral and / or shape features of the identified nucleus blobs can be analyzed to determine whether the nucleus or nucleus blob is a nucleus of a tumor cell. For example, nuclei of lymphocytes are larger than nuclei of other tissue cells, such as lung cells. In the case where the tumor cell is derived from lung tissue, nuclei of lymphocytes can be identified by identifying nucleus blobs that have a minimum size or diameter that is significantly larger than the average size or diameter of nuclei of normal lung cells. The identified nucleus blobs associated with lymphocyte nuclei can be removed (i.e., "filtered") from the set of identified nuclei. By filtering out nuclei of non-tumor cells, the accuracy of the method can be improved. Since non-tumor cells can also express the biomarker to some extent according to the biomarker, intensity signals that are not derived from tumor cells can be produced in the first digital image. By identifying and filtering out nuclei that do not belong to a tumor cell from the total number of identified nuclei, the accuracy of identifying a tumor cell that is positive for the biomarker can be improved. U.S. Patent Publication 2017 / 0103521 describes these and other methods, the disclosure of which is incorporated by reference herein in its entirety. In some embodiments, once a seed is detected, a locally adaptive thresholding method can be used and blobs can be created around the center of the detection.In some embodiments, other methods can also be introduced, for example, a marker-based watershed algorithm can also be used to identify the nuclei spots around the detected nuclei centers. PCT Publication No. WO2016 / 120442 describes these and other methods, the disclosure of which is incorporated herein by reference in its entirety.

[0162] After detecting the nuclei, features (or metrics) are derived from the input image. Deriving metrics from nuclei features is well known in the art, and any known nuclei feature can be used in the context of the present disclosure. Non-limiting examples of computable metrics include:

[0163] (A) Metrics derived from morphological features

[0164] For example, as used herein, a "morphological feature" refers to a feature indicative of the shape or size of a nucleus. Without wishing to be bound by any particular theory, it is believed that morphological features provide some important information about the size and shape of a cell or its nucleus. For example, morphological features can be computed by applying various image analysis algorithms to the pixels contained in or around a nuclei spot or seed. In some embodiments, the morphological features include area, minor and major axis length, perimeter, radius, volume, etc. At the cellular level, such features are used to classify nuclei as healthy cell class or diseased cell class. At the tissue level, these statistical features are leveraged across the tissue to classify the tissue as diseased tissue or non-diseased tissue.

[0165] (B) Metrics derived from color

[0166] In some embodiments, the metrics derived from color include color ratios, R / (R+G+B) or principal components of color. In other embodiments, the metrics derived from color include local statistics (mean / median / variance / standard deviation) of each color and / or color intensity correlations in a local image window.

[0167] (C) Metrics derived from intensity features

[0168] Between the black and white shades of the grey cells represented in the histopathology slide images, there are groups of neighboring cells with certain specific attribute values. Since the correlation of the color features defines an example of size grading, in this way, the intensity of these colored cells can be determined from the dark cell cluster around them.

[0169] (D) Metrics derived from spatial features

[0170] In some embodiments, the spatial features include local density of cells; average distance between two neighboring detected cells; and / or distance from a cell to a segmented region.

[0171] Of course, other features known to those of ordinary skill in the art can also be considered and used as the basis for the feature calculations.

[0172] Classifier module

[0173] In embodiments in which the cell detection and classification engine 255 includes a classifier, the classifier can be any classifier known to those of skill in the art. For example, in certain embodiments, the learned supervised classifier is a support vector machine (“SVM”). Generally, an SVM is a classification technique that is based on statistical learning theory, in which a set of nonlinear input data is transformed into a high-dimensional linear feature space by means of a kernel for nonlinear cases. A support vector machine projects a set of training data E representing two different classes into a high-dimensional space by means of a kernel function K. In this transformed data space, the nonlinear data is transformed such that a hyperplane (discriminating hyperplane) can be formed to separate the classes to the greatest extent possible. Test data is then projected into the high-dimensional space by K and classified based on where the test data falls in relation to the hyperplane (e.g., the features or metrics listed below). The kernel function K defines the method by which the data is projected into the high-dimensional space.

[0174] In other embodiments, the learned supervised classifier is a random forest classifier. For example, a random forest classifier can be trained by (i) creating a training set of tumor and non-tumor cells, (ii) extracting features for each cell, and (iii) training a random forest classifier to distinguish between tumor and non-tumor cells based on the extracted features (e.g., features extracted from the neural network portion of the cell detection and classification engine 255). The trained random forest classifier can then be applied to classify cells in a test image as tumor and non-tumor cells. Optionally, the random forest classifier can be further trained to distinguish between different classes of non-tumor cell nuclei, such as lymphocyte nuclei and stromal nuclei (and even between different types of lymphocytes).

[0175] Scoring module

[0176] In certain embodiments, the obtained staining intensity values, counts of particular cells, or other classification results can be used to determine various marker expression scores, such as a percentage positive, an Allred score, or an H-score, using scoring module 260. Scoring methods are described in more detail in co-assigned and co-pending applications WO / 2014 / 102130 Al “Image analysis for breast cancer prognosis” filed December 19, 2013, and WO / 2014 / 140085 Al “Tissue object-based machine learning system for automated scoring of digital whole slides” filed March 12, 2014, the contents of each are hereby incorporated by reference in their entirety. For example, based at least in part on the number of tumor cells positive for a biomarker / non-tumor cells positive for a biomarker, a score (e.g., a whole slide score or a score for a labeled region of an image, e.g., a region that can be labeled by a pathologist or histologist) can be determined. In certain embodiments, for each detected cell spot, average spot intensity, color, and geometric features such as area and shape of the detected cell spot can be computed, and the cell spot classified as a tumor cell and a non-tumor cell. The number of recognized cells output corresponds to the total number of biomarker-positive tumor cells detected in a region, as evidenced by the number of tumor cells counted. Other methods of scoring a sample are described in PCT Publication No. WO / 2017 / 093524 and U.S. Patent Publication Nos. 2017 / 0103521 and 2017 / 0270666, the disclosures of which are hereby incorporated by reference in their entirety.

[0177] In some embodiments, the expression score is an H-Score score. The H-score is a method of assessing the degree of cellular immunoreactivity, for example. Depending on the biomarker, different methods for H-score calculation can be used. As one illustrative example, the H-score for steroid receptor cells can be obtained by the following formula: 3 x percent strongly stained cells + 2 x percent moderately stained cells + percent weakly stained cells, ranging from 0 to 300.

[0178] In some embodiments, the "H" score is used to assess the percentage of tumor cells that have a membrane staining grade of "weak," "moderate," or "strong." The grades are added together, and the total score can be up to 300, with a cutoff of 100 differentiating "positive" and "negative." For example, the intensity of membrane staining (0, 1+, 2+, or 3+) is determined for each cell in a fixed field (or, here, each cell in a tumor or cell cluster). The H-score can be simply a matter of the predominant staining intensity, or more complexly, can include a sum of individual H-score scores for each intensity level seen. By one approach, the percentage of cells at each staining intensity level is calculated, and finally, an H-score is assigned using the following formula: [1 x (% cells 1+) + 2 x (% cells 2+) + 3 x (% cells 3+)]. The final score, which ranges from 0 to 300, provides more relative weight to higher intensity membrane staining in a given tumor sample. The sample can then be considered positive or negative depending on a particular cutoff threshold. Additional methods of calculating H-score are described in U.S. Patent Publication No. 2015 / 0347702, the disclosure of which is hereby incorporated by reference in its entirety.

[0179] By way of example only, in embodiments in which a sample is stained for the presence of a lymphocyte biomarker and also for the presence of PD-L1, PD-L1 expression can be scored by: (a) identifying tumor cells and lymphocytes in a tumor sample; (b) determining the number of tumor cells and lymphocytes that express PD-L1 and / or the relative intensity of PD-L1 expression in the cells; and (c) classifying the tumor according to the PD-L1 expression determined in (b). In certain embodiments, the expression of PD-L1 is determined by specific detection of PD-L1 protein and / or PD-L1 mRNA in the tumor. In certain embodiments, a cell is considered to express PD-L1 when the cell has at least partial membrane staining of PD-L1 protein detected by IHC. In certain embodiments, the tumor is classified according to one or both of a modified H-score (MHS) or a modified proportional score (MPS), both of which are calculated from step (b) (see U.S. Publication No. 2017 / 0372117 for additional information), the disclosure of which is hereby incorporated by reference in its entirety.

[0180] In some embodiments, assigning the MHS includes (i) estimating four separate percentages of unstained, weakly stained (+1), moderately stained (+2), and strongly stained (+3) cells among all viable tumor cells and stained mononuclear inflammatory cells in all examined tumor nests, wherein cells must have at least partial membrane staining to be included in the weak, moderate, or strong stained percentage, and wherein the sum of all four percentages equals 100; and (ii) inputting the estimated percentages into the formula 1 × (percentage of weakly stained cells) + 2 × (percentage of moderately stained cells) + 3 × (percentage of strongly stained cells) and assigning the result of the formula to the tissue section as the MHS; wherein assigning the MPS includes estimating the percentage of cells with at least any intensity of partial membrane staining among all viable tumor cells and mononuclear inflammatory cells in all examined tumor nests and assigning the resulting percentage to the tissue section as the MPS; and wherein if both the MHS and MPS are assigned, the assignments can be performed sequentially or simultaneously. For example, the four categories “no,” “weak,” “medium,” and “strong” can be defined as, for example, non-overlapping intensity threshold ranges; for example, if the average intensity value is less than 5%, the cell pixel region can be considered a “no-stained” cell, if the average intensity value is >5% and <25%, it can be considered a “weakly stained” cell, if the average intensity value is >=25% and <75%, it can be considered a “mediumly stained” cell, and if the average intensity value is >=75%, it can be considered a “strongly stained” cell.

[0181] In some embodiments, the expression score is the Allred score. The Allred score is a scoring system that displays the percentage of cells that are positive for the hormone receptor test, as well as the degree to which the receptor is presented after staining (referred to as "intensity"). This information is then used to score the sample on a scale of 0 to 8. A higher score indicates more receptors and a greater likelihood of their presence in the sample.

[0182] Examples

[0183] Cell clustering

[0184] During the process of learning to detect and locate cells, Model A learns features that facilitate cell clustering. Although Model A does not provide any category labels, it uses the original RGB values ​​of the cells (see [link to Model A]). FIG. 9B Compared to these, these features produced better clustering results (see...). FIG. 9A Qualitative cell clustering performance in FIG. 9A and 9B As shown in [the image]. FIG. 9AIn the case of the blue boxes, they are mainly placed on Her2 positive cells, while the green boxes are placed on Her2 negative cells, which means that by using the features derived from Faster-RCNN for clustering, the two types of cells can be well separated. Even though we did not provide cell class labels for fine-tuning Faster-RCNN, the pre-trained Faster-RCNN already provides richer feature representations that have been "transferred" to the fine-tuned Faster-RCNN. It was found that this improves the clustering performance. The features extracted from block 1 of Model A can achieve meaningful cell clustering (see FIG. 9A ).

[0185] In the case of the blue boxes, they are mainly placed on Her2 positive cells, while the green boxes are placed on Her2 negative cells, which means that by using the features derived from Faster-RCNN for clustering, the two types of cells can be well separated. Even though we did not provide cell class labels for fine-tuning Faster-RCNN, the pre-trained Faster-RCNN already provides richer feature representations that have been "transferred" to the fine-tuned Faster-RCNN. It was found that this improves the clustering performance. The features extracted from block 1 of Model A can achieve meaningful cell clustering (see FIG. 9B ). FIG. 9B ). After qualitatively observing the meaningful cluster patterns in the HER2 images, we annotated the dataset BB with two classes to enable a quantitative evaluation of the advantage of using Faster-RCNN features for clustering. FIG. 10A The clustering performance was summarized using the representations from different convolutional layers of Model A as well as the original RGB representation of the detected cells. If the number of clusters is restricted to two, the Faster-RCNN features have a clear advantage of 8% over the RGB representation. In other words, if we cluster the cells into two clusters using the Faster-RCNN features and assign a cluster label to all cells in each cluster, we can achieve a classification accuracy of 98%, while the same procedure can achieve an accuracy of 90% if we use the RGB representation. In addition, the features extracted from block0, block1 and block2 have similar performance on the BB dataset; and the block0 features have very consistent performance for different numbers of clusters.

[0186] Next, it was determined whether training an object detection engine to classify the same type of cells that will be clustered later improves the usefulness of the learned features for the clustering task compared to an object detection engine trained only to detect cells. To answer this question, we measured the performance of clustering using the features extracted from Model A, Model B, Model C and Model Zero. Although the representation learned using any of these models improves over the original RGB representation of the cells by at least 7%, we did not observe any significant differences between the results of these models. This indicates that even the features extracted from Model Zero (trained only on the COCO dataset) can be used to meaningfully cluster the cells and thus even further improve the ground truth labeling.

[0187] Invisible class clustering

[0188] BBYT images contain five classes of cells. Some of these classes can be easily distinguished by the RGB values, while others are defined by both the RGB and the morphology or the context. One class of tumor cells resembles one class of regular cells in blue, while another class of tumor cells shares the purple color representation with one class of T cells. These classes can be distinguished based on the density, shape, and context of the cells. In this section, we investigate the utility of comparing the model B learned representation to the original RGB representation of BBYP cells to cluster them into homogeneous clusters. We hypothesize that since CNN features have multiple levels of abstraction compared to the original image pixels, they provide a more global view of the objects, and thus they should better distinguish between classes of cells that look similar in individual appearance but can be distinguished by contextual information (see FIG. 10B ). In this case, using two class labels (from the BB dataset) from the features derived from the fine-tuned Faster-RCNN through clustering can well distinguish the five types of cells even with a small number of clusters (e.g., 5), with a 15% performance improvement compared to the original RGB values. It is believed that the blockO features have the best performance for all numbers of clusters.

[0189] Further metastasis

[0190] BBYP images are stained with Discovery Yellow and Discovery Purple color bodies to identify 5 classes of target cells, namely ki67+ / - tumor cells, Ki67+ / - T cells, and CD8+ T cells. Some of these classes can be easily distinguished by the color, while others are defined by both the color and the morphology or the context (see FIG. 11 ). We hypothesize that since CNN features have multiple levels of abstraction compared to the original image pixels, they provide a more global view of the objects, and thus they should better distinguish between classes of cells that look similar in individual appearance but can be distinguished by contextual information. As shown in FIG. 10B , clustering BBYT cells using the features of model B has a 15% advantage over using the original RGB representation of these cells and continues to outperform them as we increase the number of clusters, thus supporting our initial hypothesis. Using the features of other models yields similar results.

[0191] CONCLUSION

[0192] We show that pre-training on publicly available large-scale object detection datasets can reduce the need for regulation. The closest prior art (Rene Bidart et al., "Localization and classification of cell nuclei in post-neoadjuvant breast cancer surgical specimen using fully convolutional networks," in Medical Imaging 2018: Digital Pathology. International Society for Optics and Photonics, 2018, vol. 10581) uses 27K point annotations provided by pathologists, while we train model A using only 1800 bounding box annotations.

[0193] We also demonstrate the utility of Faster R-CNN to handle histology images of different resolutions simultaneously and to perform inference on test images that are 100 times larger than the training image patches. The limitation here is not the algorithm but memory related.

[0194] We suggest facilitating ground truth labeling by proposing bounding boxes that can be fine-tuned by pathologists if needed, and clustering the detected cells into several homogeneous subsets, thereby reducing the burden of ground truth labeling from the order of the number of cells in the dataset to the number of clusters.

[0195] As FIG. 8 shown, sometimes the model is penalized for detecting cells that the annotator missed. Due to the ambiguous nature of the objects present in histology images, multiple annotators are needed to annotate the same dataset, and the performance of the detection and classification model should be compared to the annotator agreement.

[0196] Other components to practice embodiments of the disclosure

[0197] Other components (e.g., systems or modules) that can be used in conjunction with the systems and methods of the present disclosure are described below.

[0198] Unmixing module

[0199] In certain embodiments, the images received as input can be multi-channel images, i.e., the received images are of a biological specimen stained with more than one stain. In these embodiments, the multi-channel images are first unmixing into their constituent channels, where each unmixing channel corresponds to a particular stain or signal, before further processing. In certain embodiments, the unmixing images, often referred to as“channel images” or“image channel images,” and can be used as input to each of the modules described herein. For example, inter-marker heterogeneity can be determined using a first H&E image stained for clusters of multiple differentiation markers (CD3, CD8, etc.), a second multi-channel image, and multiple single-channel images each stained for a particular biomarker (e.g., ER, PR, Ki67, etc.). In this example, the multi-channel image is first unmixing into its constituent channel images, and those channel images can be used along with the H&E image and the multiple single-channel images to determine inter-marker heterogeneity.

[0200] In certain embodiments, in a sample containing one or more stains and hematoxylin, a single image can be produced for each channel of the one or more stains and hematoxylin. Without wishing to be bound by any particular theory, it is believed that these channels highlight different tissue structures in the tissue image, and thus, they can be referred to as structure image channels. In certain embodiments, unmixing provides at least a hematoxylin image channel. In certain embodiments, the captured images are unmixing into a separate channel that represents the local amount of hematoxylin and highlights the cellular regions within the image. Those skilled in the art will recognize that features extracted from these channels can be used to describe different biological structures present in any image of the tissue.

[0201] The multi-spectral images provided by the imaging system 202 are a weighted mixture of the underlying spectral signals associated with individual biomarkers and noise components. At any particular pixel, the mixing weights are proportional to the marker expression of the underlying co-located biomarkers in that particular location of the tissue and the background noise at that location. Thus, the mixing weights differ between different pixels. The spectral unmixing methods disclosed herein decompose the multi-channel pixel value vector at each pixel into a set of constituent biomarker members or components and estimate the proportion of the individual constituent stains for each biomarker.

[0202] Unmixing refers to the process of decomposing a measured spectrum of a mixed pixel into a set of component spectra or endmembers representing the proportion of each endmember in the pixel, and a set of corresponding fractions or abundances. In particular, the unmixing process can extract stain-specific channels so that local concentrations of individual stains can be determined using reference spectra known for standard types of tissue and stain combinations. The unmixing can use reference spectra retrieved from control images or estimated from the images under observation. Unmixing the component signals of each input pixel can retrieve and analyze stain-specific channels, such as the hematoxylin channel and the eosin channel in H&E images, or the diaminobenzidine (DAB) channel and the counterstain (such as hematoxylin) channel in IHC images. The terms "unmixing" and "color deconvolution" (or "deconvolution") or similar terms (such as "demixing") are used interchangeably in the prior art.

[0203] In some embodiments, the multi-channel image is unmixed by the unmixing module 205 in a linear unmixing manner. Linear unmixing is described, for example, in "Zimmermann 'Spectral Imaging and Linear Unmixing in Light Microscopy' Adv Biochem Engin / Biotechnol (2005) 95:245-265" and in C. L. Lawson and R. J. Hanson, 'Solving least squares Problems', Prentice Hall, 1974, Chapter 23, page 161", the disclosures of which are incorporated herein by reference in their entirety. In linear stain unmixing, the measured spectrum at any pixel (S( )) is considered to be a linear mixture of stain spectral components, and is equal to the sum of the proportions or weights (A) of each individual stain represented at that pixel.

[0204] S( ) = A1 · R1( ) + A2 · R2( ) + A3 · R3( ).... A i ·R i ( )

[0205] More generally, this can be represented in matrix form as

[0206] S( ) = Σ A i ·R i ( ) or S = R · A

[0207] If there are M acquired channel images and N individual stains, the columns of the M x N matrix R are the best colorimetric systems derived herein, the N x 1 vector A is the unknown of individual stain proportions, and the M x 1 vector S is the measured multi-channel spectral vector at a pixel. In these equations, the signal in each pixel (S) is measured during acquisition of the multi-channel image, and the reference spectra, i.e., the best colorimetric systems, described herein are derived. Their contribution is determined by calculating the contribution of each of the various stains (A i ) to each point in the measured spectrum. In some embodiments, the solution is found using a least-squares fitting method that minimizes the squared difference between the measured and calculated spectra by solving the following system of equations.

[0208]

[0209] In this equation, j represents the number of detection channels, and i equals the number of stains. The solution to the linear equations generally allows for constrained unmixing, forcing the weights (A) to add up to one.

[0210] In other embodiments, unmixing is accomplished using the methods described in WO2014 / 195193, entitled "Image Adaptive Physiologically Plausible Color Separation," filed May 28, 2014, the disclosure of which is incorporated herein by reference in its entirety. Generally, WO2014 / 195193 describes an unmixing method that separates the component signals of the input image by using iteratively optimized reference vectors. In some embodiments, the image data in an assay is correlated to expected or ideal results specific to the characteristics of the assay to determine a quality metric. In cases where the image quality is low or the correlation is poor compared to the ideal results, one or more of the reference column vectors in the matrix R is adjusted, and the unmixing is repeated iteratively with the adjusted reference vectors until the correlation shows a high quality image that meets the physiological and anatomical requirements. The anatomical, physiological, and assay information can be used to define rules that are applied to the measured image data to determine the quality metric. This information includes how the tissue is stained, which structures within the tissue are intended or unintended to be stained, and the relationship between the structures, stains, and markers specific to the assay being processed. The iterative process results in stain-specific vectors that can generate images that accurately identify the target structures and biologically relevant information, and that are free of any noise or unwanted spectra, so the process is suitable for analysis. The reference vectors are adjusted within a search space. The search space defines the range of values that the reference vectors can represent for the stains. The determination of the search space can be accomplished by scanning various representative training assays, including those with known or commonly occurring issues, and determining a set of high quality reference vectors for the training assays.

[0211] In other embodiments, unmixing is accomplished using the methods described in WO2015 / 124772, entitled "Group Sparsity Model for Image Unmixing," filed February 23, 2015, the disclosure of which is hereby incorporated by reference in its entirety. Generally, WO2015 / 124772 describes unmixing using a set of sparsity frameworks, where fractions of stain contributions from multiple co-located markers are modeled within a "same group," and fractions of stain contributions from multiple non-co-located markers are modeled, the co-localization information of the multiple co-located markers is provided to the modeled group sparsity frameworks, the modeled frameworks are solved using group lasso to obtain a least squares solution within each group, where the least squares solution corresponds to unmixing of the co-located markers, and a sparse solution is obtained in groups corresponding to unmixing of the non-co-located markers. Further, WO2015 / 124772 describes a method of unmixing by inputting image data obtained from a biological tissue sample, reading reference data from electronic storage (the reference data describing a stain color of each of a plurality of stains), reading co-location data from electronic storage (the co-location data describing groups of stains, each group comprising stains that can be co-located in the biological tissue sample, and each group constituting a group for a group lasso criterion, at least one of the groups having a size of two or more, and using the reference data as a reference matrix to compute a solution for the group lasso criterion to obtain an unmixed image. In certain embodiments, a method for unmixing an image can include generating a group sparsity model, where fractions of stain contributions from co-located markers are assigned within a single group, and fractions of stain contributions from non-co-located markers are assigned within separate groups, and solving the group sparsity model using an unmixing algorithm to obtain a least squares solution within each group.

[0212] Other system components

[0213] The system 200 of the present disclosure can be tied to a specimen processing device capable of performing one or more preparation processes on the tissue specimen. The preparation processes can include, but are not limited to, specimen de-paraffinization, conditioning the specimen (e.g., cell conditioning), staining the specimen, performing antigen retrieval, performing immunohistochemical staining (including labeling) or other reactions, and / or performing in situ hybridization (e.g., SISH, FISH, etc.) staining (including labeling) or other reactions, among other processes that prepare the specimen for microscopy, microscopic analysis, mass spectrometry methods, or other analytical methods.

[0214] The processing device can apply a fixative to the specimen. Fixatives can include cross-linking agents (e.g., aldehydes such as formaldehyde, paraformaldehyde, and glutaraldehyde, and non-aldehyde cross-linking agents), oxidizing agents (e.g., metal ions and complexes such as osmium tetroxide and chromic acid), protein denaturants (e.g., acetic acid, methanol, and ethanol), fixatives of unknown mechanism (e.g., mercuric chloride, acetone, and picric acid), combined reagents (e.g., Carnoy's fixative, Methacarn, Bouin's fluid, B5 fixative, Rossman's fluid, and Gendre's fluid), microwaves, and other fixatives (e.g., exclusion volume fixation and vapor fixation).

[0215] If the specimen is a sample embedded in paraffin, the sample can be deparaffmized using a corresponding deparaffmization solution. After the paraffin is removed, any number of chemicals can be applied to the specimen in succession. These chemicals can be used for pretreatment (e.g., reversing protein cross-linking, exposing cellular acids, etc.), denaturation, hybridization, washing (e.g., stringent washing), detection (e.g., attaching a display or marker molecule to a probe), amplification (e.g., amplifying a protein, a gene, etc.), counterstaining, coverslipping, etc.

[0216] The specimen processing device can apply a variety of different chemicals to the specimen. These chemicals include, but are not limited to, stains, probes, reagents, rinses, and / or conditioning agents. These chemicals can be fluids (e.g., gases, liquids, or gas / liquid mixtures) or the like. The fluids can be solvents (e.g., polar solvents, non-polar solvents, etc.), solutions (e.g., aqueous solutions or other types of solutions), or the like. The reagents can include, but are not limited to, stains, wetting agents, antibodies (e.g., monoclonal antibodies, polyclonal antibodies, etc.), antigen retrieval solutions (e.g., water-based or non-water-based antigen retrieval solutions, antigen retrieval buffers, etc.), or the like. The probes can be isolated cellular acids or isolated synthetic oligonucleotides attached to a detectable label or reporter molecule. The labels can include radioisotopes, enzyme substrates, co-factors, ligands, chemiluminescent or fluorescent agents, haptens, and enzymes.

[0217] After the specimen processing is complete, the user can transport the specimen slide to an imaging device. In some embodiments, the imaging device is a brightfield imager slide scanner. One brightfield imager is the iScan Coreo brightfield scanner sold by Ventana Medical Systems, Inc. In automated embodiments, the imaging device is a digital pathology apparatus as disclosed in International Patent Application No. PCT / US2010 / 002772 entitled IMAGING SYSTEM AND TECHNIQUES (Patent Publication No. WO / 2011 / 049608); or as disclosed in U.S. Publication 2014 / 0377753 or International Patent Application No. PCT / US2010 / 002772, the disclosures of which are hereby incorporated by reference in their entireties.

[0218] The imaging system or device can be a multispectral imaging (MSI) system or a fluorescence microscope system. As used herein, the imaging system is an MSI. In general, MSI equips a computerized microscope-based imaging system for the analysis of a pathology specimen by accessing the spectral distribution of the image on a pixel layer. While various multispectral imaging systems exist, these systems share a commonality in operation in that they are capable of forming multispectral images. A multispectral image refers to an image that captures image data over a specific wavelength or specific spectral bandwidth of the electromagnetic spectrum. These wavelengths can be selected by optical filters or using other instruments capable of selecting predetermined spectral components, including electromagnetic radiation outside the visible light range, e.g., infrared (IR).

[0219] The MSI system can include an optical imaging system that includes, in part, a spectral selection system that is adjustable to define a predetermined number of N discrete optical bands. The optical system can be adapted for imaging of a tissue sample that is illuminated by a broadband light source onto an optical detector. In one embodiment, the optical imaging system can include a magnification system, e.g., a microscope, having a single optical axis that is generally spatially aligned with a single light output of the optical system. When the spectral selection system is adjusted or tuned (e.g., with a computer processor), the system forms a sequence of images of the tissue, e.g., to ensure that images are acquired in different discrete spectral bands. The apparatus can additionally include a display that can display at least one visually perceptible image of the tissue from the sequence of acquired images. The spectral selection system can include an optical dispersion element, e.g., a diffraction grating, a set of optical filters, e.g., thin film interference filters, or any other optical element adapted to select a particular passband from the spectrum of light transmitted from the light source through the sample to the detector in response to a user input or a preprogrammed processor command.

[0220] In alternative implementations, the spectral selection system defines a plurality of light outputs corresponding to the N discrete spectral bands. This type of system receives the transmitted light output from the optical system and spatially re-directs at least a portion of the light output along N spatially distinct optical paths such that the sample in an identified spectral band can be imaged onto a detector system along the optical path corresponding to the identified spectral band.

[0221] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus. Any of the modules described herein can include logic that is executed by a processor. As used herein, "logic" refers to any information having a form that is applicable to an operation of a processor, which can be applied to affect the operation of the processor. Software is an example of logic.

[0222] The computer storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non- exhaustive list) of the computer storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a memory stick, a floppy diskette, a mechanically encoded device having a magnetic, optical, or other encoding, a ROM, a RAM, a flash memory, an erasable programmable read-only memory (EPROM), an EEPROM, a portable compact disc read-only memory (CD-ROM), digital video disk (DVD), an optical disk, a conventional hard disk drive, an HD DVD, a Blu-Ray disk, a memory in a mobile device, a memory card, a solid-state drive, or any suitable combination of the foregoing. Computer storage media can be tangible and non-transitory. Computer storage media can also include non-tangible media, such as a transitory computer readable storage medium that stores the program code for a short time while the program code is loaded into a dynamic storage that is an example of a computer-readable storage medium. Like a computer-readable storage medium that is a tangible storage medium, a transitory computer-readable storage medium excludes propagated signals per se. A computer-readable medium also includes a propagated signal per se.

[0223] The term "programmable processor" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable microprocessor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0224] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0225] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0226] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0227] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a LCD (liquid crystal display), LED (light-emitting diode) display, or OLED (organic light-emitting diode) display, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. In some embodiments, a touch screen can be used to display information and receive input from a user. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0228] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example, FIG. 1 Network 20 can include one or more local area networks.

[0229] The computing system can include arbitrary numbers of clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., HTML pages) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0230] Additional embodiments

[0231] Additional Embodiment 1.A system for identifying cell clusters within a sample image of a biological specimen, the biological specimen being stained with a primary stain or stained for the presence of one or more biomarkers, the system comprising: (i) one or more processors, and (ii) one or more memories coupled with the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

[0232] a. detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine comprises a convolutional neural network adapted to detect cell features within a sample image;

[0233] b. extracting cell features from one or more layers of the convolutional neural network; and

[0234] c. clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of detected cells.

[0235] Additional Embodiment 2. The system of additional embodiment 1, wherein the trained object detection engine comprises a region-based convolutional neural network.

[0236] Additional Embodiment 3. The system of additional embodiment 2, wherein the region-based convolutional neural network comprises Fast-RCNN.

[0237] Additional Embodiment 4. The system of additional embodiment 2, wherein the trained object detection engine further comprises a region proposal network.

[0238] Additional Embodiment 5. The system of additional embodiment 4, wherein the trained object detection engine comprises Faster-RCNN.

[0239] Additional Embodiment 6. The system of additional embodiment 1, wherein the convolutional neural network comprises LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, or DenseNet.

[0240] Additional Embodiment 7. The system of additional embodiment 1, wherein the convolutional neural network is a deep residual network.

[0241] Additional Embodiment 8. The system of additional embodiment 7, wherein the deep residual network comprises ResNet.

[0242] Additional Embodiment 9. The system of additional embodiment 1, wherein the trained object detection engine comprises a convolutional neural network trained using a dataset comprising a plurality of training images, wherein each of the plurality of training images is derived from a biological specimen that is stained with a hematoxylin stain or stained for the presence of one or more biomarkers.

[0243] Additional Embodiment 10. The system of additional embodiment 9, wherein each of the plurality of training images does not contain a class label.

[0244] Additional Embodiment 11. The system of additional embodiment 9, wherein each of the plurality of training images does not contain any pathologist annotation.

[0245] Additional Embodiment 12. The system of additional embodiment 9, wherein each of the plurality of training images contains at least one class label.

[0246] Additional Embodiment 13. The system of additional embodiment 12, wherein the operations further comprise classifying the detected cells.

[0247] Additional Embodiment 14. The system of additional embodiment 1, wherein the convolutional neural network comprises at least one scaling layer.

[0248] Additional Embodiment 15. The system of additional embodiment 14, wherein the at least one scaling layer scales the sample image by a constant factor.

[0249] Additional Embodiment 16. The method of additional embodiment 1, wherein clustering the detected cells comprises performing agglomerative hierarchical clustering.

[0250] Additional Embodiment 17. The system of additional embodiment 1, wherein the system is adapted to receive one or more annotations of at least one of the one or more homogenous clusters corresponding to cells in the sample image.

[0251] Additional Embodiment 18. The system of additional embodiment 17, wherein the operations further comprise training the cell detection and classification engine using the received annotations.

[0252] Additional Embodiment 19. The system of additional embodiment 18, wherein the cell detection and classification engine comprises a neural network.

[0253] Additional Embodiment 20.The system of additional embodiment 19, wherein the neural network of the cell detection and classification engine is adapted from LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, DenseNet, or ResNet.

[0254] Additional Embodiment 21. The system of additional embodiment 18, wherein the operations further comprise detecting cells in a test image using the trained cell detection and classification engine.

[0255] Additional Embodiment 22. The system of additional embodiment 21, wherein the operations further comprise classifying cells in a test image.

[0256] Additional Embodiment 23. The system of additional embodiment 21, wherein the test image comprises at least one stain.

[0257] Additional Embodiment 24. The system of additional embodiment 23, wherein the at least one stain of the test image is a cell stain.

[0258] Additional Embodiment 25. The system of additional embodiment 23, wherein the at least one stain of the test image is a membrane stain.

[0259] Additional Embodiment 26. The system of additional embodiment 17, wherein the operations further comprise fine-tuning the trained object detection engine using the received annotations to provide a fine-tuned object detection engine.

[0260] Additional Embodiment 27. The system of additional embodiment 26, wherein the operations further comprise using the fine-tuned object detection engine to detect and / or classify cells in a test image.

[0261] Additional Embodiment 28. A non-transitory computer-readable medium storing instructions for identifying cell clusters within a sample image of a biological specimen stained for presence of one or more biomarkers, comprising:

[0262] (a) detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine comprises a convolutional neural network adapted to detect cell features within a sample image;

[0263] (b) extracting cell features from one or more layers of the convolutional neural network; and

[0264] (c) clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of detected cells.

[0265] Additional Embodiment 29. The non-transitory computer-readable medium of additional embodiment 28, wherein the convolutional neural network is adapted from LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, and DenseNet.

[0266] Additional Embodiment 30. The non-transitory computer-readable medium of additional embodiment 28, wherein the convolutional neural network is a deep residual network.

[0267] Additional Embodiment 31. The non-transitory computer-readable medium of additional embodiment 30, wherein the deep residual network is adapted from ResNet.

[0268] Additional Embodiment 32. The non-transitory computer-readable medium of additional embodiment 28, further comprising instructions for training the cell detection and classification engine using pathologist annotations corresponding to at least one of the one or more homogeneous clusters of cells in the sample image.

[0269] Additional Embodiment 33. The non-transitory computer-readable medium of additional embodiment 32, further comprising instructions for detecting cells in a test image using the trained cell detection and classification engine.

[0270] Additional Embodiment 34. The non-transitory computer-readable medium of additional embodiment 28, further comprising instructions for fine-tuning the trained object detection engine using the received annotations for at least one of the one or more homogeneous clusters of cells in the sample image to provide a fine-tuned object detection engine.

[0271] Additional Embodiment 35. The non-transitory computer-readable medium of additional embodiment 34, further comprising instructions for detecting cells in a test image using the fine-tuned object detection engine.

[0272] Additional Embodiment 36. The non-transitory computer-readable medium of additional embodiment 28, wherein the object detection engine is trained using a dataset comprising a plurality of training images from biological specimens stained for presence of one or more biomarkers.

[0273] Additional Embodiment 37.The non-transitory computer-readable medium of additional embodiment 28, wherein the object detection engine is trained using a dataset comprising a plurality of training images, wherein each training image of the plurality of training images does not contain any class label.

[0274] Additional Embodiment 38. The non-transitory computer-readable medium of additional embodiment 28, wherein the convolutional neural network comprises at least one scaling layer.

[0275] Additional Embodiment 39. A method of detecting cells in a test image of a biological specimen stained for the presence of one or more biomarkers, comprising:

[0276] (a) determining cell features within the test image of the biological specimen by inputting the test image into a first portion of a trained cell detection and classification engine, wherein the first portion comprises a convolutional neural network configured to detect the cell features; and

[0277] (b) detecting cells in the test image based on the determined cell features by inputting the determined features of the test image into a second portion of the trained cell detection and classification engine, wherein the second portion comprises at least a region proposal network.

[0278] Additional Embodiment 40. The method of additional embodiment 39, further comprising classifying the detected cells in the test image.

[0279] Additional Embodiment 41. The method of additional embodiment 39, wherein the cell detection and classification engine is trained using ground truth data, wherein the ground truth data comprises pathologist annotations of one or more identified homogeneous clusters of cells in the sample image.

[0280] Additional Embodiment 42. The method of additional embodiment 41, wherein the homogeneous clusters of cells are identified by: (i) providing the sample image to a trained object detection engine; (ii) detecting cells in the sample image using the trained object detection engine; (iii) extracting a plurality of cell features from at least one convolutional layer within the trained object detection engine; and (iv) clustering the detected cells in the test image based on the extracted cell features.

[0281] Additional Embodiment 43. The method of additional embodiment 39, wherein the convolutional neural network is a deep residual network.

[0282] Additional Embodiment 44. The method of additional embodiment 39, wherein the deep residual network is adapted from ResNet.

[0283] Additional Embodiment 45. A method of identifying cell clusters within a sample image of a biological specimen stained for the presence of one or more biomarkers, comprising:

[0284] a. detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine comprises a convolutional neural network and a region proposal network, wherein the convolutional neural network is configured to recognize cell features, and wherein the region proposal network is configured to detect cells based on the recognized cell features;

[0285] b. extracting the recognized cell features from one or more layers of the convolutional neural network; and

[0286] c. clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of detected cells.

[0287] Additional Embodiment 46. The method of additional embodiment 45, wherein the trained object detection engine comprises a region-based convolutional neural network.

[0288] Additional Embodiment 47. The method of additional embodiment 46, wherein the region-based convolutional neural network is adapted from Fast-RCNN.

[0289] Additional Embodiment 48. The method of additional embodiment 46, wherein the trained object detection engine further comprises a region proposal network.

[0290] Additional Embodiment 49. The method of additional embodiment 48, wherein the trained object detection engine is adapted from Faster-RCNN.

[0291] Additional Embodiment 50. The method of additional embodiment 45, wherein the convolutional neural network is adapted from LeNet, AlexNet, ZF Net, DetectNet, GoogleNet, VGGNet, VGG16, and DenseNet.

[0292] Additional Embodiment 51. The method of additional embodiment 45, wherein the convolutional neural network is a self-encoding neural network.

[0293] Additional Embodiment 52. The method of additional embodiment 45, wherein the convolutional neural network is a deep residual network.

[0294] Additional Embodiment 53. The method of additional embodiment 52, wherein the deep residual network is adapted from ResNet.

[0295] Additional Embodiment 54. The method of additional embodiment 45, wherein the convolutional neural network is trained using a dataset comprising a plurality of training images of a biological specimen stained for the presence of one or more biomarkers.

[0296] Additional Embodiment 55. The method of additional embodiment 54, wherein each training image of the plurality of training images does not contain a class label.

[0297] Additional Embodiment 56. The method of additional embodiment 54, wherein each training image of the plurality of training images does not contain any pathologist annotation.

[0298] Additional Embodiment 57. The method of additional embodiment 54, wherein each training image of the plurality of training images contains at least one class label.

[0299] Additional Embodiment 58. A method of detecting and classifying cells in a test image of a biological specimen stained for the presence of one or more biomarkers, comprising:

[0300] (a) determining cell features within the test image of the biological specimen by inputting the test image into a first portion of a trained cell detection and classification engine, wherein the first portion comprises a neural network configured to detect cell features; and

[0301] (b) detecting cells in the test image based on the determined cell features by inputting the determined features of the test image into a second portion of the trained cell detection and classification engine, wherein the second portion comprises a trained classifier.

[0302] Additional Embodiment 59. The method of additional embodiment 58, wherein the trained classifier comprises a support vector machine or a random forest ensemble learning method.

[0303] All of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications referred to in this specification are hereby incorporated by reference in their entirety. As necessary, modifications to various aspects of the embodiments can be made to employ the various patent, application, and publication concepts to provide other further embodiments.

[0304] While the present disclosure has been described with reference to a number of illustrative embodiments, it will be understood that the present disclosure can be embodied in many other specific forms without departing from the spirit or scope of the disclosure. More particularly, it is to be understood that changes in the proposed arrangement of the components and / or arrangement of the subject combination can be made by those skilled in the art without departing from the essence of the present disclosure. Alternative uses will also become apparent to those skilled in the art.

Claims

1. A system for identifying cell clusters within a sample image of a biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers, the system comprising: (i) one or more processors, and (ii) one or more memories coupled with the one or more processors that store computer executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: identifying one or more homogeneous clusters of cells within a sample image of the biological specimen that is stained with the primary stain or stained for the presence of one or more biomarkers, wherein identifying one or more homogeneous clusters of cells comprises: a. detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine includes a convolutional neural network adapted to detect cell features within the sample image, wherein the object detection engine is trained using a dataset that includes a plurality of training images, wherein each training image of the plurality of training images is derived from a training biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers, wherein each training image of the plurality of training images does not include any pathologist annotation; b. extracting cell features from one or more layers of the convolutional neural network; and c. clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of the detected cells; and training a cell detection and classification engine using pathologist annotation of the one or more identified homogeneous clusters of cells.

2. The system of claim 1, wherein the trained object detection engine includes one of Fast-RCNN or Faster-RCNN.

3. The system of claim 1, wherein the trained object detection engine further includes a region proposal network.

4. The system of claim 1, wherein the convolutional neural network is a deep residual network.

5. The system of claim 1, wherein each training image of the plurality of training images does not include any pathologist annotation.

6. The system of claim 1, wherein each training image of the plurality of training images includes at least one class label.

7. The system of any of the preceding claims, wherein the convolutional neural network includes at least one scaling layer.

8. The system of any of the preceding claims, wherein the clustering of the detected cells includes performing agglomerative hierarchical clustering.

9. The system of any of the preceding claims, wherein the operations further comprise: training a cell detection and classification engine using one or more class labels associated with the one or more homogeneous clusters of cells in the sample image.

10. The system of claim 9, wherein the cell detection and classification engine includes a neural network.

11. The system of claim 9, wherein the operations further comprise: detecting and / or classifying cells in a test image that includes at least one stain using the trained cell detection and classification engine.

12. A non-transitory computer-readable medium storing instructions for identifying a cluster of cells within a sample image of a biological specimen that is stained for the presence of one or more biomarkers, comprising: identifying one or more homogeneous clusters of cells within a sample image of the biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers, wherein identifying one or more homogeneous clusters of cells comprises: a. detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine comprises a first portion and a second portion, wherein the first portion is configured to identify cell features within the sample image, and wherein the second portion is configured to detect the cells based on the identified cell features, wherein the object detection engine is trained using a dataset comprising a plurality of training images, wherein each training image of the plurality of training images is derived from a training biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers, wherein each training image of the plurality of training images does not include any pathologist annotation; b. extracting at least a portion of the identified cell features from one or more layers of the first portion of the trained object detection engine; and c. clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of detected cells; and training a cell detection and classification engine using pathologist annotation of the one or more identified homogeneous clusters of cells.

13. The non-transitory computer-readable medium of claim 12, wherein the first portion comprises a convolutional neural network.

14. The non-transitory computer-readable medium of claim 12, wherein the first portion comprises a deep residual network.

15. The non-transitory computer-readable medium of any one of claims 12 to 14, wherein the second portion comprises a region proposal network.

16. The non-transitory computer-readable medium of any one of claims 12 to 15, wherein the object detection engine is trained using a dataset comprising a plurality of training images, wherein each training image of the plurality of training images is derived from a biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers.

17. A method of identifying clusters of cells within a sample image of a biological specimen that is stained for the presence of one or more biomarkers, the method comprising: identifying one or more homogeneous clusters of cells within a sample image of the biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers, wherein identifying one or more homogeneous clusters of cells comprises: a. detecting cells in the sample image using a trained object detection engine, wherein the trained object detection engine includes a convolutional neural network and a region proposal network, wherein the convolutional neural network is configured to identify cell features, and wherein the region proposal network is configured to detect cells based on the identified cell features, wherein the object detection engine is trained using a dataset including a plurality of training images, wherein each training image of the plurality of training images is derived from a training biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers, wherein each training image of the plurality of training images does not include any pathologist annotation; b. extracting the identified cell features from one or more layers of the convolutional neural network; and c. clustering the detected cells in the sample image based on the extracted cell features to provide one or more homogeneous clusters of detected cells; training a cell detection and classification engine using pathologist annotation of the one or more identified homogeneous clusters of cells.

18. The method of claim 17, wherein the convolutional neural network comprises a deep residual network.

19. The method of any one of claims 17-18, wherein the object detection engine is trained using a dataset including a plurality of training images, wherein each training image of the plurality of training images is derived from a biological specimen that is stained with a primary stain or stained for the presence of one or more biomarkers.

20. The method of claim 19, wherein each training image of the plurality of training images does not include any pathologist annotation.

21. The method of claim 19, wherein each training image of the plurality of training images includes at least one class label.

22. The method of any one of claims 17-21, wherein the convolutional neural network includes at least one scaling layer.

23. The method of any one of claims 17-22, wherein the clustering of the detected cells comprises performing agglomerative hierarchical clustering.

24. The method of any one of claims 17-23, further comprising: training a cell detection and classification engine using one or more class labels associated with the one or more homogeneous clusters of cells in the sample image.

25. The method of claim 24, wherein the cell detection and classification engine includes a neural network.

26. The method of claim 24, further comprising: detecting and / or classifying cells in a test image including at least one stain using the trained cell detection and classification engine.

Images