Deep learning systems and methods for joint cell and region classification in biological images

Abstract

The present disclosure relates to an automated system and method for training a multi-layer neural network to jointly and simultaneously classify cells and regions from a set of training images. The present disclosure also relates to an automated system and method for classifying cells within unlabeled images using a trained multi-layer neural network.

Description

[0001] priority

[0002] This application claims priority to U.S. Patent Application No. 62 / 596,036, filed on December 7, 2017. Background Art

[0003] Digital pathology involves scanning entire histopathology or cytopathology slides into digital images that can be interpreted on a computer screen. These images are then processed by imaging algorithms or interpreted by pathologists. In order to examine tissue sections (which are actually transparent), colored histochemical stains that selectively bind to cellular components are used to prepare tissue sections. Clinicians or computer-aided diagnosis (CAD) algorithms use the color-enhanced or stained cellular structures to identify morphological markers of disease and treat accordingly. Observation of this assay can enable a variety of processes, including disease diagnosis, evaluation of treatment response, and development of new drugs to combat disease.

[0004] Immunohistochemistry (IHC) slide staining can be used to identify proteins in cells of tissue slides and is therefore widely used in studies of different types of cells such as cancerous cells and immune cells in biological tissues. Therefore, IHC staining can be used in studies to understand the distribution and location of differentially expressed biomarkers of immune cells (such as T cells or B cells) in cancerous tissues for immune response studies. For example, tumors often contain infiltrates of immune cells that can prevent the development of tumors or facilitate the outward growth of tumors.

[0005] In situ hybridization (ISH) can be used to find the presence of genetic abnormalities or conditions, such as specifically amplifying oncogenes in cells that are morphologically malignant when observed under a microscope. In situ hybridization (ISH) uses DNA or RNA probe molecules that are antisense to the target gene sequence or transcript to detect or locate the target nucleic acid target gene in a cell or tissue sample. ISH is performed by exposing the cell or tissue sample fixed on a slice to a labeled nucleic acid probe that is capable of specific hybridization with a given target gene in a cell or tissue sample. By exposing the cell or tissue sample to multiple nucleic acid probes that have been labeled with multiple different nucleic acid tags, several target genes can be analyzed simultaneously. By utilizing labels with different emission wavelengths, a single target cell or tissue sample can be analyzed simultaneously in a single step using multiple color analysis. Summary of the invention

[0006] Traditional methods for the problem of image scoring and automatic image analysis of digital tissue sections of biological samples generally involve cell detection, cell segmentation and feature extraction for each cell in the image (or its region), followed by cell classification and region segmentation and classification. The present disclosure provides an alternative method, whereby cells and regions are jointly and simultaneously classified directly from the acquired images using a trained multi-layer neural network. In this way, the present disclosure utilizes a deep learning method in which cells are automatically detected, segmented and classified after training the multi-layer neural network. In some embodiments, both cell-level data and region-level data generated using a semi-automatic image analysis method are used as the basis for training the multi-layer neural network.

[0007] In some embodiments, existing feature-based segmentation techniques can be used to generate reliable estimates of cell contours, and can be used as the ground truth for training multi-layer neural networks. It is believed that manually performing this task would make the ground truth measurement process of all cells in the sample image (or its region) infeasible, because the sample image is usually very large and has thousands of cells. In some embodiments, the prior knowledge of feature vectors useful in cell-based classification is used as an additional input channel to further assist the multi-layer neural network in improving classification accuracy. In some embodiments, instead of using the user to measure the ground truth for each cell, an existing feature-based solution is first utilized to derive the initial cell labeling, and a pathologist or other medical professional browses the initial result set, and corrects the cells that have been misclassified. This semi-automatic solution helps to generate a large amount of cell ground truth data and regional ground truth data that are difficult to obtain in addition.

[0008] In view of the foregoing, in one aspect of the present disclosure, is a method for training a multi-layer neural network to detect and classify different cell types (e.g., tumor cells, stromal cells, lymphocytes, etc.) and tissue regions (e.g., tumor regions, lymphocyte-rich regions, stromal regions, necrotic regions, etc.) within a sample image (e.g., a sample image of a biological sample tissue with one or more stains), the method comprising: receiving sample image data; using a semi-automatic process to generate both tissue region ground truth data and cell classification ground truth data; and using the generated ground truth data to train the multi-layer neural network; wherein the generation of the ground truth data takes into account prior knowledge about whether certain types of cells may be present in certain types of tissues or regions.

[0009] In some embodiments, the generated tissue region basic fact data is manually identified by a pathologist. In some embodiments, an automatic image analysis algorithm is used to perform cell classification to provide an expected cell classification (e.g., detecting a seed center, and then classifying the detected cells by analyzing the features in the input image). In some embodiments, the expected cell classification determined by the image algorithm is checked by a pathologist, and any misclassification is corrected using the input from the pathologist (e.g., by using an image and annotation inspection and editing platform), so that the final cell classification data is provided as the basic fact data of a multi-layer neural network. In some embodiments, the tissue region classification data and the final cell classification data are combined so that each pixel in the sample image is marked with a single marker representing a tissue type and / or cell type (i.e., the marker is a vector). In some embodiments, the combination of tissue markers and cell markers excludes those tissue and cell combinations that are biologically unfeasible, i.e., a combination of cells in a specific region that will not occur in nature or have a very low probability of occurrence. In some embodiments, the sample image is a monomorphic image (i.e., an image stained with primary colors or stained due to the presence of a single biomarker indicated herein). In some embodiments, the sample image is stained with hematoxylin and eosin (H&E image). In some embodiments, the sample image is a simple immunohistochemically stained image or multiple images, whereby the multiple images are first unmixed (e.g., using an unmixing module as described herein) before generating the ground truth data. In some embodiments, the sample image is derived from a breast tissue sample.

[0010] In another aspect of the present disclosure, is a method for training a multi-layer neural network to detect and classify different cell types and regions within a sample image, the method comprising: identifying different tissue regions within a sample image of a biological sample in order to generate ground truth for training a classifier; detecting and classifying cells based on image features within the sample image to provide an expected cell classification; using input from a pathologist to correct incorrectly classified expected cell classifications to provide a final cell classification; assigning a label to each pixel within the sample image based on a combination of the identified tissue regions and the final cell classification, wherein only those cell and region combinations that are biologically feasible are assigned a label; and training the multi-layer neural network using the sample image and the label assigned to each pixel within the sample image (i.e., training the multi-layer neural network using ground truth data established at the pixel level).

[0011] In some embodiments, the identification of different regions within the sample image is performed manually by a pathologist (e.g., the pathologist may annotate portions of the image corresponding to different tissue types). In some embodiments, the identification of different tissue regions within the sample image is automatic or semi-automatic.

[0012] In some embodiments, detecting and classifying cells based on features within the sample image includes: (i) detecting cell nuclei; (ii) calculating a foreground segmentation mask, (iii) deriving features from the detected cell nuclei; and (iv) classifying the nuclei based on the derived features using a classifier (e.g., a support vector machine, etc.). In some embodiments, a segmentation mask is not generated, but DetectNET is used that assigns labels to the entire image spot instead of assigning labels to each pixel. In this case, the image spot corresponds to a spot centered on each cell (see DetectNet: A Deep Neural Network for Object Detection in DIGITS, available at https: / / devblogs.nvidia.com / parallelforall / detectnet-deep-neural-network-object-detection-digits / ).

[0013] In some embodiments, the detection and classification of cells are automatically performed using image analysis algorithms. In some embodiments, automatic cell detection and cell classification provide an expected cell classification that can be checked by a pathologist. In some embodiments, the pathologist checks the expected cell classification and determines whether the expected cell classification is accurate. In some embodiments, the pathologist corrects any inaccuracies, thereby providing a revised cell classification, which is provided together with the correctly identified expected cell classification to provide a final cell classification set to the multi-layer neural network for training.

[0014] In some embodiments, the label assigned to each pixel is a vector, the vector including the classified cell type and the identification of the tissue region or tissue type where the pixel is located in the sample image. In some embodiments, the tissue region includes a tumor region, a stroma region, a lymphocyte-rich region, and a necrotic region. In some embodiments, the cell type includes tumor cells, stroma cells, and lymphocytes. In some embodiments, biologically feasible combinations are those combinations in which cells are usually found in a particular type of tissue. By way of example only, biologically feasible combinations include: (i) tumor cells in a tumor region; (ii) stroma cells in a tumor region; (iii) lymphocytes in a tumor region; (iv) tumor cells in a stroma region; (v) stroma cells in a stroma region; (vi) lymphocytes in a stroma region; (vii) tumor cells in a lymphocyte-rich region; and (viii) lymphocytes in a lymphocyte-rich tissue. In some embodiments, the method further includes providing the unlabeled image to a trained multilayer neural network and receiving a predicted label for each pixel within the unlabeled image.

[0015] In some embodiments, the designated markers and sample images are provided to a multi-layer neural network, such as LeNet, AlexNet, ZF Net, GoogLeNet, VGGNet ResNet, VGGNetVGG16, VGG19 and DenseNet, as described herein. These neural networks are typically trained on large and publicly available image data archives (e.g., ImageNet) of natural images. In view of the fact that there are fewer training images commonly used for biological samples (such as tumor samples), a transfer learning method is adopted. In some embodiments, a "transfer learning" method can be used, in which a pre-trained network (such as those mentioned above) is used together with its learned weights to initialize the training process on a new training image set, and the pre-trained network and the weights are trained to adapt the weights to the current specific application. In some embodiments, the pre-trained network is trained on an existing public or private data set that may or may not be directly related to the application of interest and may or may not be biological in nature. In some embodiments, the weights of the final layer are randomly initialized to avoid falling into a suboptimal local minimum, while in other embodiments, the weights of the final layer are imported unchanged.

[0016] Another aspect of the present disclosure is a system comprising: a multi-layer neural network, and a ground truth training module, the ground truth training module being configured to: (i) identify different regions (e.g., tissue regions) within a sample image of a biological sample; (ii) detect and classify cells based on image features within the sample image to provide a suggested cell classification; (iii) use input from a pathologist to correct incorrectly classified suggested cell classifications to provide a final cell classification; (iv) assign a label to each pixel within the sample image based on a combination of the identified regions and the final cell classification, wherein only those cell and region combinations that are biologically feasible are assigned labels; and (v) train the multi-layer neural network using the sample image and the label assigned to each pixel within the sample image. In some embodiments, the sample image is a single-form image (e.g., an image stained with a primary color or a single biomarker). In other embodiments, the sample image is a multiplex image.

[0017] In some embodiments, the basic fact training module is semi-supervised. In some embodiments, the identification of different tissue regions includes annotating sample images with different regions. In some embodiments, the annotation is performed by a pathologist. In some embodiments, the detection and classification of cells utilize image analysis algorithms that identify cell features (e.g., morphological features, nuclear features, etc.). In some embodiments, detecting and classifying cells based on features within the sample image includes: (i) detecting cell nuclei; (ii) calculating foreground segmentation masks, (iii) deriving features from the detected cell nuclei; and (iv) using a classifier (e.g., support vector machine, etc.) to classify the nucleus based on the derived features. In some embodiments, the basic fact training module is suitable for identifying a biologically feasible cell classification and tissue region recognition combination, the combination including: (i) tumor cells in a tumor region; (ii) stroma cells in a tumor region; (iii) lymphocytes in a tumor region; (iv) tumor cells in a stroma region; (v) stroma cells in a stroma region; (vi) lymphocytes in a stroma region; (vii) tumor cells in a lymphocyte-rich region; and (viii) lymphocytes in a lymphocyte-rich region. In some embodiments, the multi-layer neural network is selected from existing convolutional (or fully convolutional) neural network architectures, such as LeNet, AlexNet, ZF Net, GoogLeNet, VGGNet, ResNet, VGG16, and DenseNet.

[0018] Another aspect of the present disclosure is a non-transitory computer-readable storage medium encoded with instructions executable by a processing resource of a computing system to enable the computing system to perform the following operations: identifying different tissue regions within a sample image of a biological sample; detecting and classifying cells based on features within the sample image to provide a suggested cell classification; assigning a label to each pixel within the sample image based on a combination of the identified tissue regions and a final cell classification, wherein only those cell and region combinations that are biologically feasible are assigned labels, and wherein the final cell classification includes a correctly classified suggested cell classification and a revised suggested cell classification, the revised suggested cell classification being received as input from a pathologist; and training the multi-layer neural network using the sample image and the label assigned to each pixel within the sample image. In some embodiments, the identification of different tissue regions includes receiving tissue region annotations from a pathologist.

[0019] Another aspect of the present disclosure is a method for classifying cells within an unlabeled image of a biological sample, the method comprising: calculating a foreground segmentation mask for the unlabeled image; identifying individual cells in the unlabeled image by filtering the unlabeled image using the calculated foreground segmentation mask; applying a multi-layer neural network to the unlabeled image, the multi-layer neural network being trained to jointly detect and classify different cell types and tissue regions, wherein the application of the trained multi-layer neural network provides a predicted label for each pixel within the unlabeled image, the predicted label corresponding to the different cell types as identified by the multi-layer neural network; and assigning a cell label to each identified individual cell. In some embodiments, the cell label is assigned by: (i) quantifying the number of pixels with each predicted label within the identified individual cell; and (ii) assigning the predicted label with the largest number as the cell label (majority vote). In some embodiments, the method further comprises generating an overlay containing descriptive markers for each labeled individual cell. In some embodiments, the method further comprises quantifying the differently labeled individual cells and calculating expression scores. In some embodiments, the quantification is performed in a predetermined field of view. In some embodiments, the predetermined field of view is identified by a pathologist based on morphology within the unlabeled image. In some embodiments, the number of lymphocytes in the tumor region or the stroma region is quantified.

[0020] Another aspect of the present disclosure is a system comprising: a cell detection module adapted to identify cells within an unlabeled image; a trained multi-layer neural network for providing a predicted label for each pixel within the unlabeled image, the predicted label corresponding to a different cell type; and a cell labeling module adapted to assign cell labels to cells identified by the cell detection module. In some embodiments, cells are labeled by: (i) quantifying the number of pixels with each predicted label within the identified cell; and (ii) designating the predicted label with the largest number as the cell label (majority vote). In some embodiments, the cell detection module identifies seeds within the unlabeled image. In some embodiments, the system further comprises a scoring module configured to count cell labels in the unlabeled image and output an expression score based on a ratio of various counted cell labels. In some embodiments, scoring is performed within a predetermined field of view within the unlabeled image. In some embodiments, the system further comprises an overlay mask module configured to overlay markers representing different cell types on the unlabeled image.

[0021] Another aspect of the present disclosure is a non-transitory computer-readable storage medium encoded with instructions executable by a processing resource of a computing system to enable the computing system to perform the following operations: detecting cells within an unlabeled image; applying a multi-layer neural network to the unlabeled image, the multi-layer neural network trained to jointly detect and classify different cell types and tissue regions, wherein the application of the trained multi-layer neural network provides a predicted label for each pixel within the unlabeled image, the predicted label corresponding to the different cell types as identified by the multi-layer neural network; and assigning a cell label to each identified individual cell. In some embodiments, the cell label is assigned by: (i) quantifying the number of pixels with each predicted label within the identified individual cell; and (ii) assigning the predicted label with the largest number as the cell label (majority vote). In some embodiments, the detection of cells within the unlabeled image includes: (i) calculating a foreground segmentation mask for the unlabeled image; and (ii) identifying individual cells in the unlabeled image by filtering the unlabeled image using the calculated foreground segmentation mask. In some embodiments, the system further comprises instructions for scoring the unlabeled image based on the specified cell markers. In some embodiments, the system further comprises instructions for generating an overlay including markers representing different cell layers and applying the overlay to the unlabeled image.

[0022] Given the increasing role of immunotherapy, there is a growing need to detect and quantify tumor infiltrating lymphocytes (TiL) in H&E images. The main problem to be addressed here is to perform joint cell detection and classification and region segmentation and classification for H&E breast cancer images in a unified deep learning framework. Although solutions are presented here only for two specific classification problems (cell-based classification and region-based classification) and a specific domain (H&E breast cancer images), the proposed solutions are applicable to a wider range of classification problems in other domains. The joint cell and region classification work aims to characterize and quantify the tumor microenvironment, identify the spatial relationships between and within tumors and cells, and characterize tumor infiltrating lymphocytes (including stromal TiL and intratumoral TiL) through quantitative (e.g., percentage) and spatial metrics.

[0023] The advantage of combining different classification problems in the same deep learning framework is that the combination of results of various classification problems can then be done automatically and coordinated by the learning framework without relying on any post-processing (ensemble learning-based methods, or parameter adjustment-based methods) to combine two sets of results (e.g., cell-level results and region-level results). For example, when we consider an image with k1 classification categories (set A) in one domain, and then for the same image but in a different domain, there are k2 classification categories (set B), then we can treat the problem as a (k1×k2) multi-class problem, and if there are certain infeasible biological combinations of elements in (set A, set B), we can discard them from the admissible set of (k1×k2) classes that can be specified by the network. Therefore, prior biological knowledge of the problem is used to generate consistent results. Compared with solving the cell and region classification problems separately and individually, the overall accuracy of both region and cell classification can be better because the iterative process of learning both at the same time tends to combine classification labels that are consistent with each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to generally understand the features of the present disclosure, reference is made to the accompanying drawings. In the accompanying drawings, the same reference numerals are used throughout to identify the same elements.

[0025] Figure 1 A representative digital pathology system including an image acquisition device and a computer system is shown.

[0026] Figure 2 Various modules that can be used in a digital pathology system or a digital pathology workflow are described.

[0027] Figure 3A Demonstrated training of a multi-layer neural network and use of the trained multi-layer neural network to provide classification results.

[0028] Figure 3BOutlines the steps for training a multi-layer neural network using sample images.

[0029] Figure 3C Shows the steps for using a trained multi-layer neural network to classify cells in unlabeled images.

[0030] Figure 4 A sample image (A), a tissue region overlay (B), and a cell classification overlay (C) are shown, where tumors are depicted in cyan, stroma in yellow, lymphocytes in red, and artifact / excluded areas in dark blue. For each pixel in the image, both region and cell labels are assigned. Also, region and cell labels are assigned color codes at the pixel level to indicate the specific classification to which they belong.

[0031] Figure 5 Shown are a sample image (A), tissue region overlay (B), and cell classification overlay (C), where tumor is depicted in cyan, stroma in yellow, lymphocytes in red, and artifact / excluded areas in dark blue.

[0032] Figure 6 A variety of biologically viable cell combinations that can be found in certain tissue types are demonstrated.

[0033] Figure 7 A sample image ground truth is shown along with the classification results from the combined classifier, where tumor is depicted in cyan, stroma is depicted in yellow, lymphocytes are depicted in red, and artifacts / excluded areas are depicted in dark blue. The top image is the original image, and the cell ground truth is overlaid on top of the original image.

[0034] Figure 8 shows the sample image ground truth and classification results. (A): Original image, (B): Cell-level classification ground truth generated by a pathologist, (C): Cell-level classification result obtained by the algorithm, (D): Region-level classification result obtained by the algorithm, where the tumor is depicted in cyan, stroma is depicted in yellow, lymphocytes are depicted in red, and artifacts / excluded areas are depicted in dark blue. DETAILED DESCRIPTION

[0035] It should also be understood that in any method claimed herein that includes more than one step or action, the order of the steps or actions of the method is not necessarily limited to the order in which the steps or actions of the method are described unless explicitly indicated to the contrary.

[0036] As used herein, the singular terms "a", "an", and "the" include plural referents unless the context clearly indicates otherwise. Similarly, the word "or" is intended to include "and" unless the context clearly indicates otherwise. The term "includes" is defined inclusively such that "including A or B" means including A, B, or A and B.

[0037] As used in this specification and claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating the listed items, "or" or "and / or" should be interpreted as inclusive, i.e., including multiple elements or at least one element in the listed elements, but also including more than one element, and (optionally) other unlisted items. Only terms that are clearly indicated to the contrary, such as "only one of..." or "exactly one of..." or "consisting of..." when used in the claims, etc. will refer to an element that just includes many elements or a list of elements. In general, when the term "or" as used herein is followed by exclusive terms such as "one of the two", "one of", "only one of" or "exactly one of", it should only be interpreted as indicating exclusive alternatives (i.e., "one or the other but not two"). Its ordinary meaning when "consisting essentially of..." is used in the claims is as used in the field of patent law.

[0038] The terms "comprising", "including", "having", etc. are used interchangeably and have the same meaning. Similarly, the terms "comprises", "includes", "has", etc. are used interchangeably and have the same meaning. Specifically, the definition of each term is consistent with the definition of "comprising" commonly found in U.S. patent law, and should therefore be interpreted as an open term, meaning "at least one of the following", and should also be interpreted as not excluding other features, limitations, aspects, etc. Thus, for example, "a device having components a, b, and c" means that the device includes at least components a, b, and c. Similarly, the phrase: "a method involving steps a, b, and c" means that the method includes at least steps a, b, and c. In addition, although the steps and processes may be outlined in a particular order herein, those skilled in the art will recognize that the ordering of the steps and processes may vary.

[0039] As used herein in the specification and claims, the phrase "at least one" in the context of referring to a list of one or more elements should be understood to mean at least one element selected from any one or more of these elements in the list of elements, but not necessarily including at least one of each element specifically listed within the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows that the elements referred to by the phrase "at least one" may optionally be present in addition to the elements specifically identified within the list of elements, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B", or equivalently "at least one of A and / or B") may refer to at least one A, optionally including more than one A, without B (and optionally including elements other than B) in one embodiment; to at least one B, optionally including more than one B, without A (and optionally including elements other than A) in another embodiment; to at least one A, optionally including more than one A and at least one B, optionally including more than one B (and optionally including other elements) in yet another embodiment; and so on.

[0040] As used herein, the terms "biological sample," "sample," or "tissue sample" refer to any sample including biological molecules (such as proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Other examples of organisms include mammals (such as humans; beasts such as cats, dogs, horses, cows, and pigs; and laboratory animals such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include tissue samples (such as tissue sections and tissue biopsies), cell samples (such as cytological smears (such as Pap smears or blood smears) or cell samples obtained by microdissection), or cell components, fragments, or organelles (such as obtained by lysing cells and separating their components by centrifugation or other means). Other examples of biological samples include blood, serum, urine, semen, fecal matter, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, earwax, milk, vaginal fluid, saliva, swab (such as oral swab) or any material containing biomolecules derived from the first biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor (e.g., a homogenized or liquefied sample) or a portion of a tumor obtained from a subject.

[0041] As used herein, the term "biomarker" or "marker" refers to a measurable indicator of certain biological states or conditions. Specifically, a biomarker can be a protein or peptide (such as a surface protein) that can be specifically stained and indicates the biological characteristics of a cell (e.g., a cell type or a physiological state of a cell). An immune cell marker is a biomarker that selectively indicates a characteristic associated with a mammalian immune response. Biomarkers can be used to determine how the body responds to treatment of a disease or condition, or whether a subject is susceptible to a disease or condition. In the case of cancer, a biomarker refers to a biological substance that indicates the presence of cancer in the body. A biomarker can be a molecule secreted by a tumor or a specific response of the body to the presence of cancer. Genetic, epigenetic, proteomics, glycomics and imaging biomarkers can be used for cancer diagnosis, prognosis and epidemiology. Such biomarkers can be measured in non-invasively collected biological fluids (such as blood or serum). Several gene- and protein-based biomarkers have been used in patient care, including but not limited to AFP (liver cancer), BCR-ABL (chronic myeloid leukemia), BRCA1 / BRCA2 (breast cancer / ovarian cancer), BRAF V600E (melanoma / colorectal cancer), CA-125 (ovarian cancer), CA19.9 (pancreatic cancer), CEA (colorectal cancer), EGFR (non-small cell lung cancer), HER-2 (breast cancer), KIT (gastrointestinal stromal tumor), PSA (prostate-specific antigen), S100 (melanoma), and many others. Biomarkers can be used as diagnostics (to identify early-stage cancer) and / or prognostics (to predict the aggressiveness of the cancer and / or to predict a subject's response to a particular treatment and / or the likelihood of cancer recurrence).

[0042] As used herein, the term "field of view (FOV)" refers to a portion of an image having a predetermined size and / or shape. In some embodiments, the FOV is an area in a digital image for further manual or automatic inspection and analysis. The FOV can be automatically or manually selected by analyzing some features of the digital image, such as by evaluating the intensity values ​​of the pixels of the digital image. PCT / EP2015 / 062015 describes a method for generating a specific FOV based on certain predetermined criteria or characteristics in an image (e.g., for a biological sample stained with more than two stains, identifying an area of ​​the image that includes only two stains).

[0043] As used herein, the term "image data" encompasses raw image data or pre-processed image data acquired from a biological tissue sample (such as by means of an optical sensor or a sensor array). In particular, the image data may include a pixel matrix.

[0044] As used herein, the term "immunohistochemistry" refers to a method for determining the presence or distribution of an antigen in a sample by detecting the interaction of the antigen with a specific binding agent (such as an antibody). The sample is contacted with the antibody under conditions that allow antibody-antigen binding. Antibody-antigen binding can be detected by means of a detectable label that is bound to the antibody (direct detection) or by means of a detectable label that is bound to a second antibody that specifically binds to the first antibody (indirect detection).

[0045] As used herein, the term "mask" refers to a derivative of a digital image, wherein each pixel in the mask is represented as a binary value, such as "1" or "0" (or "true" or "false"). By covering the digital image with the mask, all pixels of the digital image that map to mask pixels of a particular one of the binary values ​​are hidden, removed, or otherwise ignored or filtered out in further processing steps applied to the digital image. For example, a mask can be generated from an original digital image by designating all pixels of the original image having intensity values ​​above a threshold as true and false otherwise, thereby creating a mask that can filter out all pixels that would be covered by "false" mask pixels.

[0046] As used herein, the term "multi-channel image" or "multiple images" covers digital images acquired from biological tissue samples, in which different biological structures (such as cell nuclei and tissue structures) are simultaneously stained with specific fluorescent dyes, quantum dots, chromogens, etc., each of which emits fluorescence or is detectable in a different spectral band, thereby constituting one of the channels of the multi-channel image.

[0047] Overview

[0048] The present disclosure relates to an automated system and method for training a multi-layer neural network to jointly and simultaneously classify cells and regions within a sample image. The present disclosure also relates to an automated system and method for classifying cells within an unlabeled image using a trained multi-layer neural network.

[0049] At least some embodiments of the present disclosure relate to computer systems and methods for analyzing digital images captured from biological samples, including tissue samples stained with one or more primary colors (e.g., hematoxylin and eosin (H&E)) and one or more detection probes (e.g., probes containing specific binding entities that facilitate labeling of targets within the sample). Although the examples herein may refer to the use of specific tissues and / or specific stains or detection probes for detecting certain markers (and therefore detecting diseases), those skilled in the art will recognize that different tissues and different stains / detection probes may be used to detect different markers and different diseases.

[0050] Figure 1and Figure 2 , a digital pathology system 200 for imaging and analyzing a sample is shown. The digital pathology system 200 may include an imaging device 12 (e.g., a device having a means for scanning a microscope slide carrying a sample) and a computer 14, whereby the imaging device 12 and the computer may be communicatively coupled together (e.g., directly or indirectly via a network 20). The computer system 14 may include a desktop computer, a laptop computer, a tablet computer, etc., a digital electronic circuit system, firmware, hardware, a memory 201, a computer storage medium (240), a computer program or instruction set (e.g., where the program is stored in a memory or storage medium), one or more processors (209) (including a programmed processor), and any other hardware, software, or firmware modules or combinations thereof (such as further described herein). For example, Figure 1 The computing system 14 shown in the figure may include a computer having a display device 16 and a housing 18. The computer system may store the digital image in binary form (locally such as in a memory, a server, or another network connection device). The digital image may also be divided into a matrix of pixels. A pixel may include a digital value having one or more bits defined by a bit depth. Those skilled in the art will recognize that other computer devices or systems may be utilized, and the computer system described herein may be communicatively coupled to additional components, such as sample analyzers, microscopes, other imaging systems, automatic slice preparation equipment, etc. These additional components and some of the various computers, networks, etc. that may be utilized are further described herein.

[0051] The imaging device 12 (or other image sources including pre-scanned images stored in memory) may generally include, but are not limited to, one or more image capture devices. The image capture device may include, but is not limited to, a camera (e.g., an analog camera, a digital camera, etc.), an optical device (e.g., one or more lenses, a sensor focusing lens group, a microscope objective lens, etc.), an imaging sensor (e.g., a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, etc.), a film, etc. In a digital embodiment, the image capture device may include multiple lenses that cooperate to demonstrate instant focus. The image sensor (e.g., a CCD sensor) can capture a digital image of the sample. In some embodiments, the imaging device 12 is a bright field imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. The digitized tissue data may be generated, for example, by an image scanning system, such as the VENTANA iScanHT scanner of VENTANA MEDICAL SYSTEMS (Tucson, Arizona) or other suitable imaging equipment. Additional imaging devices and systems are further described herein. Those skilled in the art will recognize that the digital color image acquired by the imaging device 12 is generally composed of basic color pixels. Each color pixel can be encoded on three digital components, each comprising the same number of bits, each component corresponding to a primary color, typically red, green or blue, also denoted by the term "RGB" components.

[0052] Figure 2 An overview of the various modules utilized in the presently disclosed digital pathology system is provided. In some embodiments, the digital pathology system employs a computer device or computer-implemented method having one or more processors 209 and at least one memory 201 storing non-transitory computer-readable instructions to be executed by the one or more processors so that the one or more processors execute instructions (or stored data) in at least one of a multi-layer neural network 220 and a ground truth training module 210 or a testing module 230.

[0053] In some embodiments, the ground truth training module 210 is adapted to receive one or more sample images from the imaging module 202, which is communicatively coupled to the imaging device 12 or the storage module 240. After receiving the image data, the ground fact training module 210 can execute one or more modules (or submodules) (i.e., modules 203 to 207) so that the ground fact data can be generated and provided to the multi-layer neural network 220 for training. In some embodiments, the ground fact training module 210 can first execute the region identification module 203 so that the region or tissue region within the sample image can be classified. In some embodiments, the region identification module 203 receives input from a pathologist or other trained medical professional, and the received information includes image annotations. Subsequently, the ground fact training module 210 can execute the cell detection module 204 to identify cells within the sample image and classify these cells using the cell classification module 205 after performing cell detection. The classification adjustment module 206 can be run so that the cell classification results from the cell classification module 205 can be adjusted, such as by using input from a pathologist or other trained medical professional. Finally, a data combination module 207 can be used to combine the regional classification data from the region identification module 203 and the cell classification from the cell classification module 205 and / or the classification adjustment module to provide a single label for each pixel within the sample image. The pixel labeling data (i.e., the generated basic fact information) can then be provided to the multi-layer neural network 220 together with the sample image so that the multi-layer neural network can be trained. In some embodiments, the basic fact training module 210 is semi-supervised. In some embodiments, the sample image received as input from the image module 202 is a single image. In other embodiments, the sample image received as input from the imaging module is a multiple image.

[0054] In some embodiments, the testing module 230 is configured to receive one or more unlabeled images from the imaging module 202. After receiving the unlabeled image data, the testing module 230 can execute one or more modules (or submodules), namely modules 204 and 208. In some embodiments, the testing module is adapted to first execute the instructions in the cell detection module 204 so that cells within the unlabeled image can be identified. After cell identification, the unlabeled image is provided to the trained multi-layer neural network 220 so that predicted cell labels can be assigned to all pixels within the unlabeled image. The testing module 210 then uses the cell labeling module 208 to assign labels to each identified cell. In some embodiments, additional modules (not depicted) can be run by the testing module (e.g., a scoring module) to provide expression scores based on cell labels, and can be run by the overlay generation module to provide identification marks as overlays on the unlabeled image.

[0055] Those skilled in the art will also recognize that additional modules can be incorporated into the workflow and used in conjunction with the ground truth training module 210 or the testing module 230. In some embodiments, as described in more detail herein, an image processing or preprocessing module can be run to apply certain filters to the acquired images or to identify certain histological and / or morphological structures or features within the tissue sample. Similarly, a region of interest selection module can be used to select specific portions of a sample image or test for analysis. Additionally, an unmixing module can be run to unmix multiple images prior to further processing.

[0056] Image acquisition module

[0057] In some embodiments, and as an initial step and with reference to Figure 2 , the digital pathology system 200 runs an image acquisition module 202 to capture an image (e.g., using an imaging device 12) or image data (i.e., the image can be a single image or a multiple image) of a biological sample with one or more stains. In some embodiments, the image received or acquired is an RGB image or a multispectral image. In some embodiments, the captured image is stored in the memory 201 (or stored in the storage module 240). Ground truth data can be derived from the acquired image to train the neural network, that is, the sample image or sample image data can be acquired or retrieved to generate ground truth data (steps 300 to 302). Similarly, unlabeled images or unlabeled image data can be acquired or retrieved, and the cells in the unlabeled image or unlabeled image data can be classified using a trained multi-layer neural network (steps 303 to 305). Any of the sample image or unlabeled image is an entire slice image or any portion thereof (e.g., a predetermined field of view).

[0058] In some embodiments, instead of a captured original image, any set of optional pre-processed images from the captured original image can be used, either as an independent input image or in combination with the captured original image. Thus, as described herein, similar pre-processing steps can be used when applying a trained network to unlabeled images.

[0059] An image or image data (used interchangeably herein) can be acquired (such as in real time) using imaging device 12. In some embodiments, the image is acquired from a microscope or other instrument capable of capturing image data of a microscope slide carrying a sample, as noted herein. In some embodiments, the image is acquired using a 2D scanner, such as a scanner capable of scanning image blocks, or a line scanner capable of scanning an image in a line-by-line manner, such as a VENTANA DP 200 scanner. Alternatively, the image can be an image that has been previously acquired (e.g., scanned) and stored in memory 201 (or, for that matter, retrieved from a server via network 20).

[0060] In some embodiments, the system receives at least one image as an input for deriving basic fact data. In general, the images received as input each include a signal corresponding to a stain (including, chromogenic, fluorophore, quantum dot, etc.). In some embodiments, the at least one image has been stained with at least one primary color (hematoxylin or eosin). In other embodiments, the at least one image (referred to herein as a "biomarker" image) has been stained in at least one of the IHC assays or ISH assays for identifying a specific biomarker. In some embodiments, a plurality of single-form images are used as input, and these images can be derived from continuous tissue sections (i.e., continuous sections derived from the same xenograft tissue block). In some embodiments, the input image can be a multiple image, i.e., an image stained for a plurality of different markers in a multiple assay according to methods known to those skilled in the art.

[0061] Typical biological samples are processed on a staining / determination platform where staining agents are applied to the sample. In some embodiments, the sample handling device is an automatic device, such as the BENCHMARK XT instrument, SYMPHONY instrument, and BENCHMARK ULTRA instrument sold by Ventana Medical Systems. Ventana Medical Systems is the assignee of many U.S. patents that disclose systems and methods for performing automatic analysis, including U.S. Patents 5,650,327, 5,654,200, 6,296,809, 6,352,861, 6,827,901 and 6,943,029, and U.S. Publication Patent Application Nos. 20030211630 and 20040052685. Alternatively, the sample can be processed manually.

[0062] Examples of commercially available H&E stainers include the VENTANA SYMPHONY (single slide stainer) and VENTANA HE 600 (single slide stainer) series H&E stainers from Roche; the DakoCoverStainer (batch stainer) from Agilent Technologies; the Leica ST4020 Mini Linear Stainer (batch stainer), Leica ST5020 Multi-Purpose Stainer (batch stainer), and Leica ST5010 Autostainer XL series (batch stainer) H&E stainers from Leica Biosystems Nussloch GmbH. There are other commercial products on the market that are suitable for use as staining / assay platforms, one example of which is the Discovery TM .

[0063] The camera platform may also include a brightfield microscope (one example is the VENTANA iScan HT product from Ventana Medical Systems) or any microscope with one or more objectives and a digital imager and a set of spectral filters. Other techniques for capturing images at different wavelengths may be used. Additional camera platforms suitable for imaging stained biological samples are known in the art and commercially available from companies such as Zeiss, Canon, Applied Spectral Imaging, and others, and such platforms are readily adapted for use with the systems, methods, and apparatus of the subject disclosure.

[0064] As will be appreciated by those skilled in the art, biological samples may be stained for different types of nuclear and / or cell membrane biomarkers. Methods for staining tissue structures and guidelines for selecting stains suitable for various purposes are discussed, for example, in "Sambrook et al., Molecular Cloning: A Laboratory Manual, Cold Spring Harbor Laboratory Press (1989)" and "Ausubel et al., Current Protocols in Molecular Biology, Greene Publishing Associates and Wiley-Intersciences (1987)".

[0065] As a non-limiting example, and in the case of detecting breast cancer, in some embodiments, the tissue sample is stained in the IHC assay for the presence of one or more biomarkers, the one or more biomarkers including estrogen receptor markers, progesterone receptor markers, Ki-67 markers or HER2 markers. Thus, in some embodiments, the biomarker image used as input is an IHC image including a signal corresponding to the presence of at least one of an estrogen receptor (ER) marker, a progesterone receptor (PR) marker, a Ki-67 marker, or a HER2 marker (a signal corresponding to a stain that may be, for example, chromogenic or fluorescent). In some embodiments, the sample can be analyzed to detect or measure the presence of ER, HER2, Ki-67 and PR proteins in the sample, such as qualitative or quantitative measurements. As another non-limiting example, and in the case of detecting non-small cell lung cancer, in some embodiments, the tissue sample is stained for the presence of one or more biomarkers (including PD-L1 biomarkers) in the IHC assay. Thus, in some embodiments, the biomarker image used as input is an IHC image that includes signals corresponding to the presence of the PD-L1 marker, the CD3 marker, and the CD8 marker.

[0066] No matter the image acquired is used as sample image to measure the basic facts or as unlabeled image to classify, image may contain information that needs mask.In some embodiments, input image is optionally masked with tissue mask module as described herein.In some embodiments, input image is masked so that only tissue area is present in image.In some embodiments, tissue area mask is generated to mask non-tissue area by tissue area.In some embodiments, tissue area mask can be created by identifying tissue area and automatically or semi-automatically (that is, with minimized user input) excluding background area (for example, corresponding to the area of ​​full slice image of glass without sample, such as only having the area of ​​white light from imaging source).As those skilled in the art will recognize, except masking non-tissue area by tissue area, tissue mask module can also mask other interest areas as needed, such as, be identified as belonging to a certain tissue type or belonging to a part of the tissue of suspected tumor area.In some embodiments, segmentation technique is used to generate tissue area mask image by masking tissue area by non-tissue area in input image. Suitable segmentation techniques are such techniques as are known in the art (see Digital Image Processing, 3rd Edition, Rafael C. Gonzalez, Richard E. Woods, Chapter 10, page 689 and Handbook of Medical Imaging, Processing and Analysis, Isaac N. Bankman Academic Press, 2000, Chapter 2). Additional information and examples related to generating tissue region masks are disclosed in PCT / EP / 2015 / 062015 entitled "An Image Processing Method and System for Analyzing a Multi-Channel Image Obtained from a Biological Tissue Sample Being Stained by Multiple Stains".

[0067] Training a multi-layer neural network

[0068] refer to Figure 3A As shown, training of the multi-layer neural network includes the following steps: (i) receiving sample image data (step 300) (e.g., using image acquisition module 202); (ii) generating ground truth data from the sample image data (step 301) (e.g., using ground truth training module 210; and (iii) training the multi-layer neural network (220) using the sample image data and the generated ground truth data (step 302).

[0069] Given a sample image, different tissue types and / or cell types can be identified in the sample image. For example, the sample image can be segmented into different types of regions, including tumor regions, lymphocyte-rich regions, stroma regions, and necrotic regions. In addition, cells within the sample image can be detected and classified as tumor, lymphocyte, stroma cell, etc. In view of this, generating ground truth data (step 301) includes several steps, such as Figure 3B As outlined in , the steps include: (a) identifying tissue regions in sample image data (step 311); (b) detecting and classifying cells in the sample image data (step 312); (c) using input received from a medical professional (e.g., a pathologist or other medical expert trained to identify and classify different cell types in a biological sample) to adjust or correct the cell classification (step 313); and (d) combining tissue region identification and corrected / adjusted cell classification at the pixel level so that each pixel includes a single label (e.g., a vector) (steps 314 and 315). After generating the ground truth data (steps 311 to 315), the ground truth data is fed to a multi-layer neural network (220) so that the neural network can be trained (step 316). Given the input from the pathologist or medical expert, the generation of the ground truth data is semi-supervised. Each of these steps will be described in further detail herein.

[0070] Region Identification

[0071] Then, basic fact data is generated for each region within the sample image, such as by using the region identification module 203. For example, basic fact data related to various tissue regions (e.g., normal tissue, tumor tissue, necrotic tissue, lymphocyte-rich lymphocyte region, stromal tissue, etc.) can be generated from the H&E sample image. Of course, those skilled in the art will recognize that the identified region can be a peritumoral region, an immune marginal region, a vascular infiltration, a neural infiltration, an acne-like region, a plasma cell or neutrophil-rich region, an activated matrix, hemorrhage, a normal acinus or duct, etc., and the type of the identified region depends on the type of biological sample. In some embodiments, the identified portion can be a representative overexpression tumor region of a specific biomarker (e.g., a specific IHC marker).

[0072] In some embodiments, various tissue regions within the sample image may be manually identified by a pathologist or medical professional. For example, a pathologist examining a biological sample may annotate different regions or tissue regions by annotating the sample image. Assuming that the tissue region ground truth data is manually generated, the generation of the ground truth data (step 301) may be considered semi-automatic.

[0073] In some embodiments, image annotations can be drawn using annotation tools provided in a viewer application (e.g., VIRTUOSO / VERSO(TM) or similar software) and can be drawn at any magnification or resolution. In some embodiments, a pathologist manually annotates the boundaries of a particular region (e.g., a tumor), which can be visually identified by different colors, textures, histological structures, and spatial arrangements of cells. In some embodiments, the annotation boundaries are converted to solid (e.g., closed) polygons, each of which corresponds to a single structure (e.g., a single tumor nest).

[0074] Alternatively or in addition, image analysis operations can be used to automatically detect tumor regions or other regions can be detected using automatic image analysis operations (such as segmentation, thresholding, edge detection, etc.), and the FOV is automatically generated based on the detected regions. In some embodiments, image segmentation and tissue type recognition are performed according to the methods described in PCT / EP2015 / 051302, entitled "Adaptive Classification for Whole Slide Tissue Segmentation," filed on January 23, 2015 (published as WO 2015 / 113895). In general, WO 2015 / 113895 describes segmenting a tumor region from other regions in an image via operations related to classification of the region, assigning at least one of a high confidence score and a low confidence score to the classified grid points, modifying a database of known features of tissue types based on the grid points assigned the high confidence scores, and generating a modified database, and reclassifying the grid points assigned the low confidence scores based on the modified database to segment tissue (e.g., identifying tissue regions in an image), wherein the classification of the region includes: identifying grid points in a tissue image, classifying the grid points into one of multiple tissue types, and generating classified grid points based on a database of known features of tissue types.

[0075] Regardless of whether the tissue regions are identified via a manual process or an automated process, a tissue region mask of the specific tissue region may be generated, such as Figure 4 Panel (B) and Figure 5 As shown in panel (B). A method for generating a tissue region mask is described in U.S. Patent Application Publication No. 2017 / 0154420.

[0076] Cell detection and sorting

[0077] After image acquisition and / or unmixing, the input image or unmixed image channel image is provided to the cell detection module 204 to detect cells, and then provided to the cell classification module 205 to classify cells and / or nuclei (step 300). The processes and algorithms described herein can be suitable for identifying and classifying various types of cells or cell nuclei based on features within the input image, including identifying and classifying tumor cells, non-tumor cells, stromal cells, and lymphocytes. Those skilled in the art will recognize that the type of cell identified may depend on the type and staining of the sample image, for example, in the case of immune cells, different types of immune cells including CD3 and CD8 may be detected and classified. Similarly, the cell classification can be marker-positive tumor cells or marker-negative tumor cells. As another example, in an IHC image stained with an immune marker (such as CD3 or CD8 lymphocytes), cells with membrane marker staining (DAB) are positive, while counterstained lymphocytes are marker-negative. One skilled in the art will recognize that different tissues are composed of different cell types, for example, papillae in papillary thyroid carcinoma, glands in adenocarcinoma, and multinucleated giant cells in metaplastic carcinoma.

[0078] Those skilled in the art will recognize that the nucleus, cytoplasm and membrane of a cell have different properties, and differently stained tissue samples may reveal different biological characteristics. In fact, those skilled in the art will recognize that certain cell surface receptors may have a staining pattern that is localized to the membrane or to the cytoplasm. In this way, the "membrane" staining pattern is different from the "cytoplasm" staining pattern in analysis. Similarly, the "cytoplasm" staining pattern and the "nucleus" staining pattern are different in analysis. Each of these different staining patterns can be used as a feature of identifying cells and / or nuclei. For example, stromal cells may be strongly stained by FAP, while tumor epithelial cells may be strongly stained by EpCAM, and cytokeratin may be stained by panCK. Therefore, by utilizing different staining agents, different cell types can be distinguished and differentiated during image analysis to provide a classification solution.

[0079] Methods for identifying, classifying, and / or scoring nuclei, cell membranes, and cytoplasm in an image of a biological sample having one or more stains are described in U.S. Patent No. 7,760,927 ("the '927 patent"). For example, US 7,760,927 describes an automatic method for simultaneously identifying a plurality of pixels in an input image of biological tissue stained with a biological marker, the method comprising: considering a first color plane of a plurality of pixels in a foreground of the input image for simultaneously identifying cytoplasm pixels and cell membrane pixels, wherein the input image has been processed to remove a background portion of the input image and to remove counterstained components of the input image; determining a threshold level between cytoplasm pixels and cell membrane pixels in the foreground of a digital image; and using the determined threshold level, determining simultaneously with a pixel selected from the foreground and its eight neighboring pixels whether the selected pixel is a cytoplasm pixel, a cell membrane pixel, or a transition pixel in the digital image.

[0080] In some embodiments, tumor nuclei are automatically identified by first identifying candidate nuclei and then automatically distinguishing tumor nuclei from non-tumor nuclei. Many methods for identifying candidate nuclei in tissue images are known in the art. For example, automatic candidate nucleus detection can be performed by applying a radial symmetry-based method (such as to an unmixed hematoxylin image channel or a biomarker image channel) (see Parvin, Bahram et al., "Iterative voting for inference ofstructural saliency and characterization of subcellular events", Image Processing, IEEE Transactions on 16.3 (2007): 615-623).

[0081] More specifically, in some embodiments, the image received as input is processed to detect the nuclear center (seed) and / or segment the nuclear. For example, instructions can be provided to detect the nuclear center (noted above) based on radial symmetric voting using Parvin's technology. In some embodiments, the nuclear is detected using radial symmetry to detect the center of the nuclear and then the nuclear is classified based on the staining intensity around the cell center. In some embodiments, as described in the co-pending patent application WO / 2014 / 140085A1 of the common assignment, a nuclear detection operation based on radial symmetry is used. For example, the image size can be calculated within the image, and one or more votes at each pixel are accumulated by adding the sum of the sizes within the selected area. Mean shift clustering can be used to find the local center in the region, where the local center represents the actual nuclear position. Nuclear detection based on radial symmetric voting is performed on color image intensity data and makes explicit use of the prior domain knowledge that the nuclear is an elliptical spot with different sizes and eccentricities. To achieve this, together with the color intensity in the input image, image gradient information is also used for radial symmetric voting and combined with the adaptive segmentation process to accurately detect and locate the cell nucleus. As used herein, "gradient" is the intensity gradient of a pixel calculated for a specific pixel, for example, by considering the intensity value gradient of a group of pixels around the specific pixel. Each gradient can have a specific "orientation" relative to a coordinate system defined by two orthogonal edges of a digital image for x and y axes. For example, nuclear seed detection involves defining a seed as a point assumed to be located within the nucleus and used as a starting point for locating the nucleus. The first step is to detect the seed points associated with each nucleus using a highly robust method based on radial symmetry to detect elliptical spots, structures similar to the nucleus. The radial symmetry method uses a kernel-based voting process to operate on the gradient image. A voting response matrix is ​​created by processing each pixel that accumulates votes through a voting kernel. The kernel is based on the gradient direction calculated at a specific pixel and the expected minimum and maximum kernel size and the range of the voting kernel angle (usually in the range of [π / 4, π / 8]). In the resulting voting space, the local maximum position with a voting value higher than a predetermined threshold can be solved as a seed point. Later, foreign seeds can be discarded during subsequent segmentation or classification processes. Other methods are discussed in U.S. Patent Publication No. 2017 / 0140246.

[0082] Other techniques known to those of ordinary skill in the art can be used to identify the nucleus. For example, the image size can be calculated based on a specific image channel of one of the H&E or IHC images, and each pixel around the specified size can be assigned a plurality of votes based on the sum of the sizes in the area around the pixel. Alternatively, a mean shift clustering operation can be performed to find the local center of the actual position of the representation nucleus in the voting image. In other embodiments, nuclear segmentation can be used to segment the entire nucleus based on the now known nuclear center via morphological operations and local thresholding. In yet other embodiments, model-based segmentation can be used to detect the nucleus (i.e., the shape model of the nucleus is learned from the training data set and used as prior knowledge to segment the nucleus in the test image).

[0083] In some embodiments, the nuclear segmentation is then performed using a threshold value calculated separately for each nuclear segmentation. For example, the Otsu method can be used to segment in the region around the identified nuclear, because it is believed that the pixel intensity in the nuclear region is variable. As will be understood by those of ordinary skill in the art, the Otsu method is used to determine the optimal threshold by minimizing the intra-class variance and is known to those skilled in the art. More specifically, the Otsu method is used to automatically perform cluster-based image thresholds or to reduce grayscale images to binary images. The algorithm assumes that the image contains two types of pixels (foreground pixels and background pixels) that follow a bimodal histogram. Then, the optimal threshold value separating the two classes is calculated so that their combined expansion (intra-class variance) is minimal or equivalent (because the sum of paired square distances is constant), so their inter-class variance is the largest.

[0084] In some embodiments, the system and method further include automatically analyzing the spectrum and / or shape characteristics of the nucleus identified in the image to identify the nucleus of non-tumor cells. For example, spots can be identified in the first digital image in the first step. As used herein, "spots" can be, for example, regions of a digital image in which some attributes (such as intensity or grayscale values) are constant or vary within a specified range of values. In a sense, all pixels in the spots can be considered similar to each other. For example, a differential method based on the derivative of a position function on a digital image and a method based on local extrema can be used to identify spots. Nuclear spots are spots whose pixels and / or their contour shapes indicate that the spots may be generated by a nucleus stained with a first stain. For example, the radial symmetry of the spots can be evaluated to determine whether the spots should be identified as nuclear spots or any other structure, such as staining artifacts. For example, in the case where the spots have a long shape and are not radially symmetrical, the spots cannot be identified as nuclear spots, but are identified as staining artifacts. Depending on the embodiment, spots identified as "nuclear spots" can represent a group of pixels that are identified as candidate nuclei and can be further analyzed to determine whether the nuclear spots represent nuclei. In some embodiments, any kind of nuclear spots are directly used as "identified nuclei". In some embodiments, the identified nuclei or nuclear spots are filtered to identify nuclei that do not belong to biomarker-positive tumor cells, and the identified non-tumor nuclei are removed from the list of identified nuclei or the nuclei are not added to the list of identified nuclei from the beginning. For example, the other spectral and / or shape characteristics of the identified nuclear spots can be analyzed to determine whether the nucleus or nuclear spots are the nuclei of tumor cells. For example, the nuclei of lymphocytes are larger than the nuclei of other tissue cells (e.g., lung cells). In the case where tumor cells are derived from lung tissue, the nuclei of lymphocytes are identified by identifying all nuclear spots whose minimum size or diameter is significantly larger than the average size or diameter of normal lung cell nuclei. The identified nuclear spots related to the nuclei of lymphocytes can be removed from the group of identified nuclei (i.e., "filtered out"). By filtering out the nuclei of non-tumor cells, the accuracy of the method can be improved. Depending on the biomarker, non-tumor cells can also express biomarkers to a certain extent, and therefore can generate intensity signals that are not derived from tumor cells in the first digital image. By identifying and filtering out nuclei that do not belong to tumor cells from all identified nuclei, the accuracy of identifying biomarker-positive tumor cells can be improved. These and other methods are described in U.S. Patent Publication 2017 / 0103521. In some embodiments, once a seed is detected, a local adaptive threshold method can be used and spots around the detected center can be created. In some embodiments, other methods can also be combined, such as a marker-based watershed algorithm can also be used to identify nuclear spots around the detected nuclear center.These and other methods are described in co-pending application PCT / EP2016 / 051906 (published as WO 2016 / 120442).

[0085] After the nuclei are detected, features (or metrics) are derived from within the input image. Derivation of metrics from nuclei features is well known in the art, and any nuclei features known may be used in the context of the present disclosure. Non-limiting examples of metrics that may be calculated include:

[0086] (A) Metrics derived from morphological features

[0087] As used herein, "morphological features" are features that, for example, indicate the shape or size of a nucleus. Without wishing to be bound by any particular theory, it is believed that morphological features provide some important information about the size and shape of a cell or its nucleus. For example, morphological features can be calculated by applying various image analysis algorithms to pixels contained in or around nuclear spots or seeds. In some embodiments, morphological features include area, minor axis length and major axis length, perimeter, radius, firmness, etc. At the cellular level, such features are used to classify the nucleus as belonging to healthy or diseased cells. At the tissue level, the statistical data of these features in the tissue can be used to classify the tissue as being diseased.

[0088] (B) Deriving metrics from apparent features

[0089] For example, as used herein, "apparent features" are features calculated for a particular nucleus by comparing pixel intensity values ​​of pixels contained in or around a nuclear spot or seed for identifying a nucleus, whereby the compared pixel densities are derived from different image channels (e.g., background channels, channels for staining biomarkers, etc.). In some embodiments, a metric derived from an apparent feature can be calculated based on percentile values ​​of pixel intensity (e.g., 10th, 50th, and 95th percentile values) and percentile values ​​of gradient magnitude calculated from different image channels. For example, first, multiple (P) X percentile values ​​(X=10, 50, 95) of pixel values ​​of each of multiple image channels IC (e.g., three channels: HTX, DAB, brightness) within a nuclear spot representing a nucleus of interest are identified. Calculating an apparent feature metric may be advantageous because the derived metric can describe the properties of the nuclear region as well as describe the membrane region around the nucleus.

[0090] (C) Metrics derived from background features

[0091] For example, "background features" are features indicating the presence of appearance and / or staining in the cytoplasm, and cell membrane features of cells including nuclei from which background features are extracted from the image. Background features and corresponding metrics of the nuclei and corresponding cells depicted in the digital image can be calculated, for example, by the following various operations: identifying nuclear spots or seeds representing the nuclei; analyzing the pixel area directly adjacent to the calculated identified cell group (e.g., a 20-pixel (about 9 microns) thick band structure around the nuclear spot boundary), so that the appearance and staining presence are captured in the cytoplasm and membrane of cells with such nuclei and in the area directly adjacent to the cell. These metrics are similar to nuclear appearance features, but are calculated in a band structure about 20 pixels (about 9 microns) thick around each nuclear boundary, so that the appearance and staining presence are captured in the cytoplasm and cell membrane of cells with identified nuclei and in the area directly adjacent to the cell. Without wishing to be bound by any particular theory, the band structure size is selected because it is believed that it captures a sufficient amount of background tissue area around the nucleus that can be used to provide useful information for nuclear differentiation. These features are similar to those disclosed by "J. Kong et al., "A comprehensive framework for classification of nucleiin digital microscopy imaging: An application to diffuse gliomas," ISBI, 2011, pp. 2128-2131." It is believed that these features can be used to determine whether the surrounding tissue is matrix or epithelial cells (such as in H&E stained tissue samples). Without wishing to be bound by any particular theory, it is believed that these background features also capture membrane staining patterns, which are useful when staining tissue samples using appropriate membrane stains.

[0092] (D) Metrics derived from color.

[0093] In some embodiments, the metrics derived from color include color ratios (R / (R+G+B)) or color principal components. In other embodiments, the metrics derived from color include local statistics (mean / median / variance / standard deviation) for each color in the color and / or color intensity correlations in a local image window.

[0094] (E) Metrics derived from intensity features

[0095] The set of neighboring cells with certain attribute values ​​is placed between black and white shades of gray cells represented in the histopathology slice image. The correlation of color features defines instances of size classes, so the intensity of such a color cell determines the affected cell from the cluster of dark cells around it.

[0096] (F) Metrics derived from texture features

[0097] Examples of texture features and methods of deriving them are described in PCT Publication Nos. WO / 2016 / 075095 and WO / 2016 / 075096.

[0098] (G) Metrics derived from spatial features

[0099] In some embodiments, the spatial features include: the local density of cells; the average distance between two adjacent detected cells; and / or the distance from a cell to a segmented region.

[0100] (H) Metrics derived from kernel features

[0101] Those skilled in the art will also appreciate that metrics can also be derived from nuclear features. The calculation of such nuclear features is described by Xing et al., "Robust Nucleus / Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review," IEEE Rev Biomed Eng 9, 234-263, January 2016.

[0102] Of course, other features known to those of ordinary skill in the art may be considered and used as a basis for feature calculations.

[0103] As another example, cells can be classified as lymphocytes, such as described in PCT Publication No. WO / 2016 / 075096. In particular, PCT Publication No. WO / 2016 / 075096 describes a computer-implemented method for classifying cells within an image of a stained tissue sample for the presence of a PD-L1 biomarker in an IHC assay, the method comprising: calculating a nuclear feature metric from nuclear features within the tissue sample image; calculating a context information metric based on the image of the nucleus of interest and the tissue sample; and using a combination of the nuclear feature metric and the context information metric (as an input to a classifier) ​​to classify cells within the tissue sample image, wherein the cells are classified as at least one of a positive immune cell, a positive tumor cell, a negative immune cell, and a negative tumor cell or other cell. In some embodiments, the method further includes the step of creating a foreground segmentation mask to identify a single nucleus within the cell. The disclosure further describes that in the context of PD-L1-stained tissue: areas with lymphocytes that do not express the PD-L1 biomarker ("negative lymphocytes") are characterized by small blue spots; areas with lymphocytes that do express the PD-L1 biomarker ("positive lymphocytes") are characterized by small blue spots and brown spots; tumor areas with cells that predominantly express the PD-L1 biomarker ("positive tumor cells") are characterized by large blue spots and brown rings; and tumor areas in which cells do not express the PD-L1 biomarker ("negative tumor cells") are characterized only by large blue spots.

[0104] In some embodiments, the image analysis module 207 is run more than once. For example, the image analysis module 207 is run a first time to extract features and classify cells and / or nuclei in a first image; and then run a second time to extract features and classify cells and / or nuclei in a series of additional images, where the additional images can be other single-form images or unmixed image channel images, or any combination thereof.

[0105] After the features are derived, the features can be used alone or in combination with training data (e.g., during training, exemplary cells are presented with ground truth identifications provided by an expert observer according to a process known to those of ordinary skill in the art) to classify the nucleus or cell. In some embodiments, the system may include a classifier that is trained based at least in part on a set of training or reference slices for each biomarker. Those skilled in the art will recognize that different sets of slices can be used to train the classifier for each biomarker. Thus, for a single biomarker, a single classifier is obtained after training. Those skilled in the art will also recognize that, due to differences in image data obtained from different biomarkers, different classifiers can be trained for each different biomarker to ensure better performance on unseen test data, where the biomarker type of the test data will be known. The trained classifier can be selected for slice interpretation based at least in part on how best to handle differences in the training data (e.g., in terms of tissue type, staining procedures, and other features of interest).

[0106] In some embodiments, the classification module is a support vector machine ("SVM"). Generally speaking, SVM is a classification technique that is based on a statistical learning theory that a nonlinear input data set is converted into a high-dimensional linear feature space via a kernel for nonlinear situations. Without wishing to be bound by any particular theory, it is believed that a support vector machine projects a set of training data E representing two different classes into a high-dimensional space by means of a kernel function K. In this transformed data space, nonlinear data is transformed so that a flat line (distinguishable hyperplane) can be generated to separate classes, thereby maximizing class separation. Then test data is projected into the high-dimensional space via K, and the test data is classified on the basis of the descending position of the test data relative to the hyperplane. The kernel function K defines the method for projecting data into the high-dimensional space.

[0107] In other embodiments, classification is performed using the AdaBoost algorithm. AdaBoost is an adaptive algorithm that combines multiple weak classifiers to generate a strong classifier. Image pixels identified by pathologists during the training phase (e.g., image pixels with specific staining or belonging to a specific tissue type) are used to generate a probability density function for each individual texture feature Φj, which is considered to be a weak classifier for j∈{1,...,K}. Bayes' theorem is then used to generate possible scenarios Lj=(Cj,|j∈{1,....,K}) for each Φj that constitutes a weak learner. These are combined by the AdaBoost algorithm into a strong classifier Πj=∑Ti=1αjilji, where, for each pixel cj∈Cj, Πj(cj) is a combined possibility indicating that the pixel cj belongs to class ωT, where αji is the weight determined for the feature Φi during training, and T is the number of iterations.

[0108] In some embodiments, a classifier is used to distinguish tumor nuclei from other candidate nuclei that can also be further classified (e.g., by identifying lymphocyte nuclei and stromal nuclei). In some embodiments, as further described herein, a learned supervised classifier is applied to identify tumor nuclei. For example, a learned supervised classifier is trained in terms of nuclear features to identify tumor nuclei, and then the classifier is applied to classify candidate nuclei in unlabeled images as tumor nuclei or non-tumor nuclei. Alternatively, the learned supervised classifier can be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the learned supervised classifier for identifying tumor nuclei is a random forest classifier. For example, a random forest classifier can be trained in the following manner: (i) creating a training set of tumor nuclei and non-tumor nuclei, (ii) extracting features of each nucleus, and (iii) training a random forest classifier based on the extracted features to distinguish tumor nuclei from non-tumor nuclei. The trained random forest classifier can then be applied to classify nuclei in unlabeled images into tumor nuclei and non-tumor nuclei. Optionally, a random forest classifier can be further trained to distinguish between different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei.

[0109] Calibration of automated cell sorting

[0110] After using automatic image analysis to detect and classify cells as described above, pathologists or other medical professionals can adjust or otherwise correct automatic classification (step 314). In this way, cell classification is semi-automatic or semi-supervised. For example, the cell classification provided by the automatic image analysis algorithm can be displayed as an overlay on the sample image in the viewer, and then the medical professional can check the suspicious cell classification to confirm that the cell has been correctly classified. On the other hand, if the cell is misclassified, the medical professional may have the opportunity to adjust the cell classification, i.e., manually cover the automatic cell classification. For example, the medical professional can manually select the misclassified cells and re-label them with the correct classification. Those skilled in the art will recognize that by checking the automatic classification results, better basic fact training data can be provided to the multi-layer neural network. The output of step 314 is a combination of the correctly identified classification from the automatic image analysis and a group of adjusted cell classifications, thereby providing a "final" cell classification.

[0111] Combination of region-level and cell-level classification data

[0112] After the medical professional has reviewed the cell classification results, the tissue region identification and cell classification are combined at the pixel level (step 314).

[0113] This step of the process takes into account prior knowledge of the biological system. Those skilled in the art will recognize that certain types of cells may not be found in certain types of tissues. For example, it will be impossible to find stromal cells in an area rich in lymphocytes. By discarding those biologically unfeasible combinations, the ground truth data becomes enriched with background knowledge, which helps improve the training of the multi-layer neural network.

[0114] For example, Figure 6 Different regions and different cells that can be classified are shown. Although 20 different combinations are possible, only a smaller subset of combinations are biologically feasible. In this example, the cell type combinations in biologically feasible tissue regions include: (i) tumor cells in tumor regions; (ii) stromal cells in tumor regions; (iii) lymphocytes in tumor regions; (iv) tumor cells in stromal regions; (v) stromal cells in stromal regions; (vi) lymphocytes in stromal regions; (vii) tumor cells in lymphocyte-rich regions; and (viii) lymphocytes in lymphocyte-rich regions. Of course, the degree of biologically feasible combinations may vary based on sample type or sample source, and those skilled in the art will be able to provide instructions to discard certain combinations.

[0115] In some embodiments, the system can identify a region, but no cell type can be associated with the pixels within the identified region. In this case, the pixels can be marked with a region label and a label "background cell" (e.g., tumor region, background cell), wherein the label "background cell" is used as a "placeholder" indicating that the region does not contain any relevant cell type. In this way, the label "background cell" indicates that there is no cell type. In some embodiments, the detected cell can also be considered a "background cell" if it is not a cell of interest or an artifact that can be ignored.

[0116] Each pixel in the sample image is then assigned a label (step 315), the label including the cell type and region type to which the pixel belongs. For example, the pixel label may be (tumor cell, stroma region). In this way, each pixel is labeled with a biologically feasible label.

[0117] Multi-layer neural network

[0118] A multi-layer neural network is then trained using the sample images and the labels assigned to each pixel within the sample images. Figure 3A Step 302; Figure 3B Step 316 of ). For this purpose, any multi-layer neural network can be implemented. Suitable multi-layer neural networks include LeNet proposed by Yann LeCun; AlexNet proposed by Alex Krizhevsky et al.; ZF Net proposed by Matthew Zeiler et al.; GoogLeNet proposed by Szegedy et al.; VGGNet proposed by Karen Simonyan et al.; and ResNet proposed by Kaiming He et al. In some embodiments, the multi-layer neural network is VGG16 (Simonyan, 2014). In other embodiments, the multi-layer neural network is DenseNet (see Huang et al., "Densely Connected Convolutional Networks", arXiv:1608.06993). In some embodiments, a fully convolutional neural network is utilized, such as described by Long et al., “Fully Convolutional Networks for Semantic Segmentation,” Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference, June 20015 (INSPEC Accession No.: 15524435).

[0119] Neural networks are typically trained on large, publicly available image data archives of natural images, such as ImageNet, proposed by Russakovsky et al. in 2015 in the ImageNet Large Scale Visual Recognition Challenge, IJCV, 2015. Given the paucity of training images typically available for biological samples, such as tumor samples, transfer learning methods are employed. In some embodiments, a "transfer learning" approach may be used, in which a pre-trained network, such as those mentioned above, together with its learned weights, is used to initialize the training process on a new set of training images, and the pre-trained network and the weights are trained to adapt the weights to the specific application at hand (see Pan, SJ and Yang, Q., 2010. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345-1359. doi:10.1109 / tkde.2009.191). In some embodiments, the pre-trained network is trained on an existing public or private dataset that may or may not be directly relevant to the application of interest, and may or may not be biological in nature. In some embodiments, the weights of the final layer are randomly initialized to avoid falling into suboptimal local minima, while in other embodiments, the weights of the final layer are imported unchanged.

[0120] After the multi-layer neural network is trained (step 302), one or more unlabeled images of a biological sample for which ground truth is not available (as noted herein) may be provided to the trained classifier to classify regions and cells within the unlabeled images (steps 303 and 304).

[0121] Classify regions and cells using a trained multi-layer neural network

[0122] The present disclosure also provides systems and methods for classifying regions and cells using a trained multi-layer network (see Figure 3A ). In some embodiments, the unlabeled image data is provided to a trained multi-layer neural network (step 303), wherein the trained multi-layer neural network classifies the detected cells in the unlabeled image (step 304). Once the cells are classified within the unlabeled image, the classification results can be used for further analysis (e.g., creating coverage masks, scoring, etc.). Each of these steps will be described in further detail herein.

[0123] refer to Figure 3CIn some embodiments, the system first receives an unlabeled image to be classified (step 320). The unlabeled image to be classified can be acquired using the image acquisition module 202 described herein, or the unlabeled image can be retrieved from the memory 201 (or from the storage module 240). The unlabeled image can be pre-processed using any module identified herein (e.g., a tissue region mask module or a field of view determination module).

[0124] Next, the cell nucleus is detected in the unlabeled image using the cell detection module 204 (step 321). The cell nucleus can be detected according to any method described herein for detecting cells for ground truth data generation (e.g., using a cell detection method based on radial symmetry). In other embodiments, the cell nucleus can be determined using the method described in U.S. Patent Application Publication No. 2017 / 0098310. In particular, U.S. Publication No. 2017 / 0098310 describes a method for automatically identifying cell nucleus boundaries in digital images of biological tissue samples, the method comprising: a processor identifying one or more initial edges in a digital image and using pixels within the identified initial edges as initial voting pixels; a processor performing a tensor voting operation on pixels near each of the initial voting pixels using a tensor voting field; a processor accumulating multiple votes for each neighboring pixel to determine a tensor value for the neighboring pixel, the voting amount indicating whether the neighboring pixel may be an edge pixel of the same nucleus as the initial voting pixel, the tensor value being the sum of all votes cast by all initial voting pixels near the neighboring pixel; and a processor decomposing each determined tensor value to determine eigenvalues ​​and eigenvectors of the decomposed tensor value; a processor evaluating the determined eigenvalues ​​and eigenvectors to identify refined edge pixels, the refined edge pixels representing cell nucleus boundaries; and a processor generating a significant edge intensity image, the significant edge intensity image selectively including the identified refined edge pixels representing cell nucleus boundaries.

[0125] In other embodiments, image segmentation is performed according to the method described by Veta et al., "Automatic Nuclei Segmentation in H&E Stained Breast Cancer Histopathology Images," PLoS One. 2013; 8(7): e70221. According to this method, image segmentation can be divided into four main steps: 1) preprocessing; 2) marker-controlled watershed segmentation; 3) post-processing; and 4) merging of results from multiple scales. The purpose of preprocessing is to remove irrelevant content while preserving the boundaries of the nuclei. Preprocessing begins with color unmixing to separate the hematoxylin stain from the RGB image. The grayscale version of the hematoxylin image is then processed using a series of morphological operations to remove irrelevant structures. The core part of the process is marker-controlled watershed segmentation. The following two types of nuclear markers are used: markers extracted using image transformations that highlight structures with high radially symmetric region minima of the preprocessed image. In the post-processing step, regions that are unlikely to represent nuclei are removed, and the contours of the remaining regions are parameterized as ellipses. By varying the size of the structural element in a preprocessing step, the segmentation process can be tuned to find kernels at different scales, allowing for multi-scale analysis. Segmentation results from multiple scales and both marker types are then merged by resolving concurrent regions to give the final segmentation.

[0126] In other embodiments, a foreground segmentation mask is calculated (step 322), and the generated foreground segmentation mask is used to filter the unlabeled image to identify individual cells (step 323). In some embodiments, the foreground segmentation mask is generated using the method described in U.S. Patent Application Publication No. 2017 / 0337596. In particular, US2017 / 0337596 describes calculating foreground segmentation in the following manner: (1) applying a filter to enhance the image so that (a) image areas that are unlikely to have nuclei are discarded, and (b) nuclei within local areas are identified; and (2) further applying optional filters to selectively remove artifacts, remove small spots, remove discontinuities, fill holes and split larger spots. In some embodiments, the applied filter is selected from a global thresholding filter, a local adaptive thresholding filter, a morphological operation filter, and a watershed transform filter. In some embodiments, a global thresholding filter is applied first, and then a local adaptive thresholding filter is applied. In some embodiments, after applying the local adaptive thresholding filter, optional filters are applied to selectively remove artifacts, remove small spots, remove discontinuities, fill holes, and split larger spots. In some embodiments, identifying individual nuclei further includes performing a connected component labeling process on the filtered input image.

[0127] Next, the unlabeled image is provided to the trained multi-layer neural network 220 (step 324). Those skilled in the art will recognize that the multi-layer neural network 220 must have been trained to classify the cell types present (or suspected of being present) in the unlabeled image. For example, if the network is trained to identify and classify lymphocytes and tumor cells in a breast cancer sample, the unlabeled image must also be an image of a breast cancer sample. In some embodiments, the multi-layer neural network can be trained with sample images (single or multiple) derived from multiple types of cancer and classify sample images from that particular type of cancer. After the unlabeled image is provided to the trained multi-layer neural network, the network will provide a predictive classification at the pixel level, that is, a specific predictable cell type marker is assigned to each pixel within the unlabeled image, such as a tumor cell, a stromal cell, etc.

[0128] Then, using pixel-level prediction classification, a label is assigned to each identified cell (step 325). In some embodiments, assigning a cell label to each identified individual cell includes: (i) quantifying the number of pixels with each predicted classification label within the identified individual cell; and (ii) assigning the predicted label with the largest number as the cell label. In some embodiments, the generated foreground mask can be overlaid on top of the unlabeled image, and each different predicted pixel label within any area (i.e., the identified cell) can be quantified, and then the predicted label with the largest number is selected as the cell label. For example, if a particular identified cell includes a total of 100 pixels, and 55 of all of these pixels are predicted to be tumor cell pixels, 40 of all of these pixels are predicted to be stromal cells, and 5 of all of these pixels are predicted to be lymphocytes, then the cell will be assigned a tumor cell label because most of the pixels are predicted to be tumor cell pixels. Alternatively, if there is no foreground segmentation, an established core center (i.e., a small radial circle around the center, for example, a pixel grouping with a circle having a radius of approximately 5 pixels) can be utilized during label assignment. In this case, the majority pixel label is used as the label for the cell.

[0129] After the cell markers are assigned, the image can be further evaluated (step 326). For example, in some embodiments, a scoring module can be used to generate one or more expression scores. Methods for scoring are further described in detail in commonly assigned and co-pending applications WO / 2014 / 102130A1, "Image analysis for breast cancer prognosis," filed on December 19, 2013, and WO / 2014 / 140085A1, "Tissue object-based machine learning system for automated scoring of digital whole slides," filed on March 12, 2104. For example, based on the positive and negative tumor cells / nuclei detected, one or more methods can be used to calculate various slice-level scores, such as marker positive percentages.

[0130] In some embodiments, a score can be determined for a particular region or field of view of interest. The region of interest or FOV can be manually selected by a pathologist or medical professional, or can be automatically selected based on morphology within an unlabeled image (e.g., see U.S. Patent Application Publication No. 2017 / 0154420).

[0131] In other embodiments, the classification results can be overlaid on the unlabeled image. For example, each cell in the unlabeled image can be marked with a color or other mark (see Figure 4 Panel (C) and Figure 5 (C)). Figure 4 and 5 Examples of ground truth images are shown, where regions were manually annotated. Again, nuclear centers were detected using conventional methods (e.g., radial symmetry as described herein), and foreground segmentation was performed around each detected center as an associated cell mask. A conventional classifier (described herein) was used to assign labels to cells. These assigned labels were then manually corrected (where misclassified) by a pathologist using the image annotation platform.

[0132] example

[0133] Non-limiting examples are provided to illustrate the various steps outlined herein.

[0134] background

[0135] Personalized patient care requires biomarkers that predict response to HER2-targeted therapy and anthracycline-based chemotherapy. Histology-based assessment of tumor-infiltrating lymphocytes (TILs) has been shown to be prognostic and potentially treatment-predictive in breast cancer. However, in the context of these medical needs, the interactions between TILs, tumor cells, other microenvironmental mediators, their spatial relationships, numbers, and other image-based features have not been exhaustively and systematically determined. To quantify and explore these aspects in patient populations, we developed a deep learning-based image analysis algorithm for region- and cell-level segmentation and classification of whole-section images of breast cancer H&E tissue.

[0136] Methods: Overview

[0137] Deep learning (DL) is a machine learning approach in which a multi-layer (deep) convolutional neural network is trained to automatically discover salient features to distinguish different structures of interest. To automatically interpret H&E images, we intended to identify tumor cells, stromal cells, and lymphocytes as well as tumor and stromal regions and reject other regions (e.g., necrosis, artifacts). A single deep network was trained to perform combined region and cell classification at the pixel level. Region-level ground truth was manually annotated. However, manually annotating each cell is tedious and prone to errors. To mitigate this problem, a semi-automatic approach was used in which a pathologist manually corrected the misclassifications output by the image analysis algorithm.

[0138] Methods: Deep Learning Training and Validation

[0139] To train the network, 20 whole-slice images at 20× magnification were used for region and cell annotation (approximately 20,000 regions, 500,000 cells, 2×108 pixels @ 0.5 um).

[0140] For validation, four sub-regions of approximately 1 mm × 1 mm were used in each of eight whole slices (approximately 10,000 regions, 200,000 cells, 108 pixels).

[0141] result

[0142] In two experiments, the developed algorithm was compared with the pathologists’ ground truth.

[0143] In the first experiment, there were 6 H&E breast cancer cases with 14 annotated smaller regions at 20x magnification. Two pathologists manually annotated approximately 9,000 cells as ground truth. For algorithm comparison, only those ground truth annotations that were consistent from the two pathologists were retained, while others were ignored.

[0144]

[0145] In a second experiment, a semi-automatic method (present disclosure) was used to perform ground truth measurements on 10,000 regions and 200,000 cells. A single pathologist visually inspected and corrected the cell labels output by the feature-based method without discarding any ambiguous cell labels.

[0146]

[0147] For region-level ground truth verification, overlapping heterogeneous regions such as stromal and lymphocyte regions pose challenges for pathologists to perform region delineation; therefore, we evaluated the computational accuracy of combining these regions.

[0148]

[0149] In view of the above results, a novel and robust approach involving deep convolutional neural networks has been demonstrated to be feasible for combined region and cell classification in H&E breast cancer images.

[0150] exist Figure 7 The results are depicted in Figure 8. Figure 7 , in the area manually annotated by the pathologist, the semi-automatic method described herein is used to detect, segment cells and generate cell classification marks. The cell mark shown is cell-level, that is, for all pixels corresponding to the single connected spots reflecting a single outline, a specified common cell mark is shown. The regional level mark is pixel-level, that is, all pixels belonging to a regional category are shown in a specific color. The mark is color-coded. Tumor cells, lymphocytes and stromal cells are shown in green, red and yellow respectively. The lower image is the result of classifying the sample image using a trained multi-layer neural network. The regional mark image is shown in the lower right, and the cell level mark image is shown in the lower left. The regional mark image is shown at the pixel level, and the cell mark image shows the cell classification mark at the cell level. As described herein, the multi-layer neural network outputs a classified image, thereby specifying a combined (region, cell) mark for each pixel. In the subsequent step, the combined mark is divided into two different marks-cell mark and regional mark for each pixel. In the next step, a classification label is derived for each cell (at the cell level) aggregated from the pixel-level cell labels, taking the nucleus / cell center locations and the associated foreground blob segmentation mask as additional input for each cell.

[0151] Other components for practicing embodiments of the present disclosure

[0152] Other components (eg, systems or modules) that may be used in conjunction with the systems and methods of the present disclosure are described below.

[0153] Demixing module

[0154] In some embodiments, the image received as input can be a multiple image, that is, the image received is an image of a biological sample stained with more than one stain. In these embodiments, and before further processing, the multiple image is first unmixed into its constituent channels, wherein each unmixed channel corresponds to a specific stain or signal. In some embodiments, the unmixed image (commonly referred to as a "channel image" or "image channel image") can be used as the input of each module described herein. For example, the first H&E image, the second multiple image stained for multiple differentiation markers (CD3, CD8, etc.) clusters, and multiple single-shaped images stained for specific biomarkers (such as ER, PR, Ki67, etc.) can be used to determine the heterogeneity between markers. In this example, the multiple image is first unmixed into its constituent channel images, and these channel images can be used together with the H&E image and the multiple single-shaped images to determine the heterogeneity between markers.

[0155] In some embodiments, in a sample comprising one or more stains and hematoxylin, a separate image can be generated for each channel of the one or more stains and hematoxylin. Without wishing to be bound by any particular theory, it is believed that these channels highlight different tissue structures in the tissue image, and therefore, these tissue structures can be referred to as structural image channels. In some embodiments, unmixing provides at least a hematoxylin image channel image. In some embodiments, the acquired image is unmixed into separate channels representing the local hematoxylin amount and highlighted cell nucleus area in the image. Those skilled in the art will recognize that features extracted from these channels can be used to describe different biological structures present in any image of the tissue.

[0156] The multispectral image provided by the imaging system 202 is a weighted mixture of the underlying spectral signals associated with the individual biomarkers and the noise component. At any particular pixel, the mixing weight is proportional to the biomarker expression of the underlying coexisting biomarkers at a particular location in the tissue and the background noise at that location. Therefore, the mixing weight varies from pixel to pixel. The spectral unmixing method disclosed herein decomposes the multichannel pixel value vector at each pixel into a set of constituent biomarker end members or components and estimates the proportion of each constituent dye for each biomarker.

[0157] Unmixing is the process of decomposing the measured spectrum of a mixed pixel into a set of constituent spectra or end members and a set of corresponding fractions or abundances, which represent the proportion of each end member present in the pixel. Specifically, the unmixing process can extract stain-specific channels to determine the local concentration of a single stain using a reference spectrum known to a standard type of tissue and stain combination. Unmixing can use a reference spectrum retrieved from a control image or estimated from an observed image. Unmixing the component signals of each input pixel enables retrieval and analysis of stain-specific channels, such as the hematoxylin channel and the eosin channel in an H&E image, or the diaminobenzidine (DAB) channel and counterstaining (e.g., hematoxylin) channel in an IHC image. The terms "unmixing" and "color deconvolution" (or "deconvolution"), etc. (e.g., "deconvolution", "unmixing") can be used interchangeably in the art.

[0158] In some embodiments, linear unmixing is used to unmix multiple images with the unmixing module 205. For example, linear unmixing is described in 'Zimmermann, "Spectral Imaging and Linear Unmixing in Light Microscopy" Adv Biochem Engin / Biotechnol (2005) 95: pp. 245-265' and in CLLawson and RJHanson, "Solving least squares Problems", Prentice Hall, 1974, Chapter 23, p. 161". In linear dye unmixing, the spectrum measured at any pixel (S (λ)) is considered to be a linear mixture of the dye spectral components and is equal to the sum of the proportions or weights (A) of the color reference (R (λ)) of each individual dye expressed at the pixel

[0159] S(λ)=A 1 ·R 1 (λ)+A 2 ·R 2 (λ)+A 3 ·R 3 (λ).......A i ·R i (λ)

[0160] This can be expressed more generally in matrix form as follows

[0161] S(λ)=ΣA i ·R i (λ) or S = R·A

[0162] If M channel images are acquired and there are N individual dyes, then the columns of the M×N matrix R are the optimal color system as derived herein, the N×1 vector A are the unknown ratios of the individual dyes and the M×1 vector S is the multichannel spectral vector measured at the pixel. In these equations, the signal (S) in each pixel is measured during the acquisition of multiple images and the reference spectrum, i.e., the optimal color system is derived as described herein. The contribution (A) of each dye is i ) can be determined by calculating their contribution to each point in the measured spectrum. In some embodiments, the solution is obtained using a least squares inverse fitting method that minimizes the squared difference between the measured spectrum and the calculated spectrum by solving the following system of equations:

[0163]

[0164] In this equation, j represents the number of detection channels and i equals the number of stains. The linear equation solution usually involves allowing constrained unmixing to force the weights (A) to sum to unity.

[0165] In other embodiments, unmixing is accomplished using the method described in WO 2014 / 195193, entitled "Image Adaptive Physiologically Plausible Color Separation," filed on May 28, 2014. In general, WO 2014 / 195193 describes a method for unmixing by separating component signals of an input image using an iteratively optimized reference vector. In some embodiments, image data from an assay is correlated with an expected or ideal result specific to the assay characteristics to determine a quality metric. In the case of low quality images or poor correlation with the ideal result, one or more reference column vectors in the matrix R are adjusted, and unmixing is iteratively repeated using the adjusted reference vectors until the correlation shows a good quality image that matches the physiological and anatomical requirements. Anatomical, physiological, and assay information can be used to define rules that are applied to measure image data to determine quality metrics. This information includes how the tissue is stained, which structures within the tissue are intended to be stained or not stained, and the relationship between structures, stains, and markers specific to the assay being processed. The iterative process produces stain-specific vectors that can generate images that accurately identify structures of interest and biologically relevant information, without any noise or unwanted spectra, and are therefore suitable for analysis. The reference vectors are adjusted into a search space. The search space defines the range of values ​​that the reference vectors can use to represent the stain. The search space can be determined by scanning a variety of representative training assays, including known or common problems, and determining a set of high-quality reference vectors for the training assays.

[0166] In other embodiments, unmixing is accomplished using the method described in WO 2015 / 124772, entitled "Group Sparsity Model for Image Unmixing," filed on February 23, 215. In general, WO 2015 / 124772 describes unmixing using a group sparsity framework, wherein the fractions of staining contributions from multiple co-existing markers are modeled within the "same group," and the fractions of staining contributions from multiple non-co-existing markers are modeled in different groups, the modeled group sparsity framework is provided with co-localization information of multiple co-existing markers, the modeled framework is solved using a group lasso to produce a least squares solution within each group, wherein the least squares solution corresponds to unmixing of co-existing markers, and a sparse solution is produced in the group corresponding to unmixing of non-co-existing markers. In addition, WO2015124772 describes a method for unmixing by: inputting image data obtained from a biological tissue sample; reading reference data from an electronic memory, the reference data describing the stain color of each of a plurality of stains; reading coexistence data from an electronic memory, the coexistence data describing a plurality of groups of stains, each group including stains that can coexist in the biological tissue sample, and each group forming a group for a group lasso criterion, at least one of the groups having a size of two or more; and using the reference data as a reference matrix to calculate a solution to the group lasso criterion for obtaining an unmixed image. In some embodiments, the method for unmixing an image may include generating a group sparse model, wherein a portion of the staining contribution from colocalized markers is specified in a single group, a portion of the staining contribution from non-colocalized markers is specified in a separate group, and solving the group sparse model using an unmixing algorithm to produce a least squares solution within each group.

[0167] Other system components

[0168] The system 200 of the present disclosure can be bound to a sample processing device that can perform one or more preparation processes on a tissue sample. The preparation process can include, but is not limited to, dewaxing a sample, conditioning a sample (e.g., cell conditioning), staining a sample, performing antigen retrieval, performing immunohistochemical staining (including labeling) or other reactions, and / or performing in situ hybridization (e.g., SISH, FISH, etc.) staining (including labeling) or other reactions, and other processes for preparing a sample for microscopy, microanalysis, mass spectrometry, or other analytical methods.

[0169] The processing device can apply a fixative to the sample. The fixative can include cross-linking agents (such as aldehydes (e.g., formaldehyde, paraformaldehyde, and glutaraldehyde) and non-aldehyde cross-linking agents), oxidants (e.g., metal ions and complexes such as osmium tetroxide and chromic acid), protein denaturants (e.g., acetic acid, methanol, and ethanol), fixatives of unknown mechanism (e.g., mercuric chloride, acetone, and picric acid), combination reagents (e.g., Carnoy's fixative, methacarn, Bouin's fluid, B5 fixative, Rossman's fluid, Gendre's fluid), microwave and mixed fixatives (e.g., expelled volume fixation and vapor fixation).

[0170] If the sample is a sample embedded in paraffin, the sample can be dewaxed using one or more appropriate dewaxing fluids. After removing the paraffin, any number of substances can be applied to the sample in succession. The substance can be used for pretreatment (e.g., for reversing protein crosslinks, exposing nucleic acids, etc.), denaturation, hybridization, washing (e.g., stringent washing), detection (e.g., linking visual or marker molecules to probes), amplification (e.g., amplifying proteins, genes, etc.), counterstaining, coverslipping, etc.

[0171] The sample processing device can apply a wide range of substances to the sample. The substance includes, but is not limited to, a stain, a probe, a reagent, a rinse agent, and / or a regulator. The substance can be a fluid (e.g., a gas, a liquid, or a gas / liquid mixture), etc. The fluid can be a solvent (e.g., a polar solvent, a non-polar solvent, etc.), a solution (e.g., an aqueous solution or other types of solutions), etc. The reagent can include, but is not limited to, a stain, a wetting agent, an antibody (e.g., a monoclonal antibody, a polyclonal antibody, etc.), an antigen recovery fluid (e.g., an antigen repair solution based on aqueous or non-aqueous, an antigen recovery buffer, etc.), etc. The probe can be a separated nucleic acid or a separated synthetic oligonucleotide attached to a detectable marker or reporter molecule. The label can include a radioactive isotope, an enzyme substrate, a cofactor, a ligand, a chemiluminescent or fluorescent agent, a hapten, and an enzyme.

[0172] After processing the sample, the user can transport the slide carrying the sample to the imaging device. In some embodiments, the imaging device is a bright field imager slide scanner. A bright field imager is the iScan Coreo bright field scanner sold by Ventana Medical Systems. In an automated embodiment, the imaging device is a digital pathology device, such as disclosed in International Patent Application No. PCT / US2010 / 002772 (Patent Publication No. WO / 2011 / 049608) entitled IMAGING SYSTEM AND TECHNIQUES or disclosed in U.S. Patent Application No. 61 / 533,114 filed on September 9, 2011 entitled IMAGING SYSTEMS, CASSETTES, AND METHODS OF USING THE SAME.

[0173] The imaging system or device may be a multispectral imaging (MSI) system or a fluorescence microscopy system. The imaging system used herein is MSI. MSI is typically equipped for analysis of pathological samples with a computerized microscope-based imaging system by providing access to the spectral distribution of pixel-level images. Although there are various multispectral imaging systems, an operational aspect common to all of these systems is the ability to form multispectral images. A multispectral image is an image that captures image data of a specific wavelength or a specific spectral bandwidth across the electromagnetic spectrum. These wavelengths may be selected by optical filters or by using other instruments capable of selecting predetermined spectral components, which include electromagnetic radiation at wavelengths outside the visible range, such as, for example, infrared (IR).

[0174] The MSI system may include an optical imaging system, a portion of which includes a spectral selectivity system that is tunable to define a predetermined number of N discrete optical bands. The optical system may be suitable for imaging a tissue sample, irradiating an optical detector in transmission with a broadband light source. An optical imaging system that may include a magnification system (such as, for example, a microscope) in one embodiment has a single optical axis that is usually spatially aligned with a single optical output of the optical system. When the spectral selection system is adjusted or tuned (for example, with a computer processor), the system forms a series of images of the tissue, such as to ensure that images are acquired in different discrete spectral bands. The device may additionally include a display in which at least one visually perceptible tissue image from a sequence of acquired images appears. The spectral selection system may include a collection of optical dispersive elements (such as diffraction gratings), optical filters (such as thin film interference filters), or any other system suitable for selecting a specific passband from a light source through a sample toward a detector transmitted spectrum in response to a user input or a command of a preprogrammed processor.

[0175] In an alternative embodiment, the spectrally selective system defines a number of optical outputs corresponding to N discrete spectral bands. This type of system ingests the transmitted light output from the optical system and spatially redirects at least a portion of this light output along N spatially distinct optical paths in a manner that images a sample in an identified spectral band onto a detector system along the optical path corresponding to the identified spectral band.

[0176] The subject matter described in this specification and the embodiment of operation can be implemented in a digital electronic circuit system or in a computer software, firmware or hardware (including the structure disclosed in this specification and its structural equivalent) or one or more combinations thereof. The embodiment of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions for executing or controlling the operation of a data processing device encoded on a computer storage medium. Any module described herein may include logic executed by one or more processors. As used herein, "logic" refers to any information with instruction signals and / or data forms that can be applied to affect the operation of a processor. Software is an example of logic.

[0177] A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. In addition, although a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be or may be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification may be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0178] The term "programmed processor" includes all kinds of devices, apparatuses, and machines for processing data, including, for example, a programmable microprocessor, a computer, a system on a chip or multiple systems on a chip, or a combination of the foregoing. The device may include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more thereof. The device and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0179] A computer program (also referred to as a program, software, software application, script or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and the computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines or partial codes). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0180] The processes and logic flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by a device, and the device can also be implemented as a special-purpose logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0181] For example, processors suitable for executing computer programs include general and special microprocessors and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from a large-capacity storage device or to transfer data or both to a large-capacity storage device. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in a special purpose logic circuit system.

[0182] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., an LCD (liquid crystal display), an LED (light emitting diode) display, or an OLED (organic light emitting diode) display) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. In some embodiments, a touch screen may be used to display information and receive input from the user. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user (e.g., by sending a web page to a web browser in response to a request received from a web browser on a user's client device).

[0183] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or a middleware component (e.g., an application server), or a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnections (e.g., the Internet), and peer-to-peer networks (e.g., self-organizing peer-to-peer networks). For example, Figure 1 The network 20 may include one or more local area networks.

[0184] The computing system may include any number of clients and servers. The client and server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of a computer program that runs on each computer and has a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., the purpose is to display data to a user interacting with the client device and to receive user input from a user interacting with the client device). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0185] According to the embodiments disclosed herein, the multi-layer neural network is applicable to LeNet, AlexNet, ZF Net, DetectNet, GoogLeNet, VGGNet ResNet, VGG16 and DenseNet.

[0186] According to embodiments disclosed herein, the method further comprises generating an overlay comprising descriptive indicia of each labeled individual cell.

[0187] Although the present disclosure has been described with reference to a number of illustrative embodiments, it will be appreciated that those skilled in the art may devise numerous other modifications and embodiments that fall within the spirit and scope of the principles of the present disclosure. More specifically, reasonable changes and modifications may be made to the components and / or arrangements of the subject combination arrangement within the scope of the foregoing disclosure, the accompanying drawings, and the appended claims without departing from the spirit of the present disclosure. In addition to changes and modifications to the components and / or arrangements, alternative uses will also be apparent to those skilled in the art.

Claims

1. A computer-implemented method for training a multi-layer neural network to detect and classify different cell types and regions within a sample image from a set of training images, the method comprising: receiving the set of training images, each training image in the set of training images comprising a plurality of pixels and depicting a portion of a biological sample; For each corresponding training image in the set of training images: identifying regions associated with different types of tissue within the corresponding training image, the regions corresponding to subsets of pixels of the plurality of pixels; Detecting and classifying cells in the corresponding training image based on features in the corresponding training image, wherein the results of detecting and classifying the cells provide a suggested cell classification for the cells; using input from a user to correct an incorrectly classified suggested cell classification to provide a final cell classification for the cell; as well as determining pixels in the subset of pixels associated with a biologically feasible combination by comparing the different types of tissue to the final cell classification, the biologically feasible combination being a combination of a tissue type and a cell type; as well as assigning labels to the pixels, wherein the labels indicate that the cell types and tissue types associated with those pixels correspond to biologically feasible combinations; as well as The multi-layer neural network is trained using the set of training images and the labels assigned to the pixels, wherein training the multi-layer neural network enables the multi-layer neural network to predict a cell type for a pixel in an image and a biologically viable tissue region type for the pixel.

2. The computer-implemented method of claim 1 , wherein: Identification of different regions within the sample image is performed manually by a pathologist and / or automatically by one or more processors (209).

3. The computer-implemented method of claim 1 or 2, wherein: Detecting and classifying cells based on features within the sample image includes: (i) detecting cell nuclei; (ii) calculating a foreground segmentation mask; (iii) deriving features from the detected cell nuclei; and (iv) using a classifier to classify the nuclei based on the derived features.

4. The computer-implemented method according to any one of claims 1 to 3, wherein: The label assigned to each pixel is a vector that includes the classified cell type and an identification of the tissue region in which the pixel is located in the sample image.

5. The computer-implemented method according to any one of claims 1 to 4, wherein: The tissue regions include tumor regions, stroma regions, lymphocyte-rich regions, and necrotic regions, and wherein the cell types include tumor cells, stroma cells, and lymphocytes.

6. The computer-implemented method of claim 5, wherein: Biologically feasible combinations include: (i) tumor cells in a tumor area; (ii) stromal cells in a tumor area; (iii) lymphocytes in a tumor area; (iv) tumor cells in a stromal area; (v) stromal cells in a stromal area; (vi) lymphocytes in a stromal area; (vii) tumor cells in a lymphocyte-rich area; and (viii) lymphocytes in a lymphocyte-rich area.

7. The computer-implemented method according to any one of claims 1 to 6, further comprising: The trained multi-layer neural network is provided with an unlabeled image and receives a predicted label for each pixel within the unlabeled image.

8. A computer-implemented method for classifying cells within an unlabeled image of a biological sample, the method comprising: detecting cell centers in the unlabeled image; Calculating a foreground segmentation mask of the unlabeled image; identifying individual cells in the unlabeled image by filtering the unlabeled image using the calculated foreground segmentation mask; Applying a multi-layer neural network trained according to the method of any one of claims 1 to 7 to the unlabeled image, wherein application of the trained multi-layer neural network provides a predicted label for each pixel in the unlabeled image, each predicted label indicating (i) a cell type corresponding to the corresponding pixel, and (ii) a region type corresponding to the corresponding pixel; and Assign a cell label to each identified single cell.

9. The computer-implemented method of claim 8, further comprising quantifying each of the differently labeled cells and calculating expression scores.

10. The computer-implemented method of claim 9, further comprising quantifying the number and other spatial distribution characteristics of lymphocytes in the tumor region or the stromal region.

11. The computer-implemented method of claim 10, wherein: Assigning a cell label to each identified single cell includes: (i) quantifying the number of pixels with each predicted label within the identified single cell; and (ii) assigning the predicted label with the largest number as the cell label.

12. A system (200), comprising: one or more processors (209); as well as A non-transitory computer readable memory (201) storing instructions which, when executed by the one or more processors (209), cause the one or more processors (209) to perform the method according to any one of claims 1 to 7.

13. A system (200), comprising: a cell detection module (204), the cell detection module being adapted to identify cells within the unlabeled image; a multi-layer neural network (220) trained using the method of any one of claims 1 to 7, the multi-layer neural network being used to provide a predicted label for each pixel in the unlabeled image, the predicted label corresponding to a different cell type; as well as A cell labeling module (208) is adapted to classify the cells identified by the cell detection module (204) using the method according to any one of claims 8 to 11.

14. The system according to claim 13, further comprising a scoring module configured to count cell markers in the unlabeled image and output an expression score based on the ratio of various counted cell markers, wherein in particular, the scoring is performed within a predetermined field of view within the unlabeled image.

15. A non-transitory computer-readable storage medium encoded with instructions executable by processing resources of a computing system to cause the computing system to perform the method according to any one of claims 1 to 11.

16. A computer-implemented method for training a multi-layer neural network to detect and classify different cell types and regions within a sample image from a set of training images, the method comprising: receiving the set of training images, each training image in the set of training images comprising a plurality of pixels and depicting a portion of a biological sample; For each corresponding training image in the set of training images: identifying different tissue regions in the corresponding training image, wherein the tissue regions include a tumor region, a stroma region, a lymphocyte-rich region, and a necrotic region; Detecting and classifying cells based on features within the corresponding training images to provide a suggested cell classification, wherein the cell classification includes tumor cells, stromal cells, and lymphocytes; using input from a user to correct incorrectly classified suggested cell classifications to provide a final cell classification; and Determining pixels of the plurality of pixels of the corresponding training image that are associated with a biologically feasible combination, the biologically feasible combination being a combination of tissue regions of the different tissue regions and cell types associated with the final cell classification, wherein the biologically feasible combination comprises: (i) tumor cells in a tumor region; (ii) stromal cells in a tumor region; (iii) lymphocytes in a tumor region; (iv) tumor cells in a stromal region; (v) stromal cells in a stromal region; (vi) lymphocytes in a stromal region; (vii) tumor cells in a lymphocyte-rich region; and (viii) lymphocytes in a lymphocyte-rich region; and assigning labels to pixels of the plurality of pixels of the corresponding training image based on the identified different tissue regions and the final cell classification, wherein labels are assigned only to those cell and region combinations that are biologically feasible; and The multi-layer neural network is trained using the set of training images and the labels assigned to the pixels.

17. A system (200), comprising: one or more processors (209); as well as A non-transitory computer readable memory (201) storing instructions which, when executed by the one or more processors (209), cause the one or more processors (209) to perform the method of claim 16.

18. A computer-implemented method for classifying cells within an unlabeled image of a biological sample, the method comprising: detecting cell centers in the unlabeled image; Calculating a foreground segmentation mask of the unlabeled image; identifying individual cells in the unlabeled image by filtering the unlabeled image using the calculated foreground segmentation mask; applying a multi-layer neural network trained according to the method of claim 16 to the unlabeled image, wherein application of the trained multi-layer neural network provides a predicted label for each pixel within the unlabeled image, each predicted label indicating (i) a cell type corresponding to the corresponding pixel, and (ii) a region type corresponding to the corresponding pixel; and Assign a cell label to each identified single cell.

19. A system (200), comprising: a cell detection module (204), the cell detection module being adapted to identify cells within the unlabeled image; a multi-layer neural network (220) trained using the method of claim 16, the multi-layer neural network being used to provide a predicted label for each pixel in the unlabeled image, the predicted label corresponding to a different cell type; as well as A cell labeling module (208) is adapted to classify the cells identified by the cell detection module (204) using the method according to claim 18.

20. A non-transitory computer-readable storage medium encoded with instructions executable by a processing resource of a computing system to cause the computing system to perform the method of claim 16 or 18.

Images