Scalable, highly accurate, context-guided segmentation of histological structures including ducts / glands and lumens, duct / gland clusters, and individual nuclei in whole-slide images of tissue samples from spatial multi-parameter cellular and subcellular imaging platforms

Abstract

A method (and system) for segmenting one or more histological structures in a tissue image represented by multi-parameter cellular and subcellular imaging data includes receiving coarsest level image data of the tissue image, the coarsest level image data corresponding to a coarsest level of a multi-scale representation of first data corresponding to the multi-parameter cellular and subcellular imaging data. The method further includes dividing the coarsest level image data into a plurality of non-overlapping superpixels, assigning to each superpixel a probability of belonging to one or more histological structures using a number of pre-trained machine learning algorithms to create a probability map, applying a contour algorithm to the probability map to extract estimated boundaries of the one or more histological structures, and using the estimated boundaries to provide precise boundaries of the one or more histological structures.

Description

[Technical field]

[0001] <Government Contract> This invention was made with Government support under Grant #CA204826 awarded by the National Institutes of Health (NIH). The Government has certain rights in the invention.

[0002] <Technical field> The present invention relates to digital pathology, and in particular to a system and method for scalable, highly accurate, context-guided segmentation of histological structures, including but not limited to ducts / glands and lumens, clusters of ducts / glands, and individual nuclei, in multi-parameter cellular and sub-cellular imaging data of several stained tissue images, such as whole slide images, obtained from several patients or several multi-cellular in vitro models. [Background technology]

[0003] Histopathological examination of diseased tissue is essential for the diagnosis and grading of disease. Currently, pathologists make diagnostic decisions (e.g., disease grade and severity) based on visual interpretation of histopathological structures, typically in transmitted light images of diseased tissue. Such decisions are often subjective and subject to high levels of discrepancy, especially in atypical situations.

[0004] Additionally, digital pathology is gaining momentum with applications such as second opinion telepathology, immunostaining interpretation, and intraoperative telepathology. Typically, digital pathology consists of several tissue slides, which generate a large amount of patient data representing them, and then displaying the slides on a high-resolution monitor to be evaluated by a pathologist. Due to the manual steps involved, current workflow practices in digital pathology are time-consuming, error-prone, and subjective. Summary of the Invention

[0005] In one embodiment, a method for segmenting one or more histological structures in a tissue image represented by multi-parameter cellular and subcellular imaging data is provided, the method including receiving a coarsest level of image data of the tissue image, the coarsest level of image data corresponding to a coarsest level of a multi-scale representation of first data corresponding to the multi-parameter cellular and subcellular imaging data, the method further including dividing the coarsest level of image data into a number of non-overlapping superpixels, assigning to each superpixel a probability of belonging to one or more histological structures using a number of pre-trained machine learning algorithms to create a probability map, applying a contour algorithm to the probability map to extract estimated boundaries of the one or more histological structures, and using the estimated boundaries to provide an accurate boundary of the one or more histological structures. In one exemplary embodiment, the multi-scale representation includes a Gaussian multiscale pyramid decomposition, the multi-parameter cellular and subcellular imaging data includes stained tissue image data, receiving the coarsest level image data for the tissue image includes receiving normalized constituent stain image data of the stained tissue image, the coarsest level normalized constituent stain image data for a particular constituent stain of the stained tissue image and corresponding to a coarsest level of a Gaussian multiscale pyramid decomposition of the first data corresponding to the stained tissue image data, and splitting the coarsest level image data into a plurality of superpixels includes splitting the coarsest level normalized constituent stain image data into a plurality of superpixels.

[0006] In one embodiment, a computer system for segmenting one or more histological structures in a tissue image represented by multi-parameter cellular and subcellular imaging data is provided, the system including a processing device, the processing device comprising a number of components configured to perform the method described above. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram of an exemplary digital pathology system for segmenting histological structures from multi-parameter cellular and subcellular imaging data according to an exemplary embodiment of the disclosed concepts. [Figure 2A] FIG. 2A is a flow chart illustrating a method for segmenting histological structures according to certain exemplary embodiments of the disclosed concepts. [Figure 2B] FIG. 2B is a flow chart illustrating a method for segmenting histological structures according to certain exemplary embodiments of the disclosed concepts. [Diagram 3] FIG. 3 shows a non-limiting exemplary H&E stained tissue image that can be processed by the disclosed concepts, illustrating the color deconvolution step of an exemplary embodiment of the disclosed concepts. [Figure 4] FIG. 4 illustrates the staining intensity normalization step of an exemplary embodiment of the disclosed concepts. [Diagram 5] FIG. 5 illustrates the Gaussian multi-scale pyramid decomposition step of an exemplary embodiment of the disclosed concepts. [Figure 6] FIG. 6 illustrates the decomposition of image data into superpixels according to an exemplary embodiment of the disclosed concepts. [Figure 7] FIG. 7 illustrates an example superpixel pair according to an example embodiment of the disclosed concepts. [Figure 8] FIG. 8 illustrates an example probability map and an example image showing the application of a region-based active contour algorithm in accordance with an example embodiment of the disclosed concepts. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] As used herein, the singular forms "a" and "the" include plural references unless the context clearly dictates otherwise.

[0009] As used herein, a description of two or more parts or components being "coupled" means that the parts are coupled or operate together, either directly or indirectly, i.e., through one or more intermediate parts or components, to the extent that a connection occurs.

[0010] As used herein, the term "several" means one or an integer greater than one (ie, a plurality).

[0011] As used herein, the terms "component" and "system" are intended to refer to a computer-related entity, whether hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. For example, both an application running on a server and the server may be considered a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one computer and / or distributed among two or more computers. Although certain ways of displaying information to a user are shown and described herein with particular diagrams or graphs as screenshots, one of ordinary skill in the relevant art will recognize that various other alternatives may be employed.

[0012] As used herein, the term "multiparametric cellular and subcellular imaging data" refers to data obtained by generating several images from a section of tissue, providing information about several measurable parameters at the cellular and / or subcellular level in the section of tissue. Multiparametric cellular and subcellular imaging data may be generated by any of several different imaging modalities, including but not limited to, transmitted light (e.g., a combination of H&E and / or IHC (one or more biomarkers)), fluorescence, immunofluorescence (including but not limited to antibodies, nanobodies), multiplexing and / or hypermultiplexing of live cell biomarkers, and electron microscopy. Targets include but are not limited to tissue samples (human or animal) and in vitro models of tissues and organs (human or animal).

[0013] As used herein, the term "superpixel" means a connected patch or group of two or more pixels that have similar image statistics defined in an appropriate color space (eg, RGB, CIELAB, or HSV).

[0014] As used herein, the term "non-overlapping superpixel" refers to a superpixel whose boundaries do not overlap with any of its neighboring superpixels.

[0015] As used herein, "Gaussian multi-scale pyramid decomposition" refers to repeatedly smoothing and subsampling an image by 2 in the x and y directions with Gaussian filters.

[0016] As used herein, the term "region-based active contour algorithm" refers to any active contour model that takes image gradients into account to detect object boundaries.

[0017] As used herein, the term "context-ML model" refers to a machine learning algorithm that can take into account neighborhood information of superpixels.

[0018] As used herein, the term "stain-ML model" refers to a machine learning algorithm that can take into account the stain intensity of superpixels.

[0019] As used herein, the term "probability map" is intended to mean a set of pixels having a probability value ranging from 0 to 1, which represents the positional probability that a pixel is within or is not within a particular histological structure.

[0020] Directional terms used herein, such as top, bottom, left, right, upper, lower, front, back, and derivatives thereof, refer to the orientation of the elements illustrated in the drawings and do not limit the scope of the claims, unless expressly stated.

[0021] The disclosed concepts are described below, for purposes of explanation, with numerous specific details in order to provide a thorough understanding of the subject innovation. However, it will be apparent that the disclosed concepts can be practiced without these specific details without departing from the spirit and scope of the invention.

[0022] The disclosed concepts, described in further detail herein in connection with various exemplary embodiments, provide a novel approach to identify and characterize morphological properties of histopathological structures. An initial application of such tools is to perform scalable, highly accurate, context-guided segmentation of histologic structures, including, for example, but not limited to, ducts / glands and lumens, duct / gland clusters, and individual nuclei, in images of tissue samples (e.g., whole slide images) based on spatial multi-parameter cellular and subcellular imaging data representing such images. In this particular non-limiting application of the disclosed concepts, hematoxylin and eosin (H&E) image data is employed as the multi-parameter cellular and subcellular imaging data, and color-deconvolved hematoxylin image data is used to segment ducts / glands and lumens, duct / gland clusters, and individual nuclei, as described in more detail herein. However, this is merely exemplary, and it will be understood that the disclosed concepts may be employed to segment other histologic structures using other types of data. For example, connective tissue may be segmented using color deconvolved eosin image data derived from H&E image data. Still other possibilities are contemplated within the scope of the disclosed concepts.

[0023] The disclosed concepts are related to and improve upon the subject matter described in U.S. Patent Application No. 15 / 577,838 (published as 2018 / 0204085) entitled "Systems and Methods for Finding Regions of Interest in Hematoxylin and Eosin (H&E) Stained Tissue Images and Quantifying Intratumor Cellular Spatial HeterogeneityiIn Multiplexed / Hyperplexed Fluorescence Tissue Images," the disclosure of which is incorporated herein by reference. The disclosed concepts differ from the subject matter of the above application in at least two ways. First, the disclosed concepts fall into the semi-supervised or weakly supervised category in that there is user input in at least one step of the machine learning algorithm. Also, the disclosed concepts work best when the boundaries of the region of interest (ROI) are given in rough estimates. The disclosed concepts detailed herein sharpen such rough boundaries.

[0024] FIG 1 is a schematic diagram of an exemplary digital pathology system 5 constructed and configured for automated segmentation of histological structures from multi-parameter cellular and subcellular imaging data in accordance with an exemplary embodiment of the concepts disclosed herein. As seen in FIG 1, system 5 is a computing device constructed and configured to generate and / or receive multi-parameter cellular and subcellular imaging data (referenced 25 in FIG 1) and process the data as described herein to segment histological structures within tissue images represented by multi-parameter cellular and subcellular imaging data 25. System 5 may be, for example, but not limited to, a PC, a laptop computer, a tablet computer, or any other suitable computing device constructed and configured to perform the functions described herein.

[0025] The system 5 includes an input device 10 (such as a keyboard), a display 15 (such as an LCD), and a processing device 20. A user can use the input device 10 to provide input to the processing device 20, which provides output signals to the display 15 to enable the display 15 to display information to the user as described in detail herein. The processing device 20 includes a processor and a memory. The processor may be, for example, but not limited to, a microprocessor (μP), a microcontroller, an application specific integrated circuit (ASIC), or other suitable processing device, and is interfaced with the memory. The memory may be one or more of various types of internal and / or external storage media, such as RAM, ROM, EPROM, EEPROM, FLASH®, or other providing storage registers, for data storage, such as an internal storage area of ​​the computer, and may be a volatile or non-volatile memory. The memory stores several routines executable by the processor, including routines for implementing the disclosed concepts, as described herein. In particular, the processing device 20 includes a histological structure segmentation component 30 configured to identify and segment histological structures (such as, but not limited to, ducts / glands and lumens, clusters of ducts / glands, and individual nuclei) in several tissue images (e.g., H&E stained image data) represented by multi-parameter cellular and subcellular imaging data 25 obtained from various imaging modalities, in various embodiments as described herein.

[0026] 2A and 2B are flow charts illustrating a method for segmenting histological structures according to certain exemplary embodiments of the disclosed concepts. The method illustrated in FIG. 2A and 2B may be implemented, for example and without limitation, in system 5 of FIG. 1 described above, and the method is so described for illustrative purposes. Additionally, in the particular non-limiting exemplary embodiment illustrated in FIG. 2A and 2B, the multi-parameter cellular and subcellular imaging data used is H&E stained image data of a tissue sample, and the histological structures to be segmented are based on hematoxylin image data and include ducts / gland and lumens, clusters of ducts / gland, and individual nuclei. Again, it will be understood that the particular embodiment illustrated in FIG. 2A and 2B and described herein is intended to be merely exemplary and not limiting. It will be understood that other types of multi-parameter cellular and subcellular imaging data may be used in conjunction with the disclosed concepts.

[0027] 2A, the method begins at step 100, where the processing device 20 of the system 5 generates and / or receives multi-parameter cellular and subcellular imaging data representative of an H&E stained tissue image to be processed. A non-limiting exemplary H&E stained tissue image 35 that may be processed by the disclosed concepts is shown for illustrative purposes in FIG. 3. Additionally, in an exemplary embodiment, the multi-parameter cellular and subcellular imaging data generated and / or received in step 100 is in RGB format.

[0028] Next, in step 105, the multi-parameter cellular and subcellular imaging data generated and / or received in step 100 (i.e., H&E stained tissue image data in RGB format) is color deconvoluted into individual staining intensities (hematoxylin and eosin) to generate hematoxylin image data and eosin image data for the H&E stained tissue image being processed. Figure 3 illustrates the color deconvolution of step 105 by showing a hematoxylin image 40 represented by the hematoxylin image data and a resulting eosin image 45 represented by the eosin image data resulting from color deconvolution of the H&E stained tissue image 35.

[0029] The method then proceeds to step 110, where the staining intensity of the hematoxylin image data is normalized with the reference data set to generate normalized hematoxylin image data. The staining intensity normalization of step 110 is performed so that the variation in staining intensity is normalized for downstream processing. In an exemplary embodiment, to set the reference data set, a batch of whole slide images (WSIs) is first color deconvolved into hematoxylin staining intensity images and eosin staining intensity images. From this batch, a random number of 1Kx1K images are cropped and used to create a cumulative intensity histogram of the hematoxylin channel. The test WSIs are first subjected to a color deconvolution operation. Then, a histogram equalization is performed to match the intensity histogram of the hematoxylin channel with that of the reference data set. The staining intensity normalization of step 110 is illustrated in FIG. 4. 4 shows the original hematoxylin channel 50 of another (different) exemplary whole slide image and the normalized hematoxylin channel 55 of the same exemplary whole slide image, such that the intensity of the normalized hematoxylin channel 55 now matches that of the reference image dataset.

[0030] Next, in step 115, a Gaussian multi-scale pyramid decomposition (a form of pyramid representation) is performed on the normalized hematoxylin image data to generate a multi-scale representation of the normalized hematoxylin image data. The multi-scale representation created in step 115 has n levels, L1...L n where L1 is the level data representing the full resolution level of the decomposition, and L n is the level data representing the coarsest level of decomposition. Constructing a Gaussian pyramid can reduce the computational burden of detecting histological structures from whole slide images according to the methods of the disclosed concepts. FIG. 5 illustrates a Gaussian multi-scale pyramid decomposition of the exemplary normalized hematoxylin channel 55 shown in FIG. 4. In an exemplary embodiment, the image size is reduced from the original resolution of 30K×50K to the coarsest level of 1K×1.5K. Also, in an exemplary embodiment, the image size is halved at each level of the decomposition hierarchy.

[0031] The method then proceeds to step 120 of FIG. 2B. In step 120, the coarsest level of data L n is divided into non-overlapping superpixels, which in an exemplary embodiment are collections of connected pixels with similar intensity (gray) values. As will be appreciated, this can be done in a number of ways. In the simplest approach, a normal distribution of noise is assumed with mean zero and standard deviation sigma. For example, for an image with 256 gray levels per pixel, the standard deviation of the noise is typically assumed to be 4 gray levels. This value may be set by the end user. In an exemplary embodiment, this is done using a simple linear iterative clustering (SLIC) algorithm, several of which are known in the art. FIG. 6 shows the coarsest level of data L obtained from an exemplary hematoxylin image 40 described herein. n1 shows the results of step 120 when performed on a 1K×1K superpixel image. In an exemplary embodiment, the image is segmented into approximately 5K superpixels. This is recommended for a 1K×1K hematoxylin channel image because it is computationally fast and efficient for downstream processing. Additionally, in an exemplary embodiment, a 2D Delaunay triangulation of the superpixel centroids is performed to identify spatial neighbors for each superpixel.

[0032] Additionally, according to one aspect of the disclosed concept, several machine learning algorithms / models are trained to predict which superpixels belong to a particular histological structure, which in the exemplary embodiment is a duct / gland. Thus, following step 120, the method proceeds to step 125, where each superpixel is assigned a probability of belonging to a histological structure, which in the illustrated exemplary embodiment is a duct / gland, using several pre-trained machine learning algorithms. As a result, the coarsest level of data L n A probability map is created.

[0033] In a non-limiting exemplary embodiment of the disclosed concept, some trained machine learning algorithms employed in step 125 consist of a context-ML model (such as a context-Support Vector Machine (SVM) model or a context-Logistic Regression (LR) model) and a staining-ML model (such as a staining-Support Vector Machine (SVM) model or a staining-Logistic Regression (LR) model) for predicting superpixels that belong to the structure in question. In this exemplary embodiment, the RGB color histograms of the superpixels and their neighbors are used as feature vectors. Specifically, in this embodiment, two models are constructed and trained (in a supervised manner): a context-ML model and a staining-ML model. Each of these models is described in more detail below.

[0034] For the context-ML model, which in an exemplary embodiment is a context-SVM model, 2000 adjacent superpixel pairs in 10 different images from the reference image dataset are randomly selected for the training set of the exemplary embodiment. The ground truth is collected (i.e., user input is sought) by displaying the superpixel pairs on a screen and asking a subject (e.g., an experienced / expert pathologist) whether none of the displayed superpixels belong to a tube, or whether one or both belong to a tube. For illustrative purposes, FIG. 7 shows three such exemplary superpixel pairs (labeled A, B, C) of an exemplary image, where the pair labeled A has 0 superpixels present in the tube (class label 0), the pair labeled B has 1 superpixel present in the tube (class label 1), and the pair labeled C has 2 superpixels present in the tube (class label 2). This ground truth is used as class labels to train the context ML model. In an exemplary embodiment, the color histograms (i.e., R, G, B color pixel values) of each superpixel pair and their first neighbors are used as feature vectors. For each test superpixel pair, the ML model returns the probability that the pair has superpixels that do not both belong to a tube, or one or both belong to a tube. In each case, the superpixel pair is assigned to the category with the highest probability. Note that this does not determine the actual identity of the superpixels that are in the structure. Instead, a second model, staining-ML, described below, is applied for that purpose.

[0035] Ground truth is collected separately for the stain-ML model, which in the exemplary embodiment is a stain-SVM model. In particular, a subject (e.g., an experienced / expert pathologist) is asked to classify whether a particular superpixel has “no staining”, “light staining”, “medium staining”, “dark staining”, or “unknown” (i.e., user input is again required). Since some structures, such as tubes, are amorphous, one way to detect the boundaries of the structure is to carefully observe the change in staining color while moving from the inside of the tube to the surrounding connective tissue. This information can be used to identify superpixels that may be part of a tube. FIG. 7 shows four such superpixels D, E, F, G for an exemplary image, classified as “no staining”, “light staining”, “medium staining”, and “dark staining”, respectively. The superpixel is assigned to the category with the highest probability.

[0036] Therefore, in step 125 according to this particular exemplary embodiment, the two machine learning models (context-ML model and staining-ML model) are applied to the coarsest level of data L n The context-ML model is then applied sequentially to the non-overlapping superpixels of the corresponding superpixels (step 120) to create a probability map for identifying pairs of superpixels that are likely to be inside the structure of interest (a tube in this exemplary embodiment). In the exemplary embodiment, all medium to dark stained superpixels are identified as being inside the tube. In other words, the context-ML model and stain-ML model together assign a conditional probability to each superpixel that it belongs to the structure of interest.

[0037] Once the probability map has been created as described above in step 125, the method proceeds to step 130. In step 130, rough estimates of the boundaries of the histological structures are extracted by applying a region-based active contour algorithm to the probability map. An example probability map 60 and an example image 65 illustrating the application of the region-based active contour algorithm are provided in FIG.

[0038] The method then proceeds to step 135, where the just-obtained estimate is used to effect segmentation of the structure in the full resolution image. Specifically, step 135 involves successively refining the boundaries of the histological structures from coarse to fine by upsampling the structure boundaries from level K+1 to level K, and then starting a region-based active contour at level K with the upsampled boundaries. In an exemplary embodiment, the upsampling first involves simply doubling the coordinates found at K+1 to determine the coordinates of the boundaries at level K+1 to level K, and then performing an interpolation between the boundary pixels at level K.

[0039] 2A and 2B, superpixels are utilized only at the coarsest level of the pyramid. As explained, region-based contouring is performed on the probability map of the identified superpixels. However, at successively finer scales, superpixels are not required and region-based active contouring is performed directly on the associated stain-separated images.

[0040] In one particular implementation of this exemplary embodiment, the region-based active contour algorithm used is the Chan-Vese segmentation algorithm, which separates the foreground (the tube) from the background (the rest of the image). The cost function of the active contour varies with the difference in the mean value of hematoxylin staining in the foreground and background regions. For example, two superpixels that are likely to be in a tube have approximately the same staining (medium to dark staining), and their boundaries are iteratively merged by the active contour optimization.

[0041] To construct "clusters" of tubes, region-based active contours may be run on the probability maps returned by the context-ML and stain-ML models. In an exemplary embodiment, the probability maps attribute non-zero probabilities to regions that bridge tubes, and the region-based active contour model run on the probability maps is better at depicting tube clusters. To segment tubes from the full WSI, tubes and tube clusters are first identified from the lowest resolution pyramid image using the steps described above. These results are recursively upsampled, and region-based active contours are re-run at each level of the hierarchy to refine the upsampled tube boundaries. In an exemplary embodiment, the active contour image consists of a mask that indicates pixels inside and outside the tube boundary. The active contour image and the hematoxylin image are upsampled together.

[0042] The disclosed concepts may be used to identify nuclei within a tube. Once a tube is identified, superpixel segmentation is performed on the regions belonging to the tube. A staining-ML model may then be run to further separate the medium stained and darkly stained superpixels within the tube. The darkly stained superpixels would correspond to the location of nuclei within the tube. To identify nuclei outside the tube, a similar model may be deployed that builds feature vectors (histograms of red, blue, and green channels) of superpixels without first-layer neighbors. Without the average histogram of superpixels and their first layer, all the darkly stained superpixels corresponding to nuclei within and outside the tube would be identified.

[0043] In the exemplary embodiment described in connection with FIG. 2A and FIG. 2B, the color deconvolution step (105) and the stain intensity normalization step (110) are performed before the Gaussian multi-scale pyramid decomposition is performed (step 115). In another embodiment, the order of these steps is reversed. In particular, in an alternative embodiment of the disclosed concept, after the multi-parameter cellular and subcellular imaging data is received (step 100), the Gaussian multi-scale pyramid decomposition is first performed on the entire H&E stained tissue image. Next, the color deconvolution of step 105 and the stain intensity normalization of step 110 are performed only on the image data of the coarsest level, generating normalized hematoxylin image data for only the image of the coarsest level. Steps 120 to 130 are then performed using the normalized hematoxylin image data for only the image of the coarsest level, generating the probability map and the estimated boundary of the histological structure as described above. The boundary is then sequentially refined using step 135 as described.

[0044] Again, it should be noted that while certain embodiments of the disclosed concepts use color deconvolved hematoxylin image data to segment ducts / glands and lumens, clusters of ducts / glands, and individual nuclei, this is meant to be exemplary only, and it is understood that the disclosed concepts can be employed to segment other histological structures using other types of data. For example, but not by way of limitation, connective tissue may be segmented using color deconvolved eosin image data (different from color deconvolved hematoxylin image data) derived from H&E image data. Still other possibilities are contemplated within the scope of the disclosed concepts.

[0045] Furthermore, the above description of the disclosed concepts is based on and utilizes in situ multi-parameter cellular and subcellular imaging data. However, it will be understood that this is not meant to be limiting. Rather, it will be understood that the disclosed concepts can be used in conjunction with in vitro microphysiological models for basic research and clinical translation. Multicellular in vitro models allow for the study of spatiotemporal cellular heterogeneity and heterogeneous cell-cell communication that recapitulates human tissues applicable to the investigation of mechanisms of disease progression in vitro, drug testing, and characterization of the structural organization and content of these models that can be used for transplantation.

[0046] In the claims, signs placed between parentheses shall not be interpreted as limiting the scope of the claim. The word "comprises" or "including" does not exclude the presence of elements or steps other than those listed in the claim. In a device claim enumerating several means, several of these means may be embodied by one and the same hardware. The word "a" preceding an element does not exclude the presence of a plurality of such elements. In any device claim enumerating several means, several of these means may be embodied by one and the same hardware. The mere fact that certain elements are recited in mutually different dependent claims does not indicate that these elements cannot be used in combination.

[0047] Although the present invention has been described in detail for the purpose of illustrating based on what are currently considered to be the most practical and preferred embodiments, it is to be understood that such detail is for the purpose only, and that the present invention is not limited to the disclosed embodiments, but on the contrary, is intended to include modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it is to be understood that the present invention contemplates that one or more features of any embodiment can be combined with one or more features of any other embodiment, to the extent possible.

Claims

1. 1. A method for segmenting one or more histological structures in a tissue image represented by multi-parameter cellular and subcellular imaging data, comprising: receiving a coarsest level of image data for the tissue image, the coarsest level of image data corresponding to a coarsest level of a multi-scale representation of first data corresponding to the multi-parameter cellular and subcellular imaging data; decomposing said coarsest level image data into a plurality of non-overlapping superpixels; assigning a probability of belonging to the one or more histological structures to each non-overlapping superpixel using a number of pre-trained machine learning algorithms to create a probability map; extracting estimated boundaries of the one or more histological structures by applying a contour algorithm to the probability map; using the estimated boundary to provide an accurate boundary for the one or more histological structures; A method comprising:

2. 2. The method of claim 1, wherein the multi-scale representation includes a Gaussian multi-scale pyramid decomposition, the multi-parameter cellular and subcellular imaging data includes stained tissue image data, receiving the coarsest level of image data for the tissue image includes receiving the coarsest level of normalized constituent stain image data for the stained tissue image, the coarsest level of normalized constituent stain image data being for a particular constituent stain of the stained tissue image and corresponding to a coarsest level of a Gaussian multi-scale pyramid decomposition of the first data corresponding to the stained tissue image data, and decomposing the coarsest level of image data into a plurality of non-overlapping superpixels includes splitting the coarsest level of normalized constituent stain image data into the plurality of non-overlapping superpixels.

3. 3. The method of claim 2, wherein the multi-parameter cellular and subcellular imaging data includes stained tissue image data, and receiving the coarsest level image data for the tissue image includes receiving the coarsest level normalized constituent stain image data for the stained tissue image, the coarsest level normalized constituent stain image data being generated by: (i) color deconvolving the stained tissue image data into individual staining intensities to create constituent stain image data, (ii) normalizing the constituent stain image data using a reference dataset to create normalized data, and (iii) performing a Gaussian multi-scale decomposition on the normalized data to generate a multi-scale representation of the normalized data that includes the coarsest level normalized constituent stain image data, and the first data corresponding to the stained tissue image data is the normalized data.

4. 3. The method of claim 2, wherein the first data corresponding to the stained tissue image data is the stained tissue image data, and the coarsest level normalized constituent stain image data is generated by: (i) performing a Gaussian multi-scale decomposition on the stained tissue image data to create a multi-scale representation of the stained tissue image data; (ii) color deconvolving the coarsest level of the Gaussian multi-scale pyramidal decomposition of the stained tissue image data into individual stain intensities to create the coarsest level constituent stain image data; and (iii) normalizing the coarsest level constituent stain image data using a reference dataset to create the coarsest level normalized constituent stain image data.

5. The method of claim 2 , wherein the number of pre-trained machine learning algorithms are a number of supervised machine learning algorithms pre-trained based on user input.

6. 6. The method of claim 5, wherein the several pre-trained machine learning algorithms include a context-SVM model or a context-LR model applied to the plurality of non-overlapping superpixels, and a staining-SVM model or a staining-LR model applied to the plurality of non-overlapping superpixels.

7. The method of claim 2 , wherein the contour algorithm is a region-based active contour algorithm.

8. 3. The method of claim 2, wherein using the boundary estimate to yield an accurate boundary of the one or more histological structures comprises upsampling a boundary of the one or more histological structures using a level of the Gaussian multi-scale pyramid decomposition other than a coarsest level, and yielding the accurate boundary using a region-based active contour algorithm at a finest level of the Gaussian multi-scale pyramid decomposition.

9. The method of claim 2 , wherein the one or more histological structures include ducts or glands, the stained tissue image data includes H&E data, and the specific constituent stain is hematoxylin.

10. The method of claim 2 , wherein the one or more histological structures include connective tissue, the stained tissue image data includes H&E data, and the specific constituent stain is eosin.

11. The method of claim 1 , wherein dividing the coarsest level image data into a plurality of non-overlapping superpixels employs a Linear Iterative Clustering (SLIC) algorithm.

12. A non-transitory computer readable medium storing one or more programs comprising instructions that, when executed by a computer, cause the computer to perform the method of claim 1.

13. 1. A computer system for segmenting one or more histological structures in a tissue image represented by multi-parameter cellular and subcellular imaging data, comprising: receiving a coarsest level of image data for the tissue image, the coarsest level of image data corresponding to a coarsest level of a multi-scale representation of first data corresponding to the multi-parameter cellular and subcellular imaging data; decomposing said coarsest level image data into a plurality of non-overlapping superpixels; assigning a probability to each non-overlapping superpixel of belonging to the one or more histological structures using a number of pre-trained machine learning algorithms to create a probability map; extracting estimated boundaries of the one or more histological structures by applying a contour algorithm to the probability map; using the estimated boundary to provide an accurate boundary for the one or more histological structures; A system comprising a processing device including a number of components configured to perform the steps of:

14. 14. The system of claim 13, wherein the multi-scale representation includes a Gaussian multi-scale pyramid decomposition, the multi-parameter cellular and subcellular imaging data includes stained tissue image data, receiving the coarsest level of image data for the tissue image includes receiving the coarsest level of normalized constituent stain image data for the stained tissue image, the coarsest level of normalized constituent stain image data being for a particular constituent stain of the stained tissue image and corresponding to a coarsest level of a Gaussian multi-scale pyramid decomposition of the first data corresponding to the stained tissue image data, and decomposing the coarsest level of image data into a plurality of non-overlapping superpixels includes splitting the coarsest level of normalized constituent stain image data into the plurality of non-overlapping superpixels.

15. 15. The system of claim 14, wherein the multi-parameter cellular and subcellular imaging data includes stained tissue image data, and receiving the coarsest level image data for the tissue image includes receiving the coarsest level normalized constituent stain image data for the stained tissue image, the coarsest level normalized constituent stain image data being generated by: (i) color deconvolving the stained tissue image data into individual staining intensities to create constituent stain image data, (ii) normalizing the constituent stain image data using a reference dataset to create normalized data, and (iii) performing a Gaussian multi-scale decomposition on the normalized data to generate a multi-scale representation of the normalized data including the coarsest level normalized constituent stain image data, and the first data corresponding to the stained tissue image data is the normalized data.

16. 15. The system of claim 14, wherein the first data corresponding to the stained tissue image data is the stained tissue image data, and the coarsest level normalized constituent stain image data is generated by: (i) performing a Gaussian multi-scale decomposition on the stained tissue image data to create a multi-scale representation of the stained tissue image data; (ii) color deconvolving the coarsest level of the Gaussian multi-scale pyramidal decomposition of the stained tissue image data into individual stain intensities to create the coarsest level constituent stain image data; and (iii) normalizing the coarsest level constituent stain image data using a reference dataset to create the coarsest level normalized constituent stain image data.

17. 15. The system of claim 14, wherein the number of pre-trained machine learning algorithms are a number of supervised machine learning algorithms pre-trained based on user input.

18. 20. The system of claim 17, wherein the number of pre-trained machine learning algorithms includes a context-SVM model or a context-LR model applied to the plurality of non-overlapping superpixels, and a staining-SVM model or a staining-LR model applied to the plurality of non-overlapping superpixels.

19. The system of claim 14 , wherein the contour algorithm is a region-based active contour algorithm.

20. 15. The system of claim 14, wherein using the boundary estimate to yield an accurate boundary of the one or more histological structures comprises upsampling a boundary of the one or more histological structures using a level of the Gaussian multi-scale pyramid decomposition other than a coarsest level, and yielding the accurate boundary using a region-based active contour algorithm at a finest level of the Gaussian multi-scale pyramid decomposition.

21. 15. The system of claim 14, wherein the one or more histological structures include ducts or glands, the stained tissue image data includes H&E data, and the specific constituent stain is hematoxylin.

22. 15. The system of claim 14, wherein the one or more histological structures include connective tissue, the stained tissue image data includes H&E data, and the specific constituent stain is eosin.

23. 14. The system of claim 13, wherein dividing the coarsest level image data into a plurality of non-overlapping superpixels employs a Linear Iterative Clustering (SLIC) algorithm.

Images