System and method characterizing cell phenotype diversity from multiple parameter cell / intracellular imaging data

Abstract

SOLUTION: The method for characterizing cell phenotype includes receiving multiple parameter cell / intracellular imaging data for several tissue samples from several patients or several multicellular in vitro models, performing cell segmentation on multiple parameter cell / intracellular imaging data to create segmented multiple parameter cell / intracellular imaging data, and performing recursive decomposition on segmented multiple parameter cell / intracellular imaging data to identify multiple computational phenotypes. The recursive decomposition includes multiple levels of decomposition, including soft / probabilistic clustering and spatial regularization at each level of decomposition, and each cell in the segmented multiple parameter cell / intracellular imaging data is assigned probabilistically to one or more of the multiple computational phenotypes.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] <Government Contract> This invention was made with Government support under Grant #CA204826 awarded by the National Institutes of Health (NIH). The Government has certain rights in this invention.

[0002] <Field of the Invention> The present invention relates to digital pathology, and in particular to an unsupervised hierarchical learning system and method for characterizing and classifying different cell types and their activation (cell phenotypes) from multi-parameter cellular and subcellular imaging data (e.g., multiplexed or hyperplexed imaging data) obtained from various imaging modalities. [Background technology]

[0003] Digital pathology refers to the acquisition, storage, and display of histologically stained tissue samples, and has initially attracted attention for niche applications such as second opinion telepathology, immunostain interpretation, and intraoperative telepathology. Typically, in digital pathology, a large amount of patient data consisting of multiple slides is generated from biopsy samples and evaluated by a pathologist viewing the slides on a high-resolution monitor. Due to the manual steps involved, current workflow practices are time-consuming, error-prone, and subjective.

[0004] Tissues are heterogeneous cellular systems in which different cell types spatially interconnect. One of the challenges in digital pathology is how to accurately and efficiently characterize different cell types and their activation from multiparameter cellular and subcellular imaging data obtained from various multiplexed or highly multiplexed imaging modalities. Summary of the Invention

[0005] In one embodiment, a method for characterizing cell phenotypes from multi-parameter cellular and subcellular imaging data of several tissue samples from several patients or several multi-cellular in vitro models is provided, comprising receiving segmented multi-parameter cellular and subcellular imaging data, the segmented multi-parameter cellular and subcellular imaging data being generated by performing cell segmentation on the multi-parameter cellular and subcellular imaging data, and performing a recursive decomposition on the segmented multi-parameter cellular and subcellular imaging data to identify a plurality of computational phenotypes, the recursive decomposition including multiple levels of decomposition, each level of decomposition including soft / probabilistic clustering and spatial regularization, and each cell of the segmented multi-parameter cellular and subcellular imaging data is probabilistically assigned to one or more of the multiple computational phenotypes. Belongs to More specifically, for example, such a probabilistic allocation / Attribution means that a cell may have an X% probability of belonging to phenotype A, a Y% probability of belonging to phenotype B, etc. In certain implementations, cells may be deterministically assigned to only one of multiple computational phenotypes for ease of visualization and interpretation.

[0006] In another embodiment, a computer system is provided for characterizing cell phenotypes from multi-parameter cellular and sub-cellular imaging data of several tissue samples from several patients or several multi-cellular in vitro models. The system includes a processing device including a cell phenotype characterization component configured to receive segmented multi-parameter cellular and sub-cellular imaging data, the segmented multi-parameter cellular and sub-cellular imaging data being generated by performing cell segmentation on the multi-parameter cellular and sub-cellular imaging data, and to perform a recursive decomposition on the segmented multi-parameter cellular and sub-cellular imaging data to identify a plurality of computational phenotypes. The recursive decomposition includes multiple levels of decomposition, each level of decomposition including soft / stochastic clustering and spatial regularization. Each cell of the segmented multi-parameter cellular and sub-cellular imaging data is probabilistically assigned to one or more of the multiple computational phenotypes. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a flow chart of a method for characterizing cellular phenotypic diversity in multiplexed to highly multiplexed imaging data according to an exemplary embodiment of the disclosed concepts. [Diagram 2] FIG. 2 is a schematic diagram of an exemplary cellular phenotypic dendrogram for two exemplary sub-cohorts generated according to the method of FIG. [Diagram 3] FIG. 3 is a schematic diagram of an example heat map that may be generated in accordance with one aspect of the disclosed concepts. [Figure 4] FIG. 4 is an exemplary image that may be generated according to one embodiment of the disclosed concepts, in which cell boundaries are color-coded based on the results of a sub-cohort of the disclosed concepts. [Diagram 5]FIG. 5 is a schematic diagram of an exemplary digital pathology system for characterizing and classifying cell types and their activation (cell phenotypes) from multi-parameter cellular and subcellular imaging data based on an exemplary embodiment of the disclosed concepts. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] As used herein, the singular forms "a" and "the" include plural references unless the context clearly dictates otherwise.

[0009] As used herein, a description of two or more parts or components being "coupled" means that the parts are coupled or operate together, either directly or indirectly, i.e., through one or more intermediate parts or components, to the extent that a connection occurs.

[0010] As used herein, the term "several" means one or an integer greater than one (ie, a plurality).

[0011] As used herein, the terms "component" and "system" are intended to refer to a computer-related entity, whether hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. For example, both an application running on a server and the server may be a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one computer and / or distributed between two or more computers. Although some ways of displaying information to a user have been shown and described with specific diagrams or graphs as screenshots, one of ordinary skill in the relevant art will recognize that various other alternatives may be employed.

[0012] As used herein, the term "multiplexed imaging" refers to an imaging technique that uses up to seven biomarkers, and the term "multiplexed image" refers to an image produced using multiplexed imaging.

[0013] As used herein, the term "hyperplexed imaging" refers to an imaging technique that uses more than seven biomarkers, and the term "highly multiplexed image" refers to an image produced using highly multiplexed imaging.

[0014] As used herein, the term "multiplexed or highly multiplexed imaging" is intended to include multiplexed imaging and / or highly multiplexed imaging, and the term "multiplexed or highly multiplexed image" is intended to include multiplexed image and / or highly multiplexed image.

[0015] Directional terms used herein, such as top, bottom, left, right, upper, lower, front, back, and derivatives thereof, refer to the orientation of the elements illustrated in the drawings and do not limit the scope of the claims, unless expressly stated.

[0016] The disclosed concepts are described below, for purposes of explanation, with numerous specific details in order to provide a thorough understanding of the invention. However, it will be apparent that the disclosed concepts can be practiced without these specific details without departing from the spirit and scope of the invention.

[0017] The disclosed concepts provide novel unsupervised hierarchical learning techniques for characterizing cellular phenotypic diversity in multiplexed or highly multiplexed imaging data obtained from various imaging modalities. In an exemplary embodiment, the disclosed concepts are described herein in the context of characterizing and classifying various cell types and their activation (cellular phenotypes) from multi-parameter cellular and subcellular imaging data in the form of highly multiplexed immunofluorescence biomarker data. However, this is intended to be merely exemplary, and it will be understood that the disclosed concepts may be used in the context of any spatial multi-parameter cellular and subcellular imaging data, including but not limited to imaging data obtained using any of the following imaging modalities: transmitted light, a combination of H&E and IHC (one or more biomarkers), fluorescence, immunofluorescence (including but not limited to antibodies, nanobodies), multiplexed live cell biomarkers, highly multiplexed, mass spectrometry (including but not limited to CyTOF), spatial transcriptomics (including but not limited to FISH), and electron microscopy. Targets include, but are not limited to, tissue samples (human or animal) and in vitro models of tissues and organs (human or animal).

[0018] As described in detail herein, one of the main ideas of the disclosed concepts is the adoption of data-driven phenotyping rather than user-defined phenotyping. More specifically, in the pathobiology literature, it is very common for users to define phenotypes (user-defined phenotypes) as follows: "biomarker A is positive, biomarker B is positive, biomarker C is negative, this means phenotype X". This kind of user-defined phenotype means that there are no automatically discovered data-driven phenotypes. As described in detail herein, the disclosed concepts allow for automatically discovered data-driven phenotypes, and thus are significantly different from what is currently accepted in the field. Rather, the approach of the disclosed concepts is unsupervised and likely to automatically identify user-defined phenotypes. The approach of the disclosed concepts is also hypothesis-generating in that it can identify and characterize novel phenotypes (data-driven phenotypes) that emerge in the data due to disease progression.

[0019] Thus, as described in detail herein, the computational phenotyping approach of the disclosed concepts includes two steps: (i) soft / probabilistic clustering and (ii) spatial regularization, which are applied recursively to the imaging data (i.e., recursive decomposition). In an exemplary embodiment, the results of the recursive decomposition are used to generate a computational phenotype tree, where the terminal nodes (leaves) of the tree represent distinct computational phenotypes discovered in the input dataset. These distinct computational phenotypes constitute the components of the final multi-factor analysis (MFA) model determined by the recursive decomposition.

[0020] Considering phenotypic diversity (e.g., epithelial enterocytes, bone marrow macrophages, lymphoid T cells, mesenchymal fibroblasts), the disclosed concept defines each cluster as a hierarchical mixture of parametric low-dimensional subspaces. Considering a continuum of phenotypic activation (e.g., epithelial to mesenchymal transition), each cell can probabilistically be placed in any cluster in the tree. Belongs to Furthermore, the cluster assignment is spatially regularized to achieve spatial coherence that respects tissue structure. Additionally, in an exemplary embodiment, the recursive decomposition is binary, i.e., the mixture model has two components at each level of the tree. The stopping criterion for the recursion in the exemplary embodiment is a threshold applied to the angle between the subspaces of the mixture model to ensure that the resulting phenotypes are distinct from each other and avoid overfitting. The disclosed concepts also apply, in an exemplary embodiment, a Kullback-Leibler (KL) divergence metric to further quantify the difference between any two computed phenotypes using the MFA model parameters, Attribution Visualize the spatial distribution of phenotypes within a tissue sample by assigning each cell to the phenotype with the highest probability.

[0021] As mentioned above, the unsupervised machine learning algorithm of the disclosed concepts recursively applies probabilistic clustering with parametric mixture models and spatial regularization of the resulting cluster assignments. In an exemplary embodiment, the probabilistic clustering algorithms include, but are not limited to, factor analysis (FA) and probabilistic principal component analysis (PPCA). Furthermore, the probabilistic mixture models include, but are not limited to, mixture of factor analysis, Gaussian mixture models, and mixture of PPCA (MMPCA). In one particular implementation, the disclosed concepts use mixture factor analysis to learn the model parameters via an expectation maximization algorithm.

[0022] Furthermore, in another particular embodiment, the disclosed concept employs a novel cost function for spatial regularization. Specifically, the objective cost function is composed of two terms. The first term promotes sparsity in cluster assignment, which means that each cell attempts to belong entirely to one cluster. The second term promotes spatial coherence, which means that if a cell's cluster assignment is indeterminate, it seeks assistance from other nearby cells in updating its cluster assignment. Furthermore, there is an additional constraint in the objective function that the sum of the cluster assignments for each cell is 1. Also, to promote spatial coherence, each cell is connected to its neighbors that are within a cutoff distance. The disclosed concept may also use an alternating direction multiplier method (ADMM) approach.

[0023] For the probabilistic assignment of cells to clusters, in an exemplary embodiment, all levels of the hierarchy use all cell data to build and evaluate the corresponding mixture model, but the cells are weighted appropriately. These weights also influence the spatial regularization step. To find the most discriminatory biomarkers for the computationally derived phenotype, the disclosed concept may perform a ranking operation on the biomarkers based on the difference between the component mean vectors and subspace orientations in the mixture factor analysis model. Finally, the hierarchical structure is terminated when the angle between the component subspaces of the mixture model falls below a cutoff value.

[0024] FIG. 1 is a flow chart of a method for characterizing cellular phenotypic diversity with multiplexed or highly multiplexed imaging data according to an exemplary embodiment of the disclosed concepts. Referring to FIG. 1, the method begins at step 5, where multi-parameter cellular and subcellular imaging data (in a non-limiting exemplary embodiment, in the form of highly multiplexed immunofluorescence biomarker data) is generated from a plurality of tissue samples from a cohort, which in a non-limiting exemplary embodiment is a cancer patient cohort. However, it will be understood that this is intended to be illustrative only, and that data may be obtained from any patient cohort (i.e., any disease). More specifically, in a non-limiting exemplary embodiment, a plurality of HxIF image stacks are generated from formalin-fixed paraffin-embedded (FFPE) tissue microarrays (TMA) of resected tissue samples obtained from the patient cohort. As can be appreciated, generating an HxIF image stack involves generating multiple high-resolution multiplexed images from each tissue section using a multiplexed (in this embodiment highly multiplexed) imaging process that involves repeatedly labeling each tissue section with several fluorescent tags to image multiple biomarkers for each stack.

[0025] In a non-limiting exemplary embodiment described herein for purposes of illustration, a GE Healthcare Cell DIVE® (formerly known as MultiOmyx) HxIF imaging and image processing workflow instrument may be used to generate the multi-parameter cellular and subcellular imaging data acquired in step 5 of Figure 1. It will be understood that this is exemplary and that multi-parameter cellular and subcellular imaging data may be generated using other known or later developed techniques.

[0026] In an exemplary embodiment, the Cell DIVE system is capable of highly multiplexed imaging of over 50 biomarkers by sequentially multiplexing two to three biomarkers and a DAPI (4',6-diamino-2-phenylindole) nuclear counterstain through repeated cycles of labeling-imaging-dye inactivation. Extensive validation of this approach has shown that the majority of epitopes tested are highly resistant to the dye inactivation process. The biological integrity of the samples has been shown to be maintained for at least 50 repeated cycles.

[0027] More specifically, in an exemplary embodiment, data generation using Cell DIVE involves non-destructive cyclical immunofluorescence labeling with two or three antibodies labeled with distinct fluorescent probes, imaging, and subsequent quenching of the fluorescence. This process is repeated until all the required antibodies (biomarkers) are captured. In an exemplary embodiment, the data consists of image stacks taken at each region of interest, with the entire image stack consisting of multiple images for each of several imaging rounds. Each round includes a nuclear (DAPI) image that serves as a reference for registering the images of all rounds. Quantification of the images of each round includes the fluorescence intensity of each measured biomarker. Images are also acquired after a quenching round for the purpose of removing autofluorescence. Processing of Cell DIVE images also includes correction for uneven illumination across the field of view, removal of autofluorescence, registration, and automated quality control (QC) detection of several categories of defects, such as registration failure, image blurring and saturation, and other image problems. The images and data undergo a series of tissue and cell quality checks, log2 transformations, and normalization steps. To consolidate data from batch processing, each biomarker is normalized to the control median. In an exemplary embodiment, images are acquired in TIFF format, and image metadata is placed into a file with a simple structure that preserves the origin and acquisition characteristics of which image was obtained from which slide. The biomarkers selected are protein markers for specific cell lineages, oncogenes, tumor suppressors, and post-translational protein modifications indicative of cell activation status. The data also includes clinical information such as histological tumor grade, cancer stage, sex, age, and 10-year follow-up monitoring.

[0028] In certain non-limiting exemplary embodiments of the disclosed concepts described herein for illustrative purposes, 56 specific biomarkers are used to generate multi-parameter cellular and subcellular imaging data in step 5. However, it will be understood that this is intended for illustrative purposes only, and that more or less and / or different biomarkers may also be used within the scope of the disclosed concepts.

[0029] Following step 5, the method proceeds to step 10, where cell segmentation is performed on the acquired multi-parameter cellular and subcellular imaging data to segment each cell of each tissue sample. In an exemplary embodiment, cell segmentation is performed using a collection of structural biomarkers (NaKATPase (cell membrane, boundary), S6 (cytoplasm), DAPI (nucleus)). Protein expression and standard deviation are quantified at the median biomarker intensity within each cell mask and transformed to log2 scale. Cells are filtered based on the number of pixels per segmented subcellular compartment using individual QC scores generated for each cell (scores below 0.7-0.8, indicating inaccurate registration, inconsistency, or tissue loss, are not included).

[0030] Next, in step 15, a recursive decomposition is performed on the segmented multi-parameter cellular and subcellular imaging data, including soft / probabilistic clustering and spatial regularization at each level of decomposition. This recursive decomposition creates a cell phenotype tree in which each cell is probabilistically assigned to one or more of a number of computational phenotypes (i.e., each cell probabilistically belongs to one or more computational phenotypes). In one particular implementation, each cluster / computational phenotype is defined as a hierarchical mixture of parametric low-dimensional subspaces, and each cell is probabilistically assigned to all clusters / computational phenotypes. Attribution will be done.

[0031] The methodology of the disclosed concepts may be performed on inputs of cohorts of any size. In certain exemplary embodiments described herein for purposes of illustration, the disclosed concepts were implemented on sub-cohorts (NED, REC) of the original full cohort of cancer patients based on disease recurrence within 5 years. Exemplary cell phenotype trees 20A and 20B of the sub-cohorts generated according to step 15 of FIG. 1 are shown diagrammatically in FIG. 2. In the cell phenotype trees 20A and 20B, each node 22 (only a few exemplary nodes are labeled 22 for ease of illustration) represents a computational phenotype, with size being the majority cluster / phenotype (based on size key 24 shown in FIG. 2) for that cluster / phenotype. Attribution The terminal nodes 22 based on the stopping criteria are outlined in black. Inferred biological labels are written based on a set of discriminatory biomarkers, where possible. Within each phenotype, cells are Special Specialized Attribution Probability > 0.95) or non-specific Special Specialized (non-specialized) Attribution The probability of each phenotype being <0.95. Special Transformed cells and non-specific Special The percentage of transformed cells is shown relative to the entire data set (all values ​​add up to 100%). Special Note the presence of a very small subset (7%) of differentiated cells. In this example, the size of each subcohort varies greatly (NED=154, REC=59), but the phenotypic heterogeneity is greater in the REC subcohort, supporting the well-known hyperproliferative properties of the tumor. In addition, these results indicate the hyperproliferation and differentiation undergone by stem cells, since the REC subcohort identifies three stem-like phenotypes, whereas the NED subcohort identifies only one.

[0032] According to further aspects of the disclosed concepts, the differences between each pair of computational phenotypes are quantified using the Kullback-Leibler (KL) divergence metric. In particular, each computational phenotype can be considered as a distribution described by a mean vector and a covariance matrix. KL divergence is a relative similarity ranking measure. The results are visualized in a heatmap 26 as shown diagrammatically in FIG. 3, with separating lines indicating the boundaries of each subcohort and the epithelial and stromal branches of each tree. The nodes along the horizontal and diagonal directions of this panel are the same and are color-coded to the corresponding trees. For convenience, the nodes along the diagonal include manually inferred biological properties (where possible) based on the ranked discriminant biomarkers. The similarity based on the KL divergence metric between epithelial-epithelial and stromal-stromal phenotypes indicates the stability and consistency of the methods of the disclosed concepts (e.g., biologically, epithelial computational phenotypes should be more similar than epithelial-stromal computational phenotypes). This metric also illustrates the distinction between phenotypes with similar putative biological properties due to the high dimensionality of computational phenotypes (e.g., stem-stem computational phenotypes are not identical based on KL divergence). Furthermore, representative stage I, II, and III patients from each outcome-based cohort (NED, REC) were selected and tissue samples with color-coded cell boundaries based on the results of the sub-cohorts of the disclosed concepts can be visualized in the image 28 shown in Figure 4 (each cell is represented by the highest chromatin density, as described herein). Attribution can be assigned to one phenotype based on probability).

[0033] Moreover, in the particular exemplary embodiment described herein for purposes of illustration, step 15 is performed in the manner described below, although it will be understood that this is for purposes of example only and that alternative ways of performing step 15 are contemplated within the scope of the disclosed concepts.

[0034] First, in this particular exemplary embodiment, the highly multiplexed dataset is described in a high-dimensional space, and each cell

number

number

number

number

number

number

number

number

number

[0035] <Soft Clustering> Typically, cell phenotyping methods are built on the assumption that each cell belongs to only one cluster (hard clustering), leaving no room for identifying specific cells that may belong to multiple phenotypes due to the existing phenotypic continuum. Using Mixed Factor Analysis (MFA), the disclosed concept in this particular exemplary embodiment can be implemented by using the parameters

number

number

[0036] <Spatial regularization> This soft clustering does not rely on the spatial complexity of the TME, which is a key component leading to intratumor heterogeneity (ITH). Based on the properties of spatial ITH and the spatial tissue structure of the tumor, the neighborhood of cells is expected to be spatially coherent (e.g., epithelial / tumor cells are surrounded by or in spatial proximity to other epithelial / tumor cells, but the presence of tumor-infiltrating lymphocytes and other stromal cells is taken into account). To facilitate cell identification, a spatial regularization component is added to estimate the non-specific Special of transformed cells Attribution The probability is optimized. The spatial regularization step is given by Attribution We optimize an objective function consisting of two terms: ownership confidence and spatial coherence:

number

number

[0037] The objective function is optimized using the Alternating Directional Multiplier Method (ADMM). Attribution Since the confidence (first term) and spatial coherence (second term) need to have equal weighting, we assume that we compute an adjustment parameter λ to scale the second term to the range of the first term:

number

[0038] <Recursive decomposition> To automate the process of phenotype discovery with the disclosed concepts in this particular exemplary embodiment, a recursive probabilistic approach is used, where at each step, the most dominant clusters are decomposed in M=2 components. At each recursive step, a soft clustering step uses a low-dimensional latent space MFA. The resulting cell-wise clusters are then regularized by spatial regularization. Attribution Optimize the odds, Attribution Increasing confidence and spatial coherence reduces false positives Special The parameters obtained for each cluster j ( Attribution Probability Ω j , the mean vector

number

[0039] <Discrimination biomarkers> Each recursive clustering or partitioning in this particular exemplary embodiment involves a high-dimensional mean vector

number

number

[0040] <Comparison of computational phenotypes> As described elsewhere herein, the terminal nodes (i.e., leaves) of the tree represent distinct computational phenotypes detected in the input dataset and constitute the components of the final MFA model determined by the recursive decomposition. In the MFA model, each computational phenotype j is Attribution Probability c(Ω cj ) are held. Attribution Using probabilities, the method of this exemplary embodiment performs a weighted average

number

number

number

number

number

[0041] <Common-cell comparison> Furthermore, in this particular embodiment, each cell has the highest Attribution Based on the probability, a cell can be assigned to one phenotype (cell-label). If the input cohorts have common tissue samples (such as ALL-DATA and stage I), the cell-labels from the two derived sets of computational phenotypes can be compared. A confusion matrix may be calculated and the resulting cell phenotype labels compared. Specifically, for cell phenotype i and cell phenotype j, the confusion matrix probability is:

number

[0042] Systematic selection of representative patients for visualization As described above, in this embodiment, the cells have the highest Attribution For C phenotypes, the method calculates the fraction of each phenotype seen in each patient i, and gives the vector

number

number

[0043] FIG. 5 is a schematic diagram of an exemplary digital pathology system 30 for characterizing and classifying cell types and their activation (cell phenotypes) from multi-parameter cellular and subcellular imaging data according to an exemplary embodiment of the disclosed concepts described herein. As seen in FIG. 5, the system 30 is a computing device configured and configured to generate / receive multi-parameter cellular and subcellular imaging data as described herein and process the data to characterize and classify cells as described herein. The system 30 may be, for example, but not limited to, a PC, laptop computer, tablet computer, or other suitable device configured to perform the functions described herein. The system 30 includes an input device 32 (such as a keyboard), a display 34 (such as an LCD), and a processing unit 36. A user can provide input to the processing unit 36 ​​using the input device 32, and the processing unit 36 ​​provides output signals to the display 34 to enable the display 34 to display information to the user as described in detail herein (e.g., trees, heat maps, and images as described herein). The processing unit 36 ​​includes a processor and a memory. The processor may be, for example, but not limited to, a microprocessor (μP), microcontroller, or other suitable processing device, and is interfaced with the memory. The memory may be one or more of various types of internal and / or external storage media, such as RAM, ROM, EPROM, EEPROM, FLASH®, or other providing storage registers, for data storage, such as an internal storage area of ​​the computer, e.g., computer readable media, and may be volatile or non-volatile memory. The memory stores a number of routines executable by the processor, including routines for implementing the disclosed concepts as described herein.In particular, the processing device 36 includes a cell phenotype characterization component 38 that is configured to classify various cell types and their activation (cell phenotypes) from multi-parameter cellular and subcellular imaging data (e.g., multiplexed or highly multiplexed imaging data) obtained from various imaging modalities as described herein in various embodiments.

[0044] Furthermore, the above description of the disclosed concepts is based on and utilizes in situ multi-parameter cellular and subcellular imaging data. However, it will be understood that this is not meant to be limiting. Rather, it will be understood that the disclosed concepts can be used in conjunction with in vitro microphysiological models for basic research and clinical translation. Multicellular in vitro models allow for the study of spatiotemporal cellular heterogeneity and heterogeneous cell-cell communication that recapitulates human tissues, applicable to investigating mechanisms of disease progression in vitro, testing drugs, and characterizing the structural organization and content of these models that can be used for transplantation.

[0045] Finally, although imaging data obtained from tumor sections has been described above, it will be appreciated that the disclosed concepts may also be applied to imaging data obtained from other types of tissue sections and / or from unsectioned samples using imaging modalities that can penetrate solid, unsectioned samples.

[0046] In the claims, signs placed between parentheses shall not be interpreted as limiting the scope of the claim. The word "comprises" or "includes" does not exclude the presence of elements or steps other than those listed in the claim. In a device claim enumerating several means, several of these means may be embodied by one and the same hardware. The word "a" preceding an element does not exclude the presence of a plurality of such elements. In any device claim enumerating several means, several of these means may be embodied by one and the same hardware. The mere fact that certain elements are recited in mutually different dependent claims does not indicate that these elements cannot be used in combination.

[0047] Although the present invention has been described in detail for the purpose of illustrating based on what are currently considered to be the most practical and preferred embodiments, it is to be understood that such detail is for the purpose only, and that the present invention is not limited to the disclosed embodiments, but on the contrary, is intended to include modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it is to be understood that the present invention contemplates that one or more features of any embodiment can be combined with one or more features of any other embodiment, to the extent possible.

Claims

1. 1. A method for characterizing cell phenotypes from multi-parameter cellular and subcellular imaging data of several tissue samples from several patients or several multi-cellular in vitro models, comprising: receiving segmented multi-parameter cellular and sub-cellular imaging data, the segmented multi-parameter cellular and sub-cellular imaging data being generated by performing cell segmentation on the multi-parameter cellular and sub-cellular imaging data; performing a recursive decomposition on the segmented multi-parameter cellular and sub-cellular imaging data to identify a plurality of computational phenotypes, the recursive decomposition comprising a plurality of levels of decomposition, each level of decomposition comprising soft / probabilistic clustering and spatial regularization, and each cell in the segmented multi-parameter cellular and sub-cellular imaging data is probabilistically assigned to one or more of the plurality of computational phenotypes; A method comprising:

2. 2. The method of claim 1, further comprising generating a visual representation in which each cell in the segmented multi-parameter cellular and subcellular imaging data is probabilistically associated with only one of the plurality of computational phenotypes based on each cell's individual highest membership probability.

3. 2. The method of claim 1, comprising: generating a cell phenotype tree in which each cell in the segmented multi-parameter cellular and subcellular imaging data is probabilistically assigned to one or more of the plurality of computational phenotypes, the cell phenotype tree including a plurality of terminal nodes, each of the plurality of terminal nodes representing one of the plurality of computational phenotypes.

4. The method of claim 3 , wherein the size of each terminal node corresponds to the fraction of cells that are assigned to the computational phenotype of that terminal node.

5. 2. The method of claim 1, wherein the multi-parameter cellular and subcellular imaging data comprises multiplexed to highly multiplexed immunofluorescence biomarker data or multiplexed to highly multiplexed mass spectrometry-based biomarker data.

6. 2. The method of claim 1, further comprising generating parameters for a multi-factor analysis (MFA) model using the plurality of computational phenotypes.

7. 7. The method of claim 6, further comprising applying a Kullback-Leibler (KL) divergence metric with parameters of the MFA model to quantify differences between any two of the plurality of computational phenotypes.

8. The method of claim 7 , further comprising generating a heatmap based on results of applying the Kullback-Leibler (KL) divergence metric.

9. 2. The method of claim 1, wherein the soft / probabilistic clustering includes a plurality of clusters, each corresponding to each of the plurality of computational phenotypes, each cluster being defined as a hierarchical mixture of parametric low-dimensional subspaces, and each cell of the segmented multi-parameter cellular and subcellular imaging data is probabilistically assigned to all of the clusters.

10. 2. The method of claim 1, wherein the recursive decomposition ensures that the multiple computational phenotypes are distinct from one another using a recursive stopping criterion that includes a threshold applied to angles between subspaces of the hierarchical mixture.

11. 10. The method of claim 9, wherein the spatial regularization uses a function that includes a first term that promotes sparsity in cluster assignment, meaning that each cell attempts to belong entirely to one of the clusters, and a second term that promotes spatial coherence.

12. The method of claim 11 , wherein the function is optimized using the Alternating Directional Method of Multipliers (ADMM).

13. 4. The method of claim 3, wherein the recursive decomposition employs stopping criteria that stop the recursion when (i) the resulting clusters obtain attributions for less than 1% of the cells, (ii) the cell phenotype tree is determined to have exceeded a predetermined depth threshold, or (iii) the angle between the mean vectors and the factor loading space are both below a predetermined threshold.

14. A non-transitory computer readable medium having stored thereon one or more programs comprising instructions that, when executed by a computer, cause the computer to perform the method of claim 1.

15. 1. A computer system for characterizing a cell phenotype, comprising: The apparatus includes a processing device, the processing device comprising: receiving segmented multi-parameter cellular and sub-cellular imaging data, the segmented multi-parameter cellular and sub-cellular imaging data being generated by performing cell segmentation on the multi-parameter cellular and sub-cellular imaging data; performing a recursive decomposition on the segmented multi-parameter cellular and sub-cellular imaging data to identify a plurality of computational phenotypes, the recursive decomposition comprising a plurality of levels of decomposition, each level of decomposition comprising soft / probabilistic clustering and spatial regularization, wherein each cell in the segmented multi-parameter cellular and sub-cellular imaging data is probabilistically assigned to one or more of the plurality of computational phenotypes; A computer system comprising a cell phenotype characterization component configured to perform the steps of:

16. 16. The system of claim 15, wherein the cellular phenotype characterization component is further configured to generate a visual representation in which each cell in the segmented multi-parameter cellular and subcellular imaging data is probabilistically associated with only one of the plurality of computed phenotypes based on each cell's individual highest membership probability.

17. 16. The system of claim 15, wherein the cellular phenotype characterization component is further configured to generate a cellular phenotype tree in which each cell in the segmented multi-parameter cellular and subcellular imaging data is probabilistically assigned to one or more of the plurality of computational phenotypes, the cellular phenotype tree including a plurality of terminal nodes, each of the plurality of terminal nodes representing one of the plurality of computational phenotypes.

18. The system of claim 17 , wherein the size of each terminal node corresponds to the fraction of cells that are assigned to the computational phenotype of that terminal node.

19. 16. The system of claim 15, wherein the multi-parameter cellular and subcellular imaging data comprises multiplexed to highly multiplexed immunofluorescence biomarker data or multiplexed to highly multiplexed mass spectrometry-based biomarker data.

20. 16. The system of claim 15, wherein the cellular phenotype characterization component is further configured to generate parameters for a multi-factor analysis (MFA) model using the plurality of computational phenotypes.

21. 21. The method of claim 20, wherein the cellular phenotype characterization component is further configured to apply a Kullback-Leibler (KL) divergence metric with parameters of the MFA model to quantify differences between any two of the plurality of computational phenotypes.

22. 22. The system of claim 21, wherein the cell phenotype characterization component is further configured to generate a heatmap based on results of applying the Kullback-Leibler (KL) divergence metric.

23. 16. The system of claim 15, wherein the soft / probabilistic clustering includes a plurality of clusters, each corresponding to each of the plurality of computational phenotypes, each cluster being defined as a hierarchical mixture of parametric low-dimensional subspaces, and each cell of the segmented multi-parameter cellular and subcellular imaging data being probabilistically assigned to all of the clusters.

24. 16. The system of claim 15, wherein the recursive decomposition uses a recursive stopping criterion that includes a threshold applied to angles between subspaces of the hierarchical mixture to ensure that the multiple computational phenotypes are distinct from one another.

25. 24. The system of claim 23, wherein the spatial regularization uses a function that includes a first term that promotes sparsity in cluster assignment, meaning that each cell attempts to belong entirely to one of the multiple clusters, and a second term that promotes spatial coherence.

26. 26. The system of claim 25, wherein the function is optimized using an alternating direction multiplier method (ADMM).

27. 18. The system of claim 17, wherein the recursive decomposition employs stopping criteria that stop the recursion if (i) the resulting clusters obtain attributions for less than 1% of the cells, (ii) the cell phenotype tree is determined to have exceeded a predetermined depth threshold, or (iii) the angle between the average vectors and the factor loading space are both below a predetermined threshold.

Images