3D cell and tissue image analysis for cell and subcellular morphology modeling and classification

Abstract

The ability to automate the process of specimen collection, image acquisition, data preprocessing, calculation of derived biomarkers, modeling, classification, and analysis can significantly impact clinical decision making and basic research on cell deformation. The present disclosure combines 3D cell nucleus shape modeling into a highly parallel pipeline workflow protocol through robust smooth surface reconstruction and extraction of shape morphometric metrics for end-to-end morphological analysis of thousands of nuclei and nucleoli in 3D. The method allows for efficient and information-rich assessment of cell shape from imaging data and represents a reproducible technique that can be validated, modified, and reused by the biomedical community. This promotes result reproducibility, collaborative method validation, and broad knowledge dissemination.

Description

[0001] Government Terms

[0002] This invention was made with government support under NR015331, NS091856; DK089503 and EB020406 awarded by the National Institutes of Health and IIS-636840 awarded by the National Science Foundation. The government has certain rights in this invention.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to U.S. Patent Application No. 16 / 277,128, filed on February 15, 2019, and also claims the benefit of U.S. Provisional Application No. 62 / 632,663, filed on February 20, 2018. The entire disclosure of the above application is incorporated herein by reference. Technical Field

[0005] The present disclosure relates to three-dimensional cell and tissue image analysis for cellular and subcellular morphology modeling and classification. Background Art

[0006] Tissue and cell deformation are regulated by complex underlying biological mechanisms that influence spatial and temporal morphological changes. Understanding these processes through quantitative analysis of three-dimensional changes in size and shape of cellular and subcellular structures is important not only for studying cellular organization but also for the detection and treatment of pathological conditions such as cancer. However, the dimensionality and quality of imaging data and the large variability of cellular and subcellular shapes across populations pose challenges for three-dimensional shape methods, which should allow accurate morphological analysis, be scalable, robust to noise and at the same time have sufficient specificity across populations. Therefore, robust three-dimensional cellular and subcellular morphometric techniques with high-throughput capabilities are urgently needed to perform population analysis.

[0007] This section provides background information related to the present disclosure which is not necessarily prior art. Summary of the invention

[0008] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0009] An automated method for analyzing biological cells is proposed. The method includes: staining a component of a biological sample, wherein the biological sample includes at least one cell; receiving image data of the biological sample, wherein the image data provides a three-dimensional representation of the biological sample; marking a component of at least one cell in the image data; for each marked cell nucleus in the image data, constructing a mathematical representation of the boundaries defining a given cell nucleus; using the mathematical representation of the cell nucleus to extract features for each of the marked cell nuclei, wherein the features are measures of the shape and size of the marked cell nucleus; for each marked cell nucleolus, constructing a mathematical representation of the boundaries defining a given cell nucleolus; using the mathematical representation of the cell nucleolus to extract features from each of the marked cell nucleoli, wherein the features are measures of the shape and size of the marked cell nucleolus; storing two or more models for cell classification; and classifying the biological sample by comparing the extracted features of the marked cell nuclei and the marked cell nucleoli with the stored models.

[0010] The image data may be generated by a confocal microscope or another type of imaging device.

[0011] In one embodiment, the mathematical representation is constructed using iterative Laplace-Beltrami eigenprojections and boundary deformations.

[0012] In some implementations, labeling the components includes segmenting the image data into volumes representing the components.

[0013] Features extracted from the labeled cell nuclei may include, but are not limited to, determining one or more of the following: the volume of the labeled cell nucleus, the surface area of ​​the labeled cell nucleus, the average curvature of the labeled cell nucleus, the shape index of the labeled cell nucleus, the curvature index of the labeled cell nucleus, and the fractal dimension of the labeled cell nucleus.

[0014] Features extracted from the labeled nucleoli may include, but are not limited to, determining one or more of the following: the count of nucleoli in the corresponding cell nucleus, the volume of the labeled nucleoli, the surface area of ​​the labeled nucleoli, the average curvature of the labeled nucleoli, the shape index of the labeled nucleoli, the curvature index of the labeled nucleoli, and the fractal dimension of the labeled nucleoli.

[0015] In one embodiment, a random forest classification method is used to classify biological samples.

[0016] In other embodiments, the biological sample is classified using a classification method selected from the group consisting of a linear classifier, a k-nearest neighbor method, a decision tree method, a neural network, and a support vector machine.

[0017] Further areas of applicability will become apparent from the description provided herein.The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0019] Figure 1 An overview of automated methods for morphological modeling and classification is provided.

[0020] Figure 2 is a flow chart depicting an example method for segmenting image data containing one or more cells.

[0021] Figure 3 is a flow chart depicting an example method for performing surface remodeling of a cell or subcellular organelle or component (eg, nucleus, nucleolus).

[0022] 4A to 4D Images of robust smooth surface reconstruction for 3D visualization, respectively: a binary mask representation of a nucleus segmented from a fibroblast cell image; three binary masks of nucleoli segmented within the nucleus; a mesh representation of the reconstructed smooth surface of the nucleus; and three mesh representations of the nucleolus surface color-coded along the Z axis.

[0023] Figure 5 is a graph depicting the (local) geometry of a 2-manifold including the definition of the curvature of each vertex relative to a local coordinate frame.

[0024] Fig. 6A and Figure 6B An exemplary graphical end-to-end workflow in the LONI pipeline client interface is shown, including an overview of the proven end-to-end workflow protocol showing nested groups of modules; an extended volume-to-shape group including modules that perform 3D shape modeling refinement; and an extended morphometric group including modules that perform morphometric measurement extraction.

[0025] Fig. 7A and Figure 7B is a graph depicting the results of fibroblast classification. Fig. 7A In , the average AUC for various cell group sizes is shown; while Figure 7B In the figure, the top 10 features used for classification are shown by importance score (right, nucleolar feature names begin with Avg, Min, Max, or Var, and feature names also reported in the top 10 for PC3 cells are shown in blue font).

[0026] Fig. 8A and Figure 8B is a graph depicting the PC3 classification results. Fig. 8A In , the average AUC for various cell group sizes is shown; while Figure 8B Middle, top 10 features for classification by importance score (right, nucleolar feature names starting with Avg, Min, Max, or Var, feature names also reported in the top 10 for fibroblasts are shown in blue font).

[0027] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0029] Tissue and cell morphology are regulated by complex underlying biological mechanisms associated with cell differentiation, development, proliferation and disease. Changes in nuclear form reflect the reorganization of chromatin structure and are associated with altered functional properties such as gene regulation and expression. At the same time, many studies in mechanobiology have shown that geometric constraints and mechanical forces act on cell deformities and in turn affect nuclear and chromatin dynamics, gene and pathway activation. Therefore, with the emergence of studies on the reorganization of chromatin and DNA structure in spatial and temporal frameworks, known as the 4D nucleome, the morphological quantification of cells or organelles and components has become of great significance. The cell structures of interest in the context of the 4D nucleome include not only the nucleus itself, but also the nucleolus and nucleolus-associated domains, chromosome domains, topologically associated domains, lamina-associated domains, and ring domains in transcription factories. In addition, understanding these processes by quantitatively analyzing morphological changes also has many medical implications, such as the detection, understanding and treatment of pathological conditions such as cancer. In the literature, cell morphometry is generally used to analyze forms containing quantitative measurements of the size and shape of cells, organelles, components and other cell structures.

[0030] Although much effort has been made to develop cell shape features in 2D or pseudo-3D, several studies have demonstrated that 3D morphometric metrics provide better results for nuclear shape description and discrimination compared to 2D features. However, 3D shape descriptors that allow for robust morphological analysis and facilitate human interpretation are still under active investigation. In addition, the dimensionality and amount of acquired data, the various image acquisition conditions, and the large variability of cell shape in a population pose challenges to 3D shape analysis methods that should be scalable, robust to noise, and at the same time sufficiently specific across cell populations. Therefore, robust 3D cell morphometric techniques with high-throughput capabilities are urgently needed to perform population analysis.

[0031] The present disclosure has two complementary goals. The first goal is to evaluate and validate 3D morphometric metrics for cell or subcellular organelle and component shape description. First, the surface of the 3D mask extracted from microscopic data is reconstructed using Laplace-Beltrami eigen-projection and topology-preserving boundary deformation. Then, intrinsic and extrinsic geometric metrics are calculated, which are used as derived signature vectors (shape biomarkers) to characterize the complexity of 3D shape and distinguish between observed clinical and phenotypic traits. These metrics include volume, surface area, mean curvature, curvature, shape index and fractal dimension. The proposed modeling and analysis methods can be used for shape quantification of cells, nuclei and other subcellular organelles and components.

[0032] The second goal is to develop a reproducible pipeline workflow that realizes the entire process that can be customized and expanded to explore the association between 3D cells and subcellular shape phenotypes in health and disease in depth. High-throughput imaging (HTI) can include the automation of liquid handling, image acquisition based on microscope, image processing and statistical data analysis. This work focuses on the last two aspects of this limitation. A simplified multi-step protocol is realized, which relies on a variety of different tools and is seamlessly connected in the LONI pipeline workflow. The workflow meets the modern standards for high-throughput imaging processing and analysis, and is mostly automated, with an emphasis on effectiveness and reproducibility.

[0033] Figure 1 A high-level view of an end-to-end protocol is shown. As a starting point, a biological sample including at least one cell is imaged at 11 using a three-dimensional image acquisition technique, including but not limited to: microscopy such as confocal microscopy, multiphoton microscopy, light sheet microscopy, and super-resolution microscopy; tomography such as fluorescence activated cell sorting (FACS) tomography, holographic tomography, and optical coherence tomography; or photoacoustic imaging. In one embodiment, 3D confocal imaging is performed using a Zeiss LSM 710 laser scanning confocal microscope with a 63xPLAN / Apochromate 1.4na DIC objective. Microscope imaging generates image data for the biological sample, wherein the image data provides a three-dimensional representation of the biological sample. In order to improve classification, the biological sample preferably includes three to twenty cells, as will be described in more detail below. It should be understood that the biological sample can be taken from a tissue or cell culture.

[0034] The image data is segmented at 12 into individual volumes representing different components (i.e., organelles) of cells in the biological sample. For example, the boundaries of the nuclei in the cells are marked in the image data. Similarly, the nucleoli in each nucleus are marked in the image data. A separate mask (i.e., a binary image) can be generated for each component element in the cell. In an example embodiment, the mask is converted into a standardized file format such as the Neuroimaging Information Technology Initiative (NIFTI). Segmentation may also include other filtering or signal processing, as will be further described below.

[0035] For each labeled cell or subcellular organelle and component in the image data, surface reconstruction is performed at 13. That is, a mathematical representation of the boundary that defines a given cell nucleus is constructed. Therefore, the boundary surface for each labeled object is represented by a polygonal mesh. In an example embodiment, an iterative Laplace-Beltrami eigen-projection and boundary deformation method is used to construct a mathematical representation of the surface for a given nucleus. Additional information about this example surface reconstruction method is described by Yonggang Shi et al. in "Robust Surface Reconstruction via Laplace-Beltrami Eigen-Projection and Boundary Deformation" IEEE Transactions on Medical Imaging, Vol. 29, No. 12, December 2010, the entire contents of which are incorporated herein. The present disclosure also contemplates other methods for constructing mathematical representations of boundary surfaces.

[0036] Next, at 14, features are extracted for each of the labeled cells or subcellular organelles or components based on a mathematical representation of the surface boundaries of the cells or subcellular organelles or components. The features are measures of the shape and size of the labeled cells or subcellular organelles or components. For example, the features extracted from the labeled cells or subcellular organelles or components may include, but are not limited to: the volume of the cells or subcellular organelles or components, the surface area of ​​the cells or subcellular organelles or components, the average curvature of the cells or subcellular organelles or components, the shape index of the cells or subcellular organelles or components, the curvature index of the cells or subcellular organelles or components, and the fractal dimension of the cells or subcellular organelles or components.

[0037] In an example embodiment, the feature is a measure of the shape and size of the cell nucleus of the mark. For example, the feature extracted from the cell nucleus of the mark may include but is not limited to: the volume of the cell nucleus, the surface area of ​​the cell nucleus, the average curvature of the cell nucleus, the shape index of the cell nucleus, the curvature index of the cell nucleus, and the fractal dimension of the cell nucleus. Features for each of the cell nucleoli of the mark can also be extracted. Again, the feature is a measure of the shape and size of the cell nucleoli of the mark. Example features include but are not limited to: the count of the nucleoli in the cells of the mark, the volume of the cell nucleoli, the surface area of ​​the cell nucleoli, the average curvature of the cell nucleoli, the shape index of the cell nucleoli, the curvature index of the cell nucleoli, and the fractal dimension of the cell nucleolus. It is easy to understand that other types of features can be extracted and used for classification.

[0038] Finally, at step 15, the cells in the biological sample are classified. In one embodiment, the geometric features extracted from the cells can be used to distinguish between healthy cells and cancer cells. Before imaging, two or more models for classification are determined. By comparing the extracted features (i.e., in the form of feature vectors) with the stored models, the cells in the organism can be classified. In one embodiment, the cells are classified using a random forest classification method, as will be further described below. Although with reference to a specific classification method, the following other classification methods also fall within the scope of the present disclosure, including linear classifiers, k nearest neighbor methods, decision tree methods, neural networks, and support vector machines, etc.

[0039] Example embodiments are described in more detail below.Before imaging, the composition of biological samples can be marked (e.g., dyed).In example embodiments, cells are labeled with three different fluorophores: DAPI (4', 6-diamidino-2-phenylindole) is a common dye for nucleus, and fibrillarin antibody (fibrillarin) and ethidium bromide (EtBr) are both used for nucleolar staining.Although fibrillarin is a commonly used nucleolar marker, it is found that fibrillarin is too specific to shape modeling purposes due to the large local intensity variation in the detected nucleolus, which makes the extraction of mask shape a difficult problem.It has been shown that ethidium bromide can be used for dyeing dense chromatin, nucleolus and ribosomes.It is found that ethidium bromide provides a better overall representation of nucleolar shape.Combining specific fibrillarin via co-localization with ethidium bromide to confirm the correct detection of nucleolar position, as described below.

[0040] about Figure 2Segmentation is further described. Based on the labeling, the image data can be separated into two or more channels, as indicated at 21. In an example embodiment, the channels are separated and saved as individual volumes labeled c0, c1, c2 representing DAPI, fibrin, EtBr channels, respectively. Each channel-specific volume is then re-sliced ​​into a 1024×1024×Z grid (Z=[10,50]), where the regional subvolumes help align with the native tile size of the microscope. All subvolumes are then converted (lossless) and saved as multi-image 3D TIFF volumes.

[0041] For each subvolume, accompanying microscope metadata including scaling of all three dimensions and resolution is extracted from the raw data at 22 and passed further to downstream modules to account for image anisotropy. Each subvolume is also converted to grayscale (e.g., 8-bit resolution) at 23. Despeckle or other noise reduction methods may be applied to the subvolumes.

[0042] The nucleus and nucleolus are segmented from the volume at 24. For example, automatic 3D segmentation of the nucleus in the DAPI channel is performed by applying a nuclear segmentation algorithm, such as one found in the Farsight toolkit, to the DAPI channel subvolume. The Farsight toolkit was chosen for several reasons: it is a CLI tool created specifically for segmenting DAPI-stained nuclei in 2D / 3D and is very easy to use. The Farsight toolkit also does not require labeled training sets and stable results for data presentation. The nuclear segmentation algorithm implements the following multiple steps, including: a graph cut algorithm binarizes the subvolume; a multi-scale Laplacian of a Gaussian filter converts the nucleus into a spot mask; fast clustering to delineate the nucleus; and a graph cut with alpha expansion is used to refine the nuclear outline. After segmentation, the data is converted into a 16-bit 3D TIFF file. Each segmented nucleus is represented as a mask and is assigned a unique index value. It is easy to understand that other nuclear segmentation methods also fall within the scope of the present disclosure.

[0043] The automatic 3D segmentation of nucleoli in fibrin and ethidium bromide channels is also performed using, for example, the Weka data mining software package. In an example embodiment, the trainable Weka segmentation plug-in is bundled with Fiji, which is an image processing and visualization software package based on ImageJ. Intranuclear segmentation is performed independently for nucleoli stained with ethidium bromide (EtBr) and fibrin. The trainable Weka segmentation plug-in is the most popular segmentation tool in the ImageJ ecosystem and is relatively easy to use to mark biological structures in 3D images. The DAPI nuclear mask is used to define the segmentation space in the EtBr and fibrin channels, where the purpose is to isolate the subnuclear segmentation in the nucleus from the object. A classifier model is created for each channel by training with a random selection of 10% of the subvolumes in the channel. The model is then applied to all subvolumes in the channel using a random forest algorithm. A nucleolus mask is created based on the resulting probability map. The "Find Connected Regions" ImageJ / Fiji plug-in is then used to perform connected component labeling to uniquely index each nucleolus. Segmentation artifacts are filtered using the quality control protocol described below for the nuclear mask. Finally, both the EtBr and fibrin segmentation volumes were used as input to the co-localization algorithm to validate the segmented EtBr stained nucleoli based on the presence of fibrin. It will be readily appreciated that other nucleolar segmentation methods also fall within the scope of the present disclosure.

[0044] Uneven staining often results in occasional segmentation artifacts, such as complex resulting masks, including "handles", "holes" or irregular boundary shapes with singular points. This problem is usually solved by applying various topology fixation techniques, as indicated at step 25. An exemplary post-processing step for the nuclear mask includes hole filling using a set of MATLAB functions. Filtering of artifacts can also be performed using other strategies such as a Java application that measures the spherical compactness of the identified object, estimates its volume by voxel counting, and detects objects that cross the edge of the tile or are connected to other objects. Critical values ​​for compactness and voxel counts are selected empirically to remove most artifacts. This filter, together with hole filling correction, helps generate masks that are more suitable for morphometric analysis. The quality control protocol corrects or screens out potential non-zero masks and marks them for further refinement. Finally, the masks that pass quality control are converted to a neuroimaging format, such as NIFTI format, at 26. These masks are used in subsequent shape morphometric analysis in the pipeline workflow.

[0045] The dataset used as input to the morphometric feature extraction part of the workflow consisted of a collection of 3D segmented and uniquely labeled volumes from channel c0, representing binary nuclear masks, accompanied by a set of 3D binary masks of the nucleoli for each nucleus from channel c2, filtered by a colocalization process with c1, as described above. Segmented, quality-controlled nuclear and nucleolar volumes in compressed NIFTI format (.nii.gz) were converted from voxel masks to triangulated shape manifolds and automatically processed using the developed workflow.

[0046] To model the 3D shapes of the nucleus and nucleolus, the boundaries of the 3D masks of the 3D shapes of the nucleus and nucleolus extracted from the microscopy data were modeled as a 2D manifold (with 2 spheres S 2 homeomorphism), which uses iterative Laplace-Beltrami eigenprojections and a topology-preserving boundary deformation algorithm as a triangulated surface embedding About Figure 3 The method is further described.

[0047] Objects are defined in the output received from the segmentation process. For each object, a grid representation is constructed at 33 from the boundaries of the binary mask of the object. As indicated at 32, the mask is binarized before constructing the grid representation. The boundaries are projected onto a subspace of its Laplace-Beltrami eigenfunction, which allows the position of pseudo-features to be automatically located by calculating the metric distortion in the eigenprojection. The Laplace-Beltrami eigenfunction is intrinsically defined and can be easily calculated from the boundary surface without any parameterization. They are also isometry invariant and are therefore robust to the jagged characteristics of the boundary surface, which is desired for biomedical shape analysis. The magnitude of the eigenvalues ​​of the Laplace-Beltrami operator intuitively corresponds to the frequency in Fourier analysis, so it provides a convenient mechanism to control the smoothness of the reconstructed surface.

[0048] Using this information, the next step is a mask deformation process at step 34, which removes spurious features while keeping the rest of the mask intact, thereby preventing unexpected volume shrinkage. The deformation is topologically preserved and well-formed so that the boundary surface of the mask is a manifold. Repeat steps 33 and 34 as indicated at 35 until convergence. Finally, at 36, the method generates a final surface as an eigenprojection of the mask boundary, which is a smooth surface with a zero topology. As long as the shape topology is homeomorphic to a sphere, this manifold representation of the regional organelle boundary contributes to the algorithmic understanding of all shapes, such as including crescents, multi-lobed shapes, and folded shapes. The exemplary results of this step are presented in Figure 4. In this way, the robust reconstruction of the nucleus and nucleolar surfaces is performed as a zero two-dimensional manifold from their segmentation masks using iterative mask filtering. Additional details on this process are described by YG Shen et al. in “Robust Surface Reconstruction via Laplace-Beltrami Eigen-Projection and Boundary Deformation” IEEE Transactions on Medical Imaging 2010, which is incorporated herein in its entirety.

[0049] In an example implementation, six shape metrics are used as features to quantify the geometric characteristics of a 3D surface. To compute these metrics, the principal (minimum and maximum) curvatures (κ1 ≤ κ2) are computed using a triangulated surface model representing the boundary of a zero solid. The shape morphometric metrics can then be represented by the following principal curvatures: mean curvature, shape index, and curvature. The mean curvature is defined as The shape index is defined as And the curvature is limited to The principal curvatures of a surface are the eigenvalues ​​of the Hessian matrix (second fundamental form), which solves k[H-kI]=0, where I is the identity matrix. If S is a surface of second fundamental form H(X,Y), p∈M are fixed points, and u,v represent the orthogonal basis, the tangent vector at p, then the principal curvatures are the eigenvalues ​​of the symmetric Hessian matrix, Also called shape tensor. Let r = r (u, v) be the surface Parameterization, which means that with respect to u and v, r u and r v The partial derivative of is a smooth vector-valued function of two variables, such as Figure 5 Then, the Hessian coefficient H at a given point (p) in the parameter u,v plane is i,jFrom the second partial derivative of r at this point to The projection on the normal line is given by , and the Hessian coefficient H i,j The dot product operator can be used to calculate: u,u =r u,u ·n,H u,v =H u,v =r u,v ·n,H v,v =r v,v ·n.

[0050] Three other shape metrics are volume, surface area, and fractal dimension. Volume is the amount of 3D space enclosed by a closed bounding surface and can be expressed as Among them I D (x, yz) represents the index function of the region of interest (D). If r(u, v) is a continuously differentiable function, and the normal vector to the surface on the appropriate region D in the plane of parameters u, v is given by If S Ω :r=r(u,v),(uv)∈Ω is the parametric surface representation of the region boundary. Then, the surface area can be expressed as The fractal dimension calculation is based on the fractal scaling ratio ρ and the number of replacement parts N. Exact discrete approximations of these metrics are used to calculate them on surfaces represented by grids, as described by M. Meyer et al., "Discrete Differential-Geometry Operators for Triangulated 2-Manifolds" Visualization and Mathematics III, Berlin, Heidelberg: Springer Berlin Heidelberg; 2003, pp. 35-57; and by A. Jagannathan, "Segmentation and Recognition of 3D Point Clouds within Graph-theoretic and thermodynamic frameworks" a thesis. Thesis (Ph D), Northeastern University, 2005.

[0051] The extracted 3D morphometric measures are used as features for the feature vector. In one embodiment, the feature vector includes measures for each nucleus. Specifically, the features for each nucleus include the volume of the nucleus, the surface area of ​​the nucleus, the mean curvature of the nucleus, the shape index of the nucleus, the curvature index of the nucleus, and the fractal dimension of the nucleus. The feature vectors are then used to train a number of machine learning classifiers, for example using the open source Python package scikit-learn 0.17.0. In order to improve the behavior of the classification algorithm, data preprocessing includes the following standard steps: variable standardization to zero mean and unit variance; and normalization for scaling individual samples to a single norm, which is calculated separately from the training set at each step of the cross validation.

[0052] In an example embodiment, the nucleolar data is aggregated with the morphometric metric for each nucleus. For example, the number of nucleoli in each nucleus detected is included as an individual feature of the feature vector. Different methods for merging nucleolar level features were studied, including customized nucleolar level difference metrics and multiple instance learning frameworks. The best results were obtained by aggregating the nucleolar data within each nucleus by estimating the density of each morphometric metric. For each nucleus, sample statistics (e.g., average, minimum, maximum, and higher moments) are calculated for each morphometric metric across the nucleolus. These statistics are used to enhance the signature feature vector of the corresponding parent nucleus so that all feature vectors have the same length. Accordingly, nuclei that do not have any automatically detected internally positioned nucleoli are excluded from further analysis so that for each nucleus, there is at least one nucleolus. In general, due to the significant population heterogeneity of the observed cell phenotypes, the correct classification (type, stage, treatment, etc.) of each individual cell is a challenging task. For example, the same sample may contain a close mixture of intertwined "cancerous" and "non-cancerous" cell phenotypes, or these two types may include apoptotic cells that exhibit similar shapes or sizes. Given the nature, culture, preparation and collection of cell samples, the classification of cell groups rather than individual cells is considered. The rationale behind this is based on the observation that even if the algorithm misclassifies a few cells in a sample, as long as the majority of cells are correctly classified, the final (cell group) marker will still be correctly assigned. Using this strategy, classification is preferably performed on small cell groups of 3 to 19 cells each. During each fold of the internal cross-validation, these small cell groups are randomly repeated 1000 times by a bootstrap process.

[0053] Although the LONI pipeline is a popular tool in neuroimaging and bioinformatics, it has been ignored by the cell biological image analysis community so far. In the present disclosure, the LONI pipeline is used to implement a simplified multi-step protocol that relies on a variety of different tools and solutions that are seamlessly connected in the LONI pipeline workflow, as shown in Figure 6. From a high-level perspective, each step of the data processing and analysis protocol is packaged as an individual module in the workflow, which provides input specifications and output specifications that allow the pipeline to automatically connect and manage these atomic modules. Therefore, a distributed, large-scale parallel implementation of the protocol enables thousands of cores and nucleoli to be easily processed at the same time. The workflow does not depend on the total number of 3D objects, biological conditions, or a large number of running instances, because once the workflow configuration including job scheduling and resource allocation is fixed, the execution of the workflow configuration is fully automated. During execution, the workflow provides researchers with real-time information about progress, allows viewing intermediate results at each individual step, and easily checks and restarts failed modules or specific instances. This implementation is also very flexible and is not limited to the specific tools included in the workflow. This implementation can be reused for a variety of different experiments by adjusting parameters and replacing individual modules, while retaining high-throughput capabilities.

[0054] The workflow is configured in such a way that it can consume data in a format for sharing it, namely 1024×1024×Z 3D volumes as 16-bit 3D TIFF files in three channels. Each volume is processed independently in parallel, so that the workflow automatically limits the processing required to analyze all input data. In addition, the parallelization of the workflow automatically branches out and collapses as needed. For example, from preprocessing to segmentation, the workflow execution starts M processes, where M is a number of input volumes. Since segmentation produces multiple outputs, after this step, the workflow automatically starts M×[N1,N2,…,N M ] processing, where N X is a number of segmented objects (e.g., nuclei) from the input volume X. At the post-processing step, some segmented objects are filtered out by the curation module and excluded from further analysis. The workflow automatically reduces multiple processes to the number of masks that pass the curation, and performs 3D shape modeling and morphometric feature extraction independently on individual masks, which enables one to run up to 1200 jobs on the cluster simultaneously during the experiment, thereby effectively reducing the computational time. Finally, the workflow collects morphometric information from each individual mask and combines them in a result table, which is further used as input to the classification algorithm. This capability allows the use of modern computing resources, relieves researchers of the burden of low-level configuration, makes it easier to control the execution of processing, and improves the reproducibility of end-to-end processing.

[0055] To validate the shape morphometric metrics, the metrics were first applied to synthetically generated 3D masks. The Scikit-image Python library was used to create 3D entities representing a cube, an octahedron, a sphere, an ellipsoid, and three overlapping spheres with linearly aligned centers. All of these objects were processed and the resulting shape morphometric metrics were compared. In particular, it was desired to confirm the close relationship between the (expected) analytically derived measures of volume and surface area calculated using the corresponding shape parameters (e.g., radius, size) and their computationally derived counterparts reported by the processing pipeline workflow. The results show that for core-like shapes such as spheres and ellipsoids, the calculated errors are within 2%. For more geometric objects such as cubes and octahedrons, the calculated errors are within 6%. The increased error in the latter case can be explained by applying mesh smoothing at the shape vertices to resolve singularities (e.g., smooth but non-differentiable surface boundaries). To demonstrate the detection of shape differences between different types of 3D objects, overlapping spheres are compared to circumscribed ellipsoids. As expected, the mean values ​​of mean curvature and curvature measures are lower for the sphere, while the shape index values ​​are higher compared to the ellipsoid. A progressive monotonic shape morphometric metric trend is observed when comparing the sphere, ellipsoid, and overlapping spheres. This simulation demonstrates the ability to accurately measure size and shape features of 3D objects, which forms the basis for machine learning-based object classification based on boundary shape.

[0056] To evaluate the selected shape morphometric measures as discriminative features, they were compared with the SPHARM coefficients for single fibroblast nucleus classification. Fibroblasts (newborn males) were purchased from ATCC (BJ fibroblasts CRL-2522 normal) and subjected to a G0 / G1 serum starvation protocol. This protocol provides images of the following conditions or phenotypes: cell cycles synchronized by serum starvation (SS) and proliferation (PROLIF). Thus, 962 nuclear masks were successfully processed by the following two methods: 466 PROLIF and 496 SS.

[0057] For method comparison, kernel binary masks were extracted. The inventors calculated shape morphometric measures as described above. To obtain the SPHARM coefficients, the popular SPHARM-MAT toolbox was used, which uses the CALD algorithm to implement surface reconstruction and spherical parameterization, then the object surface is extended to a complete set of spherical harmonic basis functions of degree l=13 (default setting), and finally, the mean square distance between corresponding surface parts is minimized by the SHREC method, as described by L. Shen et al. in "Spherical mapping for processing of 3D closed surfaces" to calculate the SPHARM shape descriptor and use it as a feature vector for classification.

[0058] The open source Python package Scikit-learn 0.17.0 was used to test many commonly used machine learning classification methods on the derived feature vectors, with each method having default parameters and the same random seed when applicable. The performance was compared using a 5-fold cross-validation technique and the area under the receiver operating characteristic curve (AUC) as a metric. As shown in Table 1 below, the 3D shape morphometric metrics not only demonstrated discriminative performance comparable to the SPHARM coefficients, but also outperformed them using all tested algorithms.

[0059] Table 1 - Comparison of SPHARM coefficients and shape morphometric descriptors for classification of single cell fibroblast nuclei

[0060]

[0061] As mentioned above, after both core and nucleolus were segmented and morphometric feature extraction was performed, the complete data set for fibroblast classification was composed of 965 cores (498 SS and 470 PROLIF) and 2181 nucleoli (1151 SS and 1030 PROLIF) in total. The best result of a single classifier was realized using the stochastic gradient boosting classifier with 1500 base learners, maximum tree depth 8, learning rate 0.01, subsampling rate 0.5 and the minimum sample number at leaf node 3. The grid search by cross validation has been fine-tuned to hyperparameter. In order to evaluate these classification results, accuracy, precision, sensitivity and AUC were measured in 7 times of internal statistics cross validation of 10 random repetitions, which are presented in the table 2 for single cell and 9-cell group classification.

[0062] Table 2 - Classification accuracy of single fibroblast cells and 9-cell groups

[0063]

[0064] Fig. 7A The average AUC values ​​for group sizes from 3 to 19 cells are shown. When classifying groups with 9 cells, 95% accuracy is achieved, while for groups with 15 or more cells, 98% accuracy is achieved. The gradient boosting classifier also calculates and reports the cross-validated feature importances, as Figure 7B These allow one to assess which measurements differ significantly between the two cell conditions and potentially propose novel research hypotheses that can be tested using prospective data. Both nuclear (top 3) and nucleolar (5 of the top 10) morphometric features were reported to be of high importance for distinguishing SS fibroblasts from PROLIF. During feature selection, higher moment statistics for nucleolar shape morphometrics were eliminated from the feature list because they did not contribute much to the classification, which may be due to the small sample size—most cells had only 1 to 3 nucleoli per nucleus.

[0065] Throughout the process of developing into metastasis, malignant cancer cells undergo a series of reversible changes between intermediate phenotypic states defined by pure epithelium and pure mesenchyme. These changes in prostate cancer are associated with quantitative changes in nuclear structure. Microscope slides or prostate cancer cell line P3 are cultured in epithelial (EPI) and mesenchymal transition (EMT) phenotype states. The exported data set consists of 458 nuclear (310 EPI and 148 EMT) and 1101 nucleoli (649 EPI and 452 EMT) masks. Random uniform subsampling is used to solve the larger sample size imbalance between the 2 classes. In each fold of the 7-fold cross-validation process, there are about 250 cells in the training set, and about 40 cells in the test data.

[0066] In this case, the best classification by a single classifier was the result of applying a random forest model (1000 trees, maximum tree depth 12, maximum number of features for the best split of 40%). The hyperparameter tuning, accuracy metrics, and cross-validation procedures were the same as reported in the previous fibroblast experiments. Similarly, for fibroblast cell classification, a mean accuracy of 95.4% was achieved for groups of 9 cells, increasing to 98% for groups of 15 or more cells (see Table 3). Fig. 8A The AUC for different group sizes is reported to show how the classification accuracy increases with the cell group size and reaches 98% for a group of 13 cells. In this experiment, the feature importance reported by the classifier was also examined. The top 10 important features in this classification included shape morphometric features of the nucleus (3 out of the top 10, 2 of the 3 were also fibroblasts) and nucleoli (top 5).

[0067] Table 3: Classification accuracy of PC3 single cells and 9-cell groups

[0068]

[0069] In this disclosure, protocols are presented that provide a high-throughput, largely automated ("human in the loop") solution for 3D modeling, morphological feature extraction, and classification of cell types or therapeutic conditions. Compared to other studies using 2D projections, this approach operates natively in 3D space and utilizes extrinsic and intrinsic morphometric metrics that are more representative of realistic, fundamental nuclear and nucleolar geometry and enable easy human interpretation. Robust surface reconstruction allows accurate approximation of 3D object boundaries validated on synthetic data. The proposed shape morphometric metrics outperform another popular approach and demonstrate their generalizability across different cell types, conditions, and even domain structures.

[0070] The final end-to-end protocol is highly parallel, and the throughput of the end-to-end protocol is actually limited by the number of computing nodes, so it can process thousands of objects simultaneously with minimal human intervention. The pipeline workflow takes advantage of the diversity of biological image analysis software and integrates many open source tools for different steps of data processing and analysis. The modularity of the workflow enables high reusability and easy modification. This allows the same workflow to be used or customized and extended (e.g., specification of new data sets, exchange of specific atomic modules) for other purposes that require analysis of various cells, nuclei or other studies. The live demonstration of the LONI pipeline demonstrated the simplicity of use and high efficiency of parallel data processing.

[0071] 3D imaging data were produced for 2 cell lines that have been publicly shared, to promote the reproducibility of the results, promote open science development and realize collaborative verification. This 3D image data set is one of the largest publicly available data sets of this type (including 3 channel raw data with approximately 1500 nuclear markers and approximately 2700 nucleolar markers). The classification results about these data have compared epithelial cells to mesenchymal human prostate cancer cell lines and serum starvation to the fibroblast cell lines of proliferation, proving that the accuracy of using 3D morphometrics to carry out cell type prediction is high, especially when applied to cell groups. Although different classification algorithms seem to be optimal for different experiments, it is observed that the morphometrics of core and nucleolus are both important features for distinguishing treatment conditions or cell phenotypes. In the case of fibroblast classification, the results show the importance of nuclear morphometrics, the number of nucleoli in each core and various internal nucleolar morphometrics. For PC3 cells, the most important classification feature is the moment of distribution of various nucleolar morphometrics and nuclear size and shape. Interestingly, for these two cell lines, there are three common morphometric features in the first 10 most important features. This confirms previously reported observations and demonstrates that the approach extracts relevant information from cell form to successfully classify cells using a combination of criteria.

[0072] The proposed method is scalable and can handle a variety of complex large 3D imaging data, and is not limited to nuclear shape and nucleolar shape. With some changes, it can be applied to other cells and nuclear components of interest. As long as its topology is spherical, the robust smooth surface reconstruction can be directly applied to any 3D shape. Together with molecular level techniques such as Hi-C, this 3D shape morphometric workflow can form a powerful combination for studying DNA structure in the spatial and temporal framework of the 4D nuclear group. An example of a possible future application of this workflow is the study of asymmetric cell division. Stem cells and progenitor cells are characterized by their ability to self-renew and produce differentiated offspring. The balance between these processes is achieved by controlled asymmetric division, and it is necessary to generate cell diversity during development and maintain the dynamic balance of adult tissues. The disruption of this balance may lead to premature exhaustion or abnormal growth of the stem cell / progenitor cell pool. In many tissues, dysregulated asymmetric division is associated with cancer. It is unknown whether there is a fortuitous relationship between asymmetric cell division defects and cancer initiation. It is foreseeable that the shape analysis pipeline will help study the 4D nuclear group topology of morphogenesis and cancer initiation.

[0073] The ability to automate the process of specimen collection, image acquisition, data preprocessing, calculation of derived biomarkers, modeling, classification, and analysis can significantly impact clinical decision making and basic research on cell deformation. This appears to be the first attempt to combine 3D cell nucleus shape modeling with robust smooth surface reconstruction and extraction of shape morphometric metrics into a highly parallel pipeline workflow protocol for end-to-end morphometric analysis of thousands of nuclei and nucleoli in 3D. This approach allows for efficient and informative assessment of cell shape from imaging data and represents a reproducible technique that can be validated, modified, and reused by the biomedical community. This promotes result reproducibility, collaborative method validation, and widespread knowledge dissemination.

[0074] Part of the technology described herein can be implemented by one or more computer programs executed by one or more processors. The computer program includes processor executable instructions stored on a non-transitory tangible computer readable medium. The computer program may also include stored data. Non-limiting examples of non-transitory tangible computer readable media are non-volatile memory, magnetic storage devices, and optical storage devices.

[0075] Some parts of the above description present the technology described herein in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are means for those skilled in the art of data processing to most effectively convey the essence of the work of the technician to other technicians in the field. Although these operations are described functionally or logically, they should be understood to be implemented by computer programs. In addition, it has been shown that, without loss of generality, it is sometimes convenient to refer to these arrangements of operations as modules or to name them with functions.

[0076] Unless otherwise specifically noted as is apparent from the above discussion, it should be understood that throughout the specification, discussions using terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within a computer system memory or register or other such information storage, transmission or display device.

[0077] Certain aspects of the described techniques include processing steps and instructions described herein in algorithmic form. It should be noted that the described processing steps and instructions can be embodied in software, firmware, or hardware, and when embodied in software, can be downloaded to reside on and be operated by different platforms used by real-time network operating systems.

[0078] The present disclosure also relates to a device for performing the operations described herein. The device may be specifically constructed for the desired purpose, or the device may include a computer selectively activated or reconfigured by a computer program stored on a computer-readable medium, which may be accessed by a computer. Such a computer program may be stored in a tangible computer-readable storage medium, such as, but not limited to, any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card or an optical card, an application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions and each coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture that uses multiple processors to design to increase computing power.

[0079] The algorithms and operations proposed herein are not inherently related to any particular computer or other device. Various systems can also be used together with programs according to the teachings herein, or they can prove to be convenient to build more specialized devices to perform the required method steps. The structures required for various these systems and equivalent variations will be obvious to those skilled in the art. In addition, the present disclosure is not described with reference to any particular programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the present disclosure as described herein.

[0080] For the purpose of illustration and description, the foregoing description of the embodiments has been provided. This is not intended to be exhaustive or limit the present disclosure. Each element or feature of a particular embodiment is generally not limited to the particular embodiment, but is interchangeable where applicable, and can also be used in the selected embodiment even if not specifically shown or described. Each element or feature of a particular embodiment can also be changed in many ways. Such a modification is not considered to be out of the present disclosure, and all such modifications are intended to be included in the scope of the present disclosure.

Claims

1. An automated method for analyzing biological tissues and cells, comprising: staining a composition of a biological sample, wherein the biological sample comprises a tissue or a cell culture comprising at least one cell; receiving image data of the biological sample, wherein the image data provides a three-dimensional representation of the biological sample; marking a composition of the at least one cell in the image data; For each labeled tissue biomarker or cell or subcellular organelle or component in the image data, construct a mathematical representation of the boundaries defining a given cell nucleus, including by chromosome domain, topologically associated domain (TAD), lamina associated domain (LAD), or telomere boundaries; extracting features for each of the labeled tissue biomarkers or cells or subcellular organelles or components using the mathematical representation of the tissue biomarkers or cells or subcellular organelles or components, wherein the features are measures of the shape and size of the labeled tissue biomarkers or cells or subcellular organelles or components; storing two or more models for cell classification; and The biological sample is classified by comparing the extracted characteristics of the labeled tissue biomarkers or cells or subcellular organelles or components with the stored models. 2 . The method of claim 1 , further comprising generating the image data using three-dimensional image acquisition technology.

3. The method according to claim 1, wherein: Labeling components also includes segmenting the image data into volumes representing the components.

4. The method according to claim 1, wherein: Constructing the mathematical representation also includes the use of iterative Laplace-Beltrami eigenprojections and boundary deformations.

5. The method according to claim 1, wherein: The features extracted from the labeled tissue biomarkers or cells or subcellular organelles or components include determining one or more of the following: the volume of the labeled tissue biomarkers or cells or subcellular organelles or components, the surface area of ​​the labeled tissue biomarkers or cells or subcellular organelles or components, the average curvature of the labeled tissue biomarkers or cells or subcellular organelles or components, the shape index of the labeled tissue biomarkers or cells or subcellular organelles or components, the curvature index of the labeled tissue biomarkers or cells or subcellular organelles or components, and the fractal dimension of the labeled tissue biomarkers or cells or subcellular organelles or components.

6. The method of claim 1, further comprising classifying the biological sample using a random forest classification method.

7. The method of claim 1, further comprising classifying the biological sample using a classification method selected from the group consisting of: a linear classifier, a k-nearest neighbor method, a decision tree method, a neural network, and a support vector machine.

8. The method of claim 1, further comprising classifying the biological sample only if the biological sample contains more than three cells.

9. The method according to claim 1, wherein: The components of the biological sample are stained using markers for tissue biomarkers or cells or subcellular organelles or components.

10. The method according to claim 9, wherein: The markers are fibrin antibodies, ethidium bromide and 4',6-diamidino-2-phenylindole.

11. The method according to claim 1, wherein: The biological sample is from cell culture or cell tissue.

12. An automated method for analyzing biological cells, comprising: staining a composition of a biological sample, wherein the biological sample includes at least one cell; receiving image data of the biological sample, wherein the image data provides a three-dimensional representation of the biological sample; marking a composition of the at least one cell in the image data; For each labeled cell nucleus in the image data, constructing a mathematical representation of the boundaries defining the given cell nucleus; extracting features for each of the labeled nuclei using the mathematical representation of the nuclei, wherein the features are measures of the shape and size of the labeled nuclei; For each labeled nucleolus, construct a mathematical representation of the boundaries defining the given nucleolus; extracting features from each of the labeled nucleoli using the mathematical representation of the nucleolus, wherein the features are measures of the shape and size of the labeled nucleoli; storing two or more models for cell classification; and The biological sample is classified by comparing the extracted features of the labeled cell nuclei and labeled cell nucleoli with the stored model.

13. The method of claim 12, further comprising generating the image data using a confocal microscope.

14. The method according to claim 12, wherein: Labeling components also includes segmenting the image data into volumes representing the components.

15. The method according to claim 12, wherein: Constructing the mathematical representation also includes the use of iterative Laplace-Beltrami eigenprojections and boundary deformations.

16. The method according to claim 12, wherein: The features extracted from the labeled cell nuclei include determining one or more of: the volume of the labeled cell nuclei, the surface area of ​​the labeled cell nuclei, the average curvature of the labeled cell nuclei, the shape index of the labeled cell nuclei, the curvature index of the labeled cell nuclei, and the fractal dimension of the labeled cell nuclei.

17. The method according to claim 16, wherein: The features extracted from the labeled cell nucleoli include determining one or more of the following: the count of nucleoli in the corresponding cell nucleus, the volume of the labeled cell nucleoli, the surface area of ​​the labeled cell nucleoli, the average curvature of the labeled cell nucleoli, the shape index of the labeled cell nucleoli, the curvature index of the labeled cell nucleoli, and the fractal dimension of the labeled cell nucleoli.

18. The method of claim 12, further comprising classifying the biological sample using a random forest classification method.

19. The method of claim 12, further comprising classifying the biological sample using a classification method selected from the group consisting of: a linear classifier, a k-nearest neighbor method, a decision tree method, a neural network, and a support vector machine.

20. The method of claim 12, further comprising classifying the biological sample only if the biological sample contains more than three cells.

21. The method according to claim 12, wherein: The components of the biological sample were stained using fibrin antibodies, ethidium bromide, and 4',6-diamidino-2-phenylindole.

22. The method according to claim 12, wherein: The biological sample is derived from a cell culture.

Images