Parametric modeling and estimation of diagnostically relevant histological patterns in digital tissue images

Abstract

A computational pathology method includes receiving multi-parameter cellular and subcellular imaging data of a tissue sample and locating and segmenting a plurality of tissue components of the tissue sample in the multi-parameter cellular and subcellular imaging data to generate segmented multi-parameter cellular and subcellular imaging data, applying a parametric feature modeling scheme to particular tissue components in the segmented multi-parameter cellular and subcellular imaging data, the parametric feature modeling scheme being generated from a dictionary of existing diagnostically relevant histological patterns and including several structural features suitable for defining several disease entities of the disease, applying the parametric feature modeling scheme including determining a quantification of each of the several structural features for the tissue sample, and classifying a disease state in the tissue sample based on the quantification of each of the several structural features.

Description

[Technical Field]

[0001] <Government Contract> This invention was made with government support under Grant #CA204826 awarded by the National Institutes of Health (NIH). The government has certain rights in this invention.

[0002] <Technical field> The disclosed concepts relate generally to digital pathology, and more particularly to a system and method for parametrically modeling a dictionary of diagnostically relevant histological patterns and using the modeling scheme to quantify the presence of patterns in digital pathology images to enable classification of disease states in the digital pathology images. [Background technology]

[0003] Advances in digital imaging technology and computing power have paved the way for major artificial intelligence (AI)-based changes in the pathology workflow. It is now possible to rapidly image clinical-volume microscope slides, and recent regulatory approvals have permitted the use of these digital pathology images in actual clinical diagnoses. Computational pathology represents a significant advancement in the form of novel machine learning (ML) tools that pathologists could use to significantly improve diagnostic performance, particularly in terms of accuracy and efficiency. Such tools could be applied throughout the pathology laboratory for other uses, such as symptom triage and automated, real-time quality assurance. Numerous studies are currently showing promise in their early stages, demonstrating the significant benefits pathologists and patients could receive from accessing powerful computational pathology tools.

[0004] Enthusiasm for AI in digital pathology is widespread. However, this is strongly tempered by caution and reasonable concerns about potential risks. Moreover, nearly all early attempts at computational pathology use convolutional neural networks, also known as deep learning. Deep learning is powerful, but it is opaque, like a "black box" that cannot be opened to peer inside and see what it is doing, exactly how it works, or even whether it is working as intended. Such systems provide answers but do not allow pathologists to ask "why?" Summary of the Invention

[0005] In one embodiment, a computational pathology method is provided that is broadly applicable to many histologies, including histology for high-level, organ-specific disease entities, including both tumor and non-tumor pathologies. The method includes receiving multi-parameter cellular and / or subcellular imaging data for an image of a tissue sample and locating and segmenting multiple tissue components of the tissue sample in the multi-parameter cellular and subcellular imaging data to generate segmented multi-parameter cellular and subcellular imaging data. The method also includes applying a parametric feature modeling scheme to specific tissue components in the segmented multi-parameter cellular and subcellular imaging data, the parametric feature modeling scheme being generated from a dictionary of existing diagnostically relevant histological patterns and including several structural features suitable for defining several disease entities of the disease, where applying the parametric feature modeling scheme includes determining a quantification of each of the several structural features for the tissue sample, and classifying a disease state in the tissue sample based on the quantification of each of the several structural features.

[0006] In another embodiment, a computerized computational pathology system is provided for identifying diagnostic tissue patterns in multi-parameter cellular and subcellular imaging data for several tissue samples from several patients or several multicellular in vitro models. Similar to the previously described methods, the system is broadly applicable to many histologies, including histology for high-level, organ-specific disease entities, including both tumor and non-tumor pathologies. The system includes a processing device including several components configured for: (i) locating and segmenting multiple tissue components of the tissue sample in the multi-parameter cellular and subcellular imaging data to generate segmented multi-parameter cellular and subcellular imaging data; (ii) applying a parametric feature modeling scheme to specific tissue components in the segmented multi-parameter cellular and subcellular imaging data, the parametric feature modeling scheme being generated from a dictionary of existing diagnostically relevant histological patterns and including several structural features suitable for defining several disease entities of the disease; applying the parametric feature modeling scheme includes determining a quantification of each of the several structural features for the tissue sample; and (iii) classifying a disease state in the tissue sample based on the quantification of each of the several structural features. [Brief explanation of the drawings]

[0007] A complete understanding of the present invention can be obtained from the following description of the preferred embodiment when read in conjunction with the accompanying drawings.

[0008] [Figure 1] FIG. 1 is a schematic diagram of an exemplary digital pathology system for classifying tissue components in a tissue sample, according to an exemplary embodiment of the disclosed concepts. [Figure 2] FIG. 2 is a flowchart illustrating a method for generating a parametric feature modeling scheme according to an exemplary embodiment of the disclosed concepts. [Figure 3]FIG. 3 is a flow chart illustrating a method for classifying tissue components in a tissue sample based on imaging data of the tissue sample and application of a parametric feature modeling scheme, according to an exemplary embodiment of the disclosed concepts. [Figure 4] FIG. 4 is a schematic diagram illustrating a parametric model of a histological pattern in the form of a single feature, according to an exemplary embodiment of the disclosed concepts. [Figure 5] FIG. 5 is a schematic diagram illustrating a parametric model of a histological pattern in the form of a dual feature, according to an exemplary embodiment of the disclosed concepts. [Figure 6-1] FIG. 6-1 is a schematic diagram illustrating a parametric model of a histological pattern in the form of a ternary feature, according to an exemplary embodiment of the disclosed concepts. [Figure 6-2] FIG. 6-2 is a schematic diagram illustrating a parametric model of a histological pattern in the form of a ternary feature, according to an exemplary embodiment of the disclosed concepts. [Figure 6-3] FIG. 6-3 is a schematic diagram illustrating a parametric model of a histological pattern in the form of a ternary feature, according to an exemplary embodiment of the disclosed concepts. [Figure 7-1] FIG. 7-1 is a table providing a list of all model parameters derived in connection with a particular exemplary embodiment of the disclosed concepts. [Figure 7-2] FIG. 7-2 is a table providing a list of all model parameters derived in connection with a particular exemplary embodiment of the disclosed concepts. [Figure 8] FIG. 8 is a schematic diagram illustrating how likelihood scores are calculated to reveal dominant histological patterns according to an exemplary embodiment of the disclosed concepts. [Figure 9] FIG. 9 is a schematic diagram illustrating the dominant histological patterns in representative images of low-risk and high-risk lesions, according to an exemplary embodiment of the disclosed concepts. DETAILED DESCRIPTION OF THE INVENTION

[0009] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise.

[0010] As used herein, a description of two or more parts or components being "coupled" means that the parts are connected or operate together, either directly or indirectly, i.e., through one or more intermediate parts or components, to the extent that a connection occurs.

[0011] As used herein, the term "several" means one or an integer greater than one (ie, a plurality).

[0012] As used herein, the terms "component" and "system" are intended to refer to a computer-related entity, whether hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. For example, both an application running on a server and the server may be considered a component. One or more components may reside within a process and / or thread of execution, and components may be localized on one computer and / or distributed among two or more computers. While some methods of displaying information to a user have been shown and described with specific diagrams or graphs as screenshots, those skilled in the relevant art will recognize that various alternatives may be employed.

[0013] As used herein, the term "multiparametric cellular and subcellular imaging data" refers to data obtained by generating several images from a tissue section, providing information about multiple measurable parameters at the cellular and / or subcellular level in the tissue section. Multiparametric cellular and subcellular imaging data can be generated by several different imaging modalities, including, but not limited to, transmitted light (e.g., a combination of H&E and / or IHC (one or more biomarkers)); fluorescence; immunofluorescence (including, but not limited to, antibodies and nanobodies, including, but not limited to, multiplex (4-7) biomarkers and hyperplex biomarkers (more than seven biomarkers)); multiplex and / or hyperplex live-cell biomarkers; electron microscopy; toponome imaging; matrix-assisted laser desorption / ionization mass spectrometry imaging (MALDI MSI); complementary spatial imaging (e.g., FISH, MxFISH, FISHSEQ, CyTOF), multiparametric ion beam imaging, and in vitro imaging. Targets include, but are not limited to, tissue samples (human or animal) and in vitro models (human or animal) of tissues and organs.

[0014] Directional terms used herein, such as top, bottom, left, right, upper, lower, front, rear, and derivatives thereof, refer to the orientation of the elements illustrated in the drawings and do not limit the scope of the claims, unless expressly stated.

[0015] The disclosed concepts will be described below, for purposes of explanation, with reference to numerous specific details in order to provide a thorough understanding of the present innovation. However, it will be apparent that the disclosed concepts can be practiced without these specific details without departing from the spirit and scope of the present invention.

[0016] The disclosed concepts provide a novel technological approach to computational pathology using explainable AI (xAI). The xAI approach of the disclosed concepts enables a completely different relationship between pathologists and software tools: transparency and accountability of ML tools. The goal is to foster pathologists' trust and acceptance of novel, powerful computational tools. If AI tools are to truly assist pathologists in their work with complex and difficult decisions, they must be able to provide evidence and data for their conclusions. With sufficient situational awareness, pathologists can make the best diagnostic decisions (e.g., benign vs. malignant, high-risk vs. low-risk, etc.) that only they can make.

[0017] Building on our previous work in xAI tools, the disclosed concepts provide a framework for parametrically modeling a dictionary of diagnostically relevant histological patterns and quantifying their presence in digital pathology images. This framework allows pathologists to quantitatively visualize clinically relevant histological structures given a parametric model. In a non-limiting exemplary embodiment, the disclosed concepts access a latent dictionary of various histological patterns from several different public sources, including the World Health Organization (WHO) tumor classification and consultations with expert teams of pathologists from various subspecialties. These sources help assemble a comprehensive framework of high-level and organ-specific disease entities, e.g., breast, lung, gastrointestinal tract, skin, and genitourinary tract, including both tumor and non-tumor pathologies. These include, for example, inflammatory infiltrate patterns (crypt abscesses in active colitis or the presence of aberrant plasmacytoid dendritic cells in the dermis), apoptosis in gastrointestinal biopsies of transplant patients (GVHD takes time), subtle non-tumor patterns in lung pathology, examining the biological environment and making special stains available, finding small tumor metastases in almost all solid tumors, and discovering incidental lymphomas in lymph nodes taken for solid tumor staging (CLL / SLL are the most common).

[0018] Many of these disease entities share characteristic patterns (e.g., gland formation in adenocarcinoma, duct formation in breast tissue), while others are disease-specific, such as colonic crypt distortion, which is diagnostic of chronic mucosal ulcerative colitis. Utilizing existing disease classifications, the disclosed conceptual approach may be generalized to multiple organ systems, thereby providing explainable, justifiable, and appropriate disease-specific diagnostic recommendations to pathologists using xAI techniques.

[0019] These guidelines will analytically model the visual pattern dictionary that traditionally defines standards for tumor classification / nomenclature among pathologists worldwide. The disclosed conceptual model, built in an exemplary embodiment on WHO guidelines, when integrated with a transparent and interpretable ML interface, provides a solution to opacity, which in turn facilitates a better understanding of computational tools and organizational mechanisms.

[0020] FIG. 1 is a schematic diagram of an exemplary digital pathology system 5, constructed and configured to classify tissue elements in a tissue sample (e.g., benign or malignant, high-risk or low-risk, etc.) based on a parametric feature modeling scheme in accordance with an exemplary embodiment of the concepts disclosed herein. As seen in FIG. 1, system 5 is a computing device constructed and configured to generate and / or receive multi-parameter cellular and subcellular imaging data (25 in FIG. 1) and process the data as described below to classify a particular disease state in the tissue sample through application of a parametric feature modeling scheme to the image data. System 5 may comprise, for example, but not limited to, a PC, laptop computer, tablet computer, or other suitable computing device constructed and configured to perform the functions described herein.

[0021] The system 5 includes an input device 10 (e.g., a keyboard), a display 15 (e.g., an LCD), and a processing unit 20. A user can use the input device 10 to provide input to the processing unit 20, which provides output signals to the display 15, enabling the display 15 to display information to the user (e.g., a segmented tissue image and a current pathology classification of specific tissue components in the tissue image), as described in detail herein. The processing unit 20 includes a processor and memory. The processor may be, for example, but not limited to, a microprocessor (μP), a microcontroller, an application-specific integrated circuit (ASIC), or other suitable processing device, interfaced with the memory. The memory may be one or more of various types of internal and / or external storage media, such as RAM, ROM, EPROM, EEPROM, FLASH®, or other computer-readable media providing storage registers, for data storage, such as an internal storage area of ​​a computer, and may be volatile or non-volatile memory. The memory stores several routines executable by the processor, including routines for implementing the disclosed concepts in various embodiments, as described herein. In particular, the processing device 20 includes a histological structure segmentation component 30 configured to identify and segment histological structures (such as, but not limited to, ducts / glands and lumens, clusters of ducts / glands, and individual nuclei) in several tissue images (e.g., H&E stained image data) represented by multi-parameter cellular and / or subcellular imaging data 25 obtained from various imaging modalities, in various embodiments as described herein.In a non-limiting exemplary embodiment, the histological structure segmentation component 30 employs the segmentation approach described in U.S. Provisional Patent Application No. 62 / 990,264, filed March 16, 2020, entitled "Scalable and High Precision Context Guided Segmentation of Histological Structures," the disclosure of which is incorporated herein by reference. The segmentation approach can locate and segment histological components such as, but not limited to, ducts, nuclei, blood vessels, alveoli, and colonic glands.

[0022] The processing device 20 further includes a dictionary 35 of existing diagnostically relevant visual histological patterns, which traditionally define criteria for classifying specific disease states, such as breast cancer. In a non-limiting exemplary embodiment, the dictionary of existing diagnostically relevant histological patterns is obtained at least in part from the World Health Organization (WHO) Blue Book (including the images contained therein). As is well known, the WHO Blue Book is an essential international reference source for pathologists, clinicians, and researchers. The WHO Blue Book articulates a body of knowledge regarding how histological patterns for the differential diagnosis of diseases, such as tumors, can be structurally described in terms of (1) cell morphology (e.g., round, large, mitotic, etc.), (2) spatial cellular organization (e.g., picket-fence, cribriform, etc.), and (3) architectural tissue organization (e.g., tumor-infiltrated lymph nodes, fat at the tumor border, etc.). All of these histological patterns can be visually evaluated by a pathologist. Pathologists typically arrive at a diagnosis by correlating the patterns in tissue samples with those in WHO criteria.

[0023] Additionally, the processing unit 20 further includes several parametric feature models that define a parametric feature modeling scheme derived from a dictionary 35 of existing diagnostically relevant histological patterns. A specific method for generating several parametric feature models 40 according to certain exemplary embodiments is described below in conjunction with FIG. 2 . The parametric feature modeling scheme of the disclosed concepts models histological patterns from the dictionary 35 of existing diagnostically relevant histological patterns and defines several quantitative structural features. These features may then be used to define several disease entities for several specific diseases. As known in the art, a disease entity is a collection of features that defines a classification of a disease, such as cancer. In this manner, the parametric feature modeling scheme analytically models the visual pattern dictionary that traditionally defines the basis for disease classification.

[0024] As described elsewhere herein, in exemplary embodiments, the quantifiable features of the parametric feature modeling scheme may include one or more unary, binary, and / or ternary features, where each unary feature is a single morphological feature such as the size, shape, or spatial extent of a tissue component in the image, each binary feature is a pairwise combination of two binary features, and each ternary feature is a combination of three or more unary features. Specific examples of such unary, binary, and ternary features may be employed in connection with one or more specific exemplary embodiments of the disclosed concepts and are described in detail elsewhere herein (FIGS. 4-6).

[0025] In addition, the processing device 20 also includes a tissue classification component 45. As described in detail elsewhere herein, the tissue classification component 45 is configured to apply one or more of the parametric feature models 40 to the multi-perimeter cellular and / or subcellular imaging data 25 to classify a particular disease state in the tissue sample represented by the multi-perimeter cellular and / or subcellular imaging data 25.

[0026] Referring to FIG. 2 , a flowchart illustrating a method for generating several parametric feature models 40 in accordance with an exemplary embodiment of the disclosed concepts is provided. The method illustrated in FIG. 2 may be implemented in the form of one or more routines stored on and executable by the processing device 20. The method begins at step 50, where the processing device 20 receives a dictionary 35 of existing diagnostically relevant histological patterns. Next, at step 55, the method analyzes the dictionary 35 of existing diagnostically relevant histological patterns to identify several quantifiable structural features suitable for defining several disease entities of a particular disease. In an exemplary embodiment, the identification of the several quantifiable structural features is based on information obtained through consultation with several expert pathologists. Next, at step 60, a parametric feature modeling scheme including several parametric feature models 40 is generated. In step 60, in a non-limiting exemplary embodiment, the several parametric feature models 40 are based on the identified structural features and information obtained through consultation with several expert pathologists. The parametric feature modeling scheme embodied by several parametric feature models 40 is suitable for application to multi-perimeter cellular and / or subcellular imaging data 25 to classify specific disease states in the tissue samples represented thereby based on the quantification of each of the structural features.

[0027] Referring to FIG. 3 , a flowchart illustrating a method for classifying tissue components in a tissue sample represented by multiperimeter cellular and / or subcellular imaging data 25 is provided, in accordance with an exemplary embodiment of the disclosed concepts. The method illustrated in FIG. 3 may be implemented in the form of one or more routines stored in and executable by the processing device 20. In the illustrated embodiment, the one or more routines form part of the tissue classification component 45 illustrated in FIG. 1 . The method of FIG. 3 begins at step 65, where multiperimeter cellular and / or subcellular imaging data 25 is received for at least an image of a tissue sample of interest. Next, at step 70, the histological structure segmentation component 30 is used to locate and segment multiple tissue components of the tissue sample in the received multiperimeter cellular and / or subcellular imaging data 25. Next, at step 75, a parametric modeling scheme including several parametric feature models 40 is applied to some of the tissue components represented by the multiperimeter cellular and / or subcellular imaging data 25 to determine quantifications of each of the structural features of the parametric modeling scheme in the tissue sample. Step 75 also includes classifying specific disease states in the tissue samples based on those determined quantifications.

[0028] 1-3, the disclosed concepts provide systems and methods for parametric feature modeling of existing dictionaries of histological patterns (obtained from digitized tissue samples) that define specific criteria for disease classification, using a parametric feature modeling scheme to analyze digital pathology images, quantify the presence of specific features in the image (and thus the presence of one or more associated histological patterns), and classify the image's disease state based on the feature values. As a result, the disclosed concepts mimic the actions taken by clinical pathologists when arriving at a diagnosis using accepted visual histological patterns as criteria for defining disease classification.

[0029] For illustrative purposes, the disclosed concepts will now be described with respect to a specific exemplary embodiment that is an approach for analyzing and classifying breast lesions. More specifically, this specific exemplary embodiment employs a specific parametric feature modeling scheme, described in detail below, to enable automatic classification of tumors in associated breast lesion images. In this specific exemplary embodiment, step 70 of FIG. 3 (i.e., the tissue component localization and segmentation step) is implemented using the duct and nucleus segmentation approach described in the above-referenced provisional patent application, which is incorporated herein by reference. However, it will be understood that this is merely exemplary, and the disclosed concepts may be used to analyze and classify other disease conditions in tissue sample images.

[0030] A particular embodiment for analyzing and classifying breast lesions described herein implements a parametric feature model 40 for the histological pattern within each segmented duct using a mixture of unidimensional, binary, and ternary features, as shown in Figures 4-6. More specifically, because cells and tissue components can combine multiple features, in this particular embodiment, the disclosed concepts define three types of decompositions. The hatched bar over each feature in Figures 4-6 indicates the lesion in which that feature is most likely to be found. For example, large, round nuclei are often found in high-risk lesions, small, oval nuclei are often found in low-risk lesions, and cribriform patterns tend to be restricted to atypical ductal hyperplasia (ADH). Figures 7-1 and 7-2 are tables listing all model parameters derived in connection with this particular embodiment.

[0031] The unitary features of this particular embodiment are shown in Figure 4. As seen in Figure 4, and as will be discussed below, such unitary features include a spectrum of morphological features based on the size, shape, and spatial extent around each nucleus.

[0032] The first group of unitary features in this embodiment ("small" and "large" features) is based on nuclei size (quantified using area), which is known to be a diagnostic clue in pathological grading, with small and large nuclei tending to belong to low-risk and high-risk lesions, respectively, as shown in Figure 4. To construct an analytical model of large and small features, the disclosed concept first constructs a histogram of nuclei area obtained from a set of regions of interest (ROIs) representing typical example regions within ducts containing small and large nuclei (Figure 4), and then models this histogram with a gamma distribution.

[0033] The second group of unitary features of this embodiment ("Circularity" and "Ovality" features) are based on nuclear shapes that have been identified as diagnostically meaningful. For example, as shown in FIG. 4, columnar cell change (CCC) lesions are known to exhibit predominantly oval nuclei. Thus, in aspects of this particular embodiment, the disclosed concepts define these features as (4π × area) / perimeter. 2 The circularity feature is quantified by a gamma distribution, which is measured as the circularity index, and the ellipticity feature is given by the ratio of the minor axis length to the major axis length. Circularity ranges from 0 (irregular, star-like appearance) to 1 (perfect circle), and ellipticity characterizes the "flatness" of an object, with lower values ​​indicating more elliptical nuclei (Figure 4). Due to the inherent heterogeneity of these measurements in each case, the disclosed concept models the distribution of circularity with a gamma distribution and ellipticity with a binary mixture of Gaussian (MoG) model, taking into account the spatial neighborhood around each nucleus (Figure 4).

[0034] Furthermore, several studies have shown that examining the spatial organization of nuclei can provide insight into cellular abnormalities that may ultimately lead to malignancy. For example, nuclear arrangement in CCC lesions frequently exhibits crowding and / or overlapping. However, cases belonging to high-risk atypical lesions (Flat Epithelial Atypia (FEA) and ADH) tend to have uniform and evenly spaced nuclei. Therefore, the third group of unitary features (crowdness and spacedness features) in this embodiment are based on the spatial organization of nuclei in an image. To quantify the crowding of each nucleus, the average distance to the 10 nearest nuclei is calculated. Next, an analytical model of crowding is constructed by considering local ROIs within a tract where clusters of nuclei exhibit significant crowding behavior and calculating their spatial density. On the other hand, to capture an evenly spaced / uniformly distributed pattern around a nucleus, the disclosed concept begins by placing a regular 3x3 grid around a reference nucleus and then measures the density of 20 neighboring nuclei by counting the number of nuclei in each grid cell as described in Sergio Rey, Wei Kang, Hu Shao, Levi John Wolf, Mridul Seth, James Gaboardi, and Dani Arribas-Bel, “pointpats: Point Pattern Analysis in PySAL,” PySAL: The Python Spatial Analysis Library, July 2019. The disclosed concept then compares this observed population to the number of nuclei expected under the assumption of complete spatial randomness, which is calculated as χ 2 Use the test statistic, and χ 2 A distribution table is used to obtain the corresponding p-value to assert the occurrence of points (here, nuclei) in the grid in a random manner similar to a Poisson point process. The larger the p-value, the more likely it is to observe a uniform / equally spaced distribution of nuclei around a reference nucleus.

[0035] While the above single features exhibit some degree of inferential strength (shown by the hatched bars above each feature in FIG. 4 ), pathologists typically make informed decisions by paying attention to pairwise combinations of those features. For example, CCC lesions (low risk) exhibit a densely packed, oval nuclear arrangement, while high-risk lesions tend to have a higher likelihood of large-round, evenly spaced-large, and evenly spaced-round nuclei. Furthermore, lesions exhibiting predominant areas of small nuclei coupled with densely packed and / or evenly spaced behavior are typical of normal ducts. Thus, this embodiment of the disclosed concepts considers seven such binary features obtained from pairwise combinations of single features, as shown in FIG. 5 . 5, the binary features of this exemplary embodiment include: "Nuclear Large-Circularity" feature, "Nuclear Small-Ovality" feature, "Nuclear Evenness-Large" feature, "Nuclear Compactness-Small" feature, "Nuclear Evenness-Small" feature, "Nuclear Compactness-Ovality" feature, and "Nuclear Evenness-Circularity" feature. In an exemplary implementation, to generate the binary features, the disclosed concepts take a z-score for each single feature and model the joint distribution of z-scores from the feature pairs with a binary two-dimensional mixture of Gaussian distributions.

[0036] Furthermore, some diagnostically relevant histological patterns are best represented by a combination of three or more unitary features. Thus, this embodiment of the disclosed concepts contemplates three such ternary features, as shown in Figures 6-1-3, resulting from a combination of three or more unitary features (including, but not limited to, the specific unitary features listed above). As seen in Figures 6-1-3, the ternary features of this embodiment include the "nuclear largeness-circularity-uniformity" feature, the "cribriform" feature, and the "picket fence" feature.

[0037] In particular, to determine the large-circularity-equidistance feature, the disclosed concept takes the z-scores from each unit feature, i.e., size, circularity, and equiidistance, and constructs a three-component 3D mixture of Gaussian models using ground truth examples.

[0038] Regarding cribriform features, this pattern is characterized by the polarization of epithelial cells within a space formed by multiple, "nearly" circular lumens (>2), each 5-6 cells wide and resembling the "holes in Swiss cheese" in appearance. This complex structural pattern can be identified by analytically modeling three (single-dimensional) subfeatures: the clustering coefficient, the distance from the two nearest lumens to the nucleus, and the circularity of the lumen adjacent to the nucleus. The polarization of epithelial cells around the lumens is characterized by the clustering coefficient, calculated according to the method described by Naiyun Zhou, Andrey Fedorov, Fiona Fennessy, Ron Kikinis, and Yi Gao, "Large Scale Digital Prostate Pathology Image Analysis Combining Feature Extraction and Deep Neural Network," arXiv preprint arXiv:1705.02678, 2017, and shown in Figure 6-2. Furthermore, groups of nuclei occupying the space between two lumens tend to exhibit a cribriform pattern around them. Therefore, the disclosed concepts measure the average distance between each nucleus and its two nearest lumens and model its distribution using a gamma function (see Figure 6-2). The final likelihood of a cribriform pattern is obtained from a weighted sum of the likelihood scores of the sub-features. In this aspect, the disclosed concepts perform a grid search for blending coefficients and learned that, in the exemplary embodiment, the likelihood scores from the three sub-features should be blended in proportions of 0.2, 0.5, and 0.3, respectively.

[0039] For the picket fence feature, this pattern is identified from a group of closely packed oval nuclei oriented perpendicular to the basilar membrane (lumen). An analytical model of this high-order visual feature is obtained by constructing a parametric model of four simple (monodimensional) subfeatures: the distance of the nucleus to the nearest lumen, the nuclear ellipticity, the angular extent of the long axes of 10 nearby nuclei, and their local angle with respect to the basilar membrane, as shown in Figure 6-3. Because each subfeature contributes equally to the observation of this ternary feature, in an exemplary embodiment, the disclosed concepts assign a blending coefficient of 0.25 when combining likelihood scores from the four subfeatures to determine the presence of a picket fence pattern.

[0040] As described above, the parametric model of a histological pattern is, in an exemplary embodiment, a probability distribution. For example, a cytological feature, such as the nuclear ovality of a particular nucleus within an ROI, accepts probability under the mixture of Gaussian models shown in Figure 4. However, this feature can be made diagnostically more powerful by considering its spatial context, i.e., elliptical nuclei are typically surrounded by other elliptical nuclei. In one aspect, the disclosed concept evaluates this by first constructing a distribution (hyperparameters optimized by trial and error) of feature values ​​for all nuclei found within a 100 μm radius of a reference nucleus and comparing it to a canonical distribution obtained from the ground truth examples used to derive the parametric model. The disclosed concept optionally uses two distance measures: the Kullback-Leibler divergence for mixtures of Gaussian distributions and the two-sample Kolmogorov-Smirnov test for the unimodal gamma distribution derived as described above. A shorter distance implies greater evidence for the pattern. The disclosed concept converts the distance into a likelihood score using an inverse S-function, as shown in Figure 8.

[0041] Regarding differential diagnosis strategies, in exemplary embodiments, the disclosed concepts employ a nonlinear strategy to find subregions within the ROIs using non-maximum suppression (a threshold of 0.85 on the likelihood score) where evidence for one or more of the unidimensional, binary, or ternary features is dominant, similar to how expert pathologists do so. Figure 9 visually illustrates likelihood maps of dominant patterns in representative images of low-risk and high-risk lesions. Low-risk lesions are characterized by round, small, evenly spaced, and evenly spaced-small islands in the normal ROI, while elliptical, round, evenly spaced-small, densely spaced-small, and picket-fence islands are dominant in the CCC ROI. High-risk lesions, on the other hand, are characterized by dominantly evenly spaced-large and evenly spaced-round regions in the FEA-labeled ROI, and by accentuated large cribriform patterns along with densely spaced traces in the ADH-labeled ROI. These patterns verify the normal morphology shown in Figures 4 through 6-1 through 6-3.

[0042] Furthermore, having identified regions of dominant unitary, binary, and ternary features, the disclosed concepts, in an exemplary embodiment, use three descriptive statistics: the median likelihood score of all nuclei found in each subregion, the median number of nuclei found in each subregion, and the number of subregions. This is calculated for each unitary, binary, and ternary feature (total = 16), resulting in a 48-column feature vector for an image. In an exemplary embodiment, feature vectors were calculated for all 1,441 labeled tube ROIs extracted from the entire slide image, resulting in a feature map of size 834 x 48 used to train the classifier and a 607 x 48 data matrix for testing. To analyze the benefits of including binary and ternary features, the disclosed concepts further slice the 48-column feature vector appropriately for three scenarios: unitary (U)-only features, unitary and binary (UB) features, and unitary, binary, and ternary features (UBT). Due to the inherent training and testing class imbalance that reflects the actual prevalence statistics of atypical lesions, the disclosed concept upsampled high-risk examples using the SMOTE technique described in Chawla N. et al., “Smote: Synthetic Minority Over-sampling Technique”, Journal of Artificial Intelligence Research, 16:321-357, 2002.

[0043] Additionally, before classifying lesions, the disclosed concept pays close attention to the presence of cribriform patterns, a visual primitive iconic for the ADH (high-risk) category. ROIs predicted to exhibit cribriform patterns are classified as high-risk if the number of nuclei forming the cribriform subregion is eight or more (hyperparameters optimized for the training data). The reduced dataset, lacking cribriform patterns, is tested for each scenario (U, UB, and UBT) using logistic regression (LR), support vector machine (SVM), random forest (RF), and gradient boosting classifier algorithms. The best model was selected by optimizing parameters using GridSearchCV based on precision, recall, and F-score, and then checked for overfitting by performing 10-fold stratified cross-validation.

[0044] The approach of this particular embodiment of the concepts described and disclosed herein can be easily illustrated because it has ∼150 parameters (see Figures 7-1 and 7-2), which cannot be provided by current deep learning (DL) methods, which typically require ∼10-50 million parameters and large training data sets. Furthermore, there are no widely reported DL methods for the analysis or classification of challenging pathologies such as atypical breast lesions, adenomatous polyps in the colon, or idiopathic pulmonary fibrosis in the lung. These algorithms for cancer versus non-cancer datasets are abundant.

[0045] While particular embodiments of the present invention have been described in detail, it will be understood by those skilled in the art that various modifications and alterations to those details may be developed in light of the overall teachings of the present disclosure. Accordingly, the particular configurations disclosed are illustrative only and do not limit the scope of the disclosed concepts, which are to be given the full scope of the appended claims and their equivalents.

Claims

1. Storing in a processing device several computerized parametric feature models, the several computerized parametric feature models analytically modeling a dictionary of existing diagnostically relevant visual histological patterns that define criteria for disease classification, the several computerized parametric feature models being based on information obtained through consultation with several expert pathologists, the several computerized parametric feature models defining several quantifiable structural features of the disease suitable for defining several disease entities that classify the disease state; receiving, at the processing device, multi-parameter cellular and / or subcellular imaging data relating to an image of a tissue sample; locating and segmenting, in the processing device, a plurality of tissue components of the tissue sample in the multi-parameter cellular and subcellular imaging data to generate segmented multi-parameter cellular and subcellular imaging data; determining, in the processing device, a quantification of each of the number of structural features for the tissue sample by applying the number of computerized parametric feature models to some of the plurality of tissue components in the segmented multi-parameter cellular and subcellular imaging data; classifying, in the processing device, the disease state in the tissue sample based on the quantification of each of the number of structural features; A computational pathology method comprising:

2. The method of claim 1 , wherein the number of structural features comprises a number of cell morphological features and a number of spatial cell organization features.

3. 3. The method of claim 2, wherein locating and segmenting the plurality of tissue components comprises locating and segmenting a plurality of nuclei in the tissue sample, some of the plurality of tissue components comprising the plurality of nuclei, and the several cytomorphological features include several size features each based on a nuclear size of each of the plurality of nuclei, several shape features each based on a nuclear shape of each of the plurality of nuclei, and several spatial extent features each based on a degree of nuclear spacing of each of the plurality of nuclei.

4. 4. The method of claim 3, wherein the number of size features comprises a small nucleus feature and a large nucleus feature, the number of shape features comprises a nuclear roundness feature and a nuclear ovality feature, and the number of spatial extent features comprises a nuclear compactness feature and a nuclear equidistant feature.

5. 5. The method of claim 4, wherein the small nuclei feature is based on a first histogram of nuclei areas obtained from prototypical regions with nuclei of a first size class that includes the small class, the first histogram being modeled with a gamma distribution, and the large nuclei feature is based on a second histogram of nuclei areas obtained from prototypical regions with nuclei of a second size class that includes the large class, the second histogram being modeled with a gamma distribution.

6. 5. The method of claim 4, wherein the nuclear circularity characteristic is based on several first measurements, each given by (4π × area) / perimeter2, and the nuclear ellipticity characteristic is based on several second measurements, each given by the ratio of the length of the minor axis to the length of the major axis.

7. 7. The method of claim 6, wherein the nuclear circularity feature ranges from 0 (indicating an irregular star-like appearance) to 1 (indicating a perfect circle), and the nuclear ovality feature characterizes the flatness of the nucleus, with lower values ​​indicating a highly oval nucleus.

8. 7. The method of claim 6, wherein the nuclear circularity feature considers a spatial neighborhood around each nucleus, where the distribution of circularity is modeled by a gamma distribution, and the nuclear ovality feature considers a spatial neighborhood around each nucleus, where the distribution of ovality is modeled by a binary mixture of Gaussian models (MoG).

9. The method of claim 4 , wherein the nuclear density feature is quantified by calculating, for each nucleus, the average distance to a number of nearest neighbor nuclei.

10. 5. The method of claim 4, wherein the nuclear isodencity feature is quantified by, for each nucleus, placing a grid cell centered on a reference nucleus and measuring the density of neighboring nuclei by counting the population of nuclei within the grid cell.

11. The method of claim 10, wherein the population is compared to the expected number of nuclei under the complete spatial randomness assumption.

12. 5. The method of claim 4, wherein the several spatial cellular organization features include a sieve-like feature indicating the degree to which some of the plurality of tissue components exhibit a sieve-like pattern, and a picket-fence feature indicating the degree to which some of the plurality of tissue components exhibit a picket-fence pattern.

13. 2. The method of claim 1, wherein the number of structural features includes some unitary features, some binary features consisting of a combination of two of the number of unitary features, and some ternary features consisting of a combination of three or more features selected from the number of unitary features or other structural features.

14. The method of claim 13 , wherein each binary feature comprises a joint distribution of z-scores from that unidimensional feature with a two-component two-dimensional mixture Gaussian distribution.

15. 13. The method of claim 12, wherein the number of structural features comprises some unitary features, some binary features, and some ternary features, wherein the number of unitary features comprises the nuclear small feature, the nuclear large feature, the nuclear circularity feature, the nuclear ovality feature, the nuclear compactness feature, and the nuclear equality feature, wherein the number of binary features comprises nuclear large-circularity feature, nuclear small-ovality feature, nuclear equality-large feature, nuclear compactness-small feature, nuclear equality-small feature, nuclear compactness-ovality feature, and nuclear equality-circularity feature, and wherein the number of ternary features comprises nuclear large-circularity-equality feature, the cribriform feature, and the picket fence feature.

16. A non-transitory computer-readable medium having stored thereon one or more programs comprising instructions that, when executed by a computer, cause the computer to perform the method of claim 1.

17. 1. A computational pathology system for identifying diagnostic tissue patterns in multi-parameter cellular and subcellular imaging data of several tissue samples from several patients or several multi-cellular in vitro models, comprising: a processing unit, the processing unit storing a number of computerized parametric feature models; the number of computerized parametric feature models analytically model a dictionary of existing diagnostically relevant visual histological patterns that define criteria for disease classification, the number of computerized parametric feature models being based on information obtained through consultation with several expert pathologists, and the number of computerized parametric feature models defining a number of quantifiable structural features suitable for defining a number of disease entities of the disease that classify the disease state; The processing device includes: locating and segmenting a plurality of tissue components of the tissue sample in the multi-parameter cellular and subcellular imaging data to generate segmented multi-parameter cellular and subcellular imaging data; applying the several computerized parametric feature models to several of the tissue components in the segmented multi-parameter cellular and subcellular imaging data to determine a quantification of each of the several structural features for the tissue sample; classifying the disease state in the tissue sample based on the quantification of each of the several structural features; A system including several components configured to:

18. 20. The system of claim 17, wherein the number of structural features comprises a number of cell morphological features and a number of spatial cell organization features.

19. 20. The system of claim 18, wherein locating and segmenting the plurality of tissue components comprises locating and segmenting a plurality of nuclei in the tissue sample, some of the plurality of tissue components comprising the plurality of nuclei, and the number of cytomorphological features comprises several size features each based on a nuclear size of each of the plurality of nuclei, several shape features each based on a nuclear shape of each of the plurality of nuclei, and several spatial extent features each based on a degree of nuclear spacing of each of the plurality of nuclei.

20. 20. The system of claim 19, wherein the number of size features comprises a small nucleus feature and a large nucleus feature, the number of shape features comprises a nuclear circularity feature and a nuclear ovality feature, and the number of spatial extent features comprises a nuclear compactness feature and a nuclear equidistant feature.

21. 21. The system of claim 20, wherein the small nuclei feature is based on a first histogram of nuclei areas obtained from prototypical regions with nuclei of a first size class that includes the small class, the first histogram being modeled with a gamma distribution, and the large nuclei feature is based on a second histogram of nuclei areas obtained from prototypical regions with nuclei of a second size class that includes the large class, the second histogram being modeled with a gamma distribution.

22. 21. The system of claim 20, wherein the nuclear circularity feature is based on several first measurements, each given by (4π × area) / perimeter2, and the nuclear ellipticity feature is based on several second measurements, each given by the ratio of the length of the minor axis to the length of the major axis.

23. 23. The system of claim 22, wherein the nuclear circularity feature ranges from 0 (indicating an irregular stellate appearance) to 1 (indicating a perfect circle), and the nuclear ovality feature characterizes the flatness of the nucleus with lower values ​​indicating a highly oval nucleus.

24. 23. The system of claim 22, wherein the nuclear circularity feature considers a spatial neighborhood around each nucleus, where the distribution of circularity is modeled by a gamma distribution, and the nuclear ovality feature considers a spatial neighborhood around each nucleus, where the distribution of ovality is modeled by a binary mixture of Gaussian models (MoG).

25. 21. The system of claim 20, wherein the nuclear density feature is quantified by calculating, for each nucleus, the average distance to a number of nearest neighbor nuclei.

26. 21. The system of claim 20, wherein the nuclear isodencity feature is quantified by measuring, for each nucleus, the density of neighboring nuclei by placing a grid cell centered on a reference nucleus and counting the population of nuclei within the grid cell.

27. 27. The system of claim 26, wherein the population is compared to the expected number of nuclei under the complete spatial randomness assumption.

28. 21. The system of claim 20, wherein the number of spatial cellular organization features comprises a sieve-like feature indicating the degree to which some of the plurality of tissue components exhibit a sieve-like pattern, and a picket-fence feature indicating the degree to which some of the plurality of tissue components exhibit a picket-fence pattern.

29. 20. The system of claim 17, wherein the number of structural features includes some unitary features, some binary features consisting of a combination of two of the number of unitary features, and some ternary features consisting of a combination of three or more features selected from the number of unitary features or other structural features.

30. 30. The system of claim 29, wherein each binary feature comprises a joint distribution of z-scores from that unidimensional feature with a two-component two-dimensional mixture Gaussian distribution.

31. 29. The system of claim 28, wherein the number of structural features include some unitary features, some binary features, and some ternary features, wherein the number of unitary features include the nuclear small feature, the nuclear large feature, the nuclear circularity feature, the nuclear ovality feature, the nuclear compactness feature, and the nuclear equality feature, wherein the number of binary features include nuclear large-circularity feature, nuclear small-ovality feature, nuclear equality-large feature, nuclear compactness-small feature, nuclear equality-small feature, nuclear compactness-ovality feature, and nuclear equality-roundness feature, and wherein the number of ternary features include nuclear large-circularity-equality feature, the cribriform feature, and the picket fence feature.

32. The method of claim 1, wherein the several disease entities include several organ-specific disease entities including both tumor and non-tumor pathologies.

33. The system described in claim 17, wherein the several disease entities include several organ-specific disease entities including both tumor and non-tumor pathologies.

Images