Image enhancement for improved nucleus detection and segmentation

Abstract

Various aspects of the present disclosure relate to systems and methods for enhancing brightfield or darkfield images to better detect nuclei. In some embodiments, the systems and methods described herein are used to identify membrane-staining biomarkers and nuclear / cytoplasmic-staining biomarkers in stained images of biological samples. In some embodiments, the systems and methods disclosed herein enable rapid and accurate nuclei detection in stained images of biological samples, particularly in raw stained images of biological samples in which nuclei appear blurred.

Description

[0001] Cross-references to Related Patent Applications

[0002] This patent application claims priority to and the benefit of U.S. patent application No. 62 / 745,730, filed on October 15, 2018, the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] Digital pathology involves scanning whole histopathology or cytopathology slides into digital images that can be interpreted on a computer screen. These images are then processed by imaging algorithms or interpreted by pathologists. To examine tissue sections (which are nearly transparent), they are prepared with colored histochemical stains that selectively bind to cellular components. Clinicians or computer-aided diagnosis (CAD) algorithms use the color-enhanced or stained cellular structures to identify morphological markers of disease and address them with appropriate therapies. Visual measurement enables a variety of processes, including diagnosing disease, assessing response to treatment, and developing new anti-disease drugs.

[0004] Immunohistochemistry (IHC) slide staining can be used to identify proteins in cells in tissue sections and is therefore widely used to study different cell types, such as cancer cells and immune cells in biological tissues. Therefore, IHC staining can be used to understand the distribution and localization of biomarkers differentially expressed by immune cells (such as T cells or B cells) in cancer tissues for immune response research. For example, tumors often contain infiltrates of immune cells, which may prevent tumor development or promote tumor growth.

[0005] In situ hybridization (ISH) can be used to determine whether there is a genetic malformation or, for example, a specific amplification of an oncogene in a malignant cell when observed under a microscope. In situ hybridization (ISH) uses a labeled DNA or RNA probe molecule that is antisense to the target gene sequence or transcript to detect or locate the targeted nucleic acid target gene in a cell or tissue sample. ISH is completed by exposing the cell or tissue sample fixed on a slide to a labeled nucleic acid probe, which can specifically hybridize with a given target gene in the cell or tissue sample. Multiple target genes can be analyzed simultaneously by exposing the cell or tissue sample to multiple nucleic acid probes, which have been labeled by multiple different nucleic acid tags. Utilizing labels with different emission wavelengths, simultaneous multicolor analysis can be performed on a single target cell or tissue sample in a single step. For example, the INFORM HER2 Dual ISH DNA probe mixture assay from Ventana Medical Systems, Inc. is intended to determine the status of the HER2 gene by calculating the ratio of the HER2 gene to chromosome 17. HER2 and chromosome 17 probes were detected using dual-color chromogenic ISH in formalin-fixed, paraffin-embedded human breast cancer tissue specimens. Summary of the Invention

[0006] Cell detection is an important task for quantitatively evaluating biomarker expression in histopathology images using IHC assays. Many nuclear detection and segmentation methods have been reported in the literature for both brightfield images (e.g., IHC and H&E) and darkfield images (e.g., immunofluorescence). Most methods focus on developing feature extraction or machine learning techniques for nuclear detection / segmentation in raw RGB images or single channel images representing nuclear staining (e.g., for brightfield IHC, hematoxylin channel images obtained by color unmixing algorithms; or DAPI channel images obtained directly by immunofluorescence multiplex imaging). In some instances, the quality of the nuclear stain in the image may be poor or even absent (see, e.g., Figure 6A and Figure 8A ), making nuclear detection challenging.

[0007] Various aspects of the present disclosure relate to systems and methods designed to enhance brightfield or darkfield images to better enable nuclear detection within stained images of biological samples. In some embodiments, the image processing systems and methods disclosed herein enable existing cell detection algorithms to be applied to a wider range of cell staining patterns, including those with poor quality nuclear stains. In some embodiments, an image preprocessing algorithm is applied to enhance structures defined by cell boundaries (e.g., natural boundaries defined by nuclear stains, or boundaries defined by separate membrane / cytoplasmic stains) so that all cells (regardless of their original morphology) appear as similar patterns (e.g., "speckles") in the enhanced image (i.e., the image generated after application of the image preprocessing algorithm). In some embodiments, "boundary structures" include natural boundaries defined by nuclear stains, or boundaries defined by separate membrane stains. In some embodiments, a Frangi filter is used to enhance boundary structures or continuous edges around cell nuclei. In some embodiments, the image enhanced by applying the Frangi filter can be further processed before nuclei are detected. For example, the enhanced image can be combined with at least a second image, such as one of the original input images, to generate a refined image, to which an automated nuclear detection algorithm can be applied. Automated cell detection algorithms can then be run against the enhanced images to provide computationally faster and more accurate detection results.Disclosed herein are examples of methods that are applied to both darkfield and brightfield images.

[0008] In one aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the method comprising: obtaining one or more input image channel images (e.g., image channel images obtained after unmixing a multiplexed brightfield image), wherein each of the obtained input image channel images contains a signal corresponding to one of a membrane stain (e.g., a lymphocyte biomarker membrane stain) or a nuclear stain (hematoxylin); enhancing the signal corresponding to the membrane stain or the nuclear stain in the first of the one or more obtained input channel images by applying a filter suitable for enhancing boundary structure (e.g., a Frangi filter) to the first of the one or more obtained input channel images to provide a first enhanced image; and detecting nuclei within a refined image derived from at least the first enhanced image (e.g., using an automated nucleus detection algorithm). In some embodiments, the membrane stain is 3,3'-diaminobenzidine (DAB) and the nuclear stain is hematoxylin. In some embodiments, the obtained input channel images are unmixed brightfield images. In some embodiments, the obtained input channel images are darkfield images.

[0009] In some embodiments, the refined image is generated by thresholding the first enhanced image. In some embodiments, the refined image is generated by: (i) combining the first enhanced image with at least the second image to provide a combined image; and (ii) thresholding the combined image. In some embodiments, the second image is a second enhanced image (i.e., a second obtained input channel image to which a Frangi filter has been applied). In some embodiments, the second enhanced image is derived from a second one of the one or more input channel images. In some embodiments, the second enhanced image is derived from a first one of the one or more input channel images, and wherein the first enhanced image and the second enhanced image are generated by applying the Frangi filter with different scaling factors (e.g., applying a scaling factor of 2 to one image and a scaling factor of 5 to the other image).

[0010] In some embodiments, the first and second of the one or more input channel images comprise a membrane stain. In some embodiments, the membrane stain is selected from the group consisting of a tumor membrane stain and a lymphocyte membrane stain. In some embodiments, one of the first and second of the one or more input channel images comprises a nuclear stain.

[0011] In some embodiments, the combined image is derived from at least a first enhanced image, a second image, and a third image. In some embodiments, at least one of the second image or the third image is enhanced by applying a Frangi filter. In some embodiments, the second image comprises a membrane stain, and wherein the third image comprises a nuclear stain. In some embodiments, the combined image is further derived from a fourth image.

[0012] In some embodiments, the refined image is a combined image obtained by combining an inverse of the first enhanced image with at least a second image. In some embodiments, one of the first enhanced image or the second image comprises a nuclear stain. In some embodiments, the at least second image is a second one of the one or more input channel images. In some embodiments, the second one of the one or more input channel images comprises a nuclear stain, and wherein the first one of the one or more input channel images comprises a membrane stain.

[0013] In another aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the method comprising: (a) obtaining one or more input image channel images, wherein each obtained input image channel image comprises a signal corresponding to one of a membrane stain or a nuclear stain; (b) enhancing the signal corresponding to the membrane stain or the nuclear stain in the first of the one or more obtained input channel images by applying a Frangi filter to the first of the one or more obtained input channel images to provide a first enhanced image; (c) generating a refined image based at least on the first enhanced image; and (d) automatically detecting nuclei within the generated refined image, wherein the nuclei are detected by applying an automated nuclear detection algorithm. In some embodiments, the detected nuclei are then superimposed on (i.e., made visible) the original entire slice image or any portion thereof. In some embodiments, the membrane stain is DAB and the nuclear stain is hematoxylin. In some embodiments, the nuclear stain is DAPI. In some embodiments, the obtained input channel images are unmixed brightfield images. In some embodiments, the obtained input channel images are darkfield images.

[0014] In some embodiments, the refined image is a segmentation mask image. In some embodiments, the segmentation mask image is derived from a combined image generated from at least the first enhanced image and the second image, thereby combining the at least the first enhanced image and the second image in an additive or weighted manner. In some embodiments, the combined image is derived from at least the first enhanced image and the second enhanced image, such as a combination of a membrane enhanced image and a nuclear stain enhanced image.

[0015] In some embodiments, the refined image is a combined image derived from at least a first enhanced image and a second image (e.g., the second enhanced image, another obtained input image, or a further processed variant thereof). In some embodiments, the first image and the second image are combined in an additive manner, i.e., they are summed. In some embodiments, the first enhanced image is further processed prior to generating the combined image. In some embodiments, the further processing of the first enhanced image includes generating an inverse image of the first enhanced image.

[0016] In another aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the method comprising: obtaining one or more input image channel images, wherein each obtained input image channel image contains a signal corresponding to one of a membrane stain or a nuclear stain; enhancing the signal corresponding to the membrane stain or the nuclear stain in the first of the one or more obtained input channel images by applying a Frangi filter to the first of the one or more obtained input channel images to provide a first enhanced image; generating a refined image based at least on the first enhanced image; and detecting nuclei within the refined image. In some embodiments, the refined image is generated by thresholding the first enhanced image. In some embodiments, the refined image is generated by combining the first enhanced image with at least a second image to provide a combined image. In some embodiments, the refined image is generated by: (i) combining the first enhanced image with at least a second image to provide a combined image; and (ii) thresholding the combined image. In some embodiments, the second image is a second of the one or more input channel images. In some embodiments, the second image is a second enhanced image. In some embodiments, any combined image can be derived from three or more images, including any combination of the original input image channel images or enhanced images.

[0017] In another aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the method comprising: (a) obtaining at least one input channel image comprising a signal corresponding to a nuclear stain; (b) enhancing the signal corresponding to the nuclear stain in the at least one acquired input channel image (comprising the signal corresponding to the nuclear stain) by applying a Frangi filter to the at least one acquired input channel image (comprising the signal corresponding to the nuclear stain) to provide a first enhanced image; (c) generating a refined image from the at least first enhanced image; and (d) automatically detecting nuclei within the generated refined image, wherein the nuclei are detected by applying an automated nucleus detection algorithm. In some embodiments, the method further comprises: obtaining at least one input channel image comprising a signal corresponding to a membrane stain; enhancing the membrane stain in the at least one acquired input channel image (comprising the signal corresponding to the membrane stain) by applying a Frangi filter to the at least one acquired input channel image (comprising the signal corresponding to the membrane stain) to provide a second enhanced image. In some embodiments, the at least first enhanced image and the second enhanced image are combined, such as in an additive or weighted manner, to provide the refined image. In some embodiments, the combined image may be used to detect the kernel; or alternatively, a threshold may be applied to the combined image, and the resulting segmentation mask image may be used to detect the kernel.

[0018] In another aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the method comprising: (a) obtaining at least one input channel image comprising a signal corresponding to a membrane stain; (b) enhancing the signal corresponding to the membrane stain in the at least one acquired input channel image (comprising the signal corresponding to the membrane stain) by applying a Frangi filter to the at least one acquired input channel image (comprising the signal corresponding to the membrane stain) to provide a first enhanced image; (c) generating a refined image from at least the first enhanced image; and (d) automatically detecting nuclei within the generated refined image, wherein the nuclei are detected by applying an automated nucleus detection algorithm. In some embodiments, the refined image is a combined image comprising: (i) a further processed version of the first enhanced image, and (ii) a second acquired input channel image comprising the signal corresponding to the nuclear stain. In some embodiments, the further processed version of the first enhanced image is an inverse image of the first enhanced image. In some embodiments, the other acquired input channel image comprises the signal corresponding to hematoxylin.

[0019] In another aspect of the present disclosure, a method for enhancing detection of cell nuclei within an image of a biological sample stained with hematoxylin and eosin and / or stained for the presence of multiple biomarkers is provided, the method comprising: obtaining one or more input image channel images, wherein each obtained input image channel image contains a signal corresponding to hematoxylin, DAPI, or a stain indicative of the presence or absence of a biomarker; enhancing boundary structures within at least a first one of the one or more image channel images by applying a Frangi filter to at least a first one of the one or more image channel images to provide at least a first Frangi-enhanced image channel image; and detecting nuclei within a refined image derived from at least one Frangi-enhanced image channel.

[0020] In some embodiments, the refined image is a segmentation mask image. In some embodiments, the segmentation mask image is generated by thresholding at least a first Frangi-enhanced image channel image. In some embodiments, the segmentation mask image is generated by thresholding a combined image derived from at least the first Frangi-enhanced image channel image. In some embodiments, the combined image is derived by combining: (i) at least the first Frangi-enhanced image channel image; and (ii) at least one of (a) a second of one or more input image channel images and / or (b) a second Frangi-enhanced image channel image. In some embodiments, the combined image comprises a combination of at least the first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image. In some embodiments, the first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image are derived from different input image channel images. In some embodiments, the first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image are derived from the same input image channel image. In some embodiments, the first and second Frangi-enhanced image channel images are derived by applying a Frangi filter at different scaling factors. In some embodiments, one of the first Frangi-enhanced image channel image or the second Frangi-enhanced image channel image comprises enhanced membrane boundaries; and the other of the first Frangi-enhanced image channel image or the second Frangi-enhanced image channel image comprises enhanced nuclear boundaries. In some embodiments, the combined image comprises a combination of at least the first Frangi-enhanced image channel image, the second Frangi-enhanced image channel image, and the third Frangi-enhanced image channel image, wherein at least one of the first, second, or third Frangi-enhanced image channel images is derived from a different input channel image.

[0021] In some embodiments, the refined image is a combined image obtained by combining: (i) an inverse image of the first Frangi-enhanced image channel image; and (ii) a second one of the one or more input channel images. In some embodiments, the combined image is obtained by combining: (i) at least the first Frangi-enhanced image channel image; and (ii) at least one of (a) the second one of the one or more input image channel images and / or (b) the second Frangi-enhanced image channel image.

[0022] In another aspect of the present disclosure, a system for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories configured to store computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: obtaining one or more input image channel images, wherein each obtained input image channel image comprises a signal corresponding to a membrane stain or a nuclear stain; enhancing the signal corresponding to the membrane stain or the nuclear stain in a first of the one or more obtained input channel images by applying a Frangi filter to the first of the one or more obtained input channel images to provide a first enhanced image; generating a refined image based at least on the first enhanced image; and detecting nuclei within the generated refined image.

[0023] In some embodiments, a first one of the one or more acquired input channel images comprises a membrane stain. In some embodiments, the refined image is generated by combining (i) the first enhanced image or a further processed variant thereof with (ii) a second one of the one or more acquired input channel images, wherein the second one of the one or more acquired input channel images comprises a nuclear stain. In some embodiments, the system further comprises instructions for generating a second enhanced image from the second one of the one or more acquired input channel images. In some embodiments, the refined image is generated by thresholding the first enhanced image.

[0024] In some embodiments, the refined image is generated by: (a) calculating a combined image derived from at least a first enhanced image and a second enhanced image, and (b) applying a threshold to the calculated combined image. In some embodiments, the second of the one or more acquired input channel images comprises a membrane stain. In some embodiments, the second of the one or more acquired input channel images comprises a nuclear stain. In some embodiments, the combined image is derived from the first and second enhanced images and at least a third enhanced image, the third enhanced image comprising a signal corresponding to a nuclear stain.

[0025] In another aspect of the present disclosure, a non-transitory computer-readable medium storing instructions for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, comprising: obtaining one or more input image channel images, wherein each of the obtained input image channel images contains a signal corresponding to one of a membrane stain or a nuclear stain; enhancing the signal corresponding to the membrane stain or the nuclear stain in the first of the one or more obtained input channel images by applying a filter adapted to enhance boundary structure to a first of the one or more obtained input channel images to provide a first enhanced image; generating a combined image based at least on the first enhanced image; and detecting nuclei within the generated combined image or a segmentation mask image derived from the generated combined image. In some embodiments, the filter adapted to enhance boundary structure is an image processing algorithm. In some embodiments, the filter adapted to enhance boundary structure is a Frangi filter. In some embodiments, the combined image is generated by calculating the sum of the first enhanced image or a further processed variant thereof and at least a second image. In some embodiments, the second image is a second of the one or more obtained input channel images. In some embodiments, the first of the one or more obtained input channel images contains a membrane stain, and the second of the one or more obtained input channel images contains a nuclear stain. In some embodiments, the combined image is derived from an inverse image of the first enhanced image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The features of the present disclosure are generally understood with reference to the accompanying drawings. In the drawings, like reference numerals are used to identify like elements throughout.

[0027] Figure 1 A representative digital pathology system including an image scanner and a computer system is shown, according to some embodiments.

[0028] Figure 2 Various modules that may be used in a digital pathology system or a digital pathology workflow according to some embodiments are described.

[0029] Figure 3A and Figure 3B A flow chart is set forth illustrating steps for detecting nuclei in an obtained input image according to some embodiments.

[0030] Figure 4A and Figure 4B A flow chart is set forth illustrating steps for detecting nuclei in an obtained input image according to some embodiments.

[0031] Figure 5A A flow chart is set forth showing steps for enhancing membrane-stained cells in an acquired input image according to some embodiments.

[0032] Figure 5B A flow chart is set forth showing steps for enhancing a blurred kernel in an obtained input image according to some embodiments.

[0033] Figure 5C and Figure 5D A flow chart is set forth showing steps for enhancing blurred nuclei and enhancing membrane-stained cells in an acquired input image according to some embodiments.

[0034] Figures 6A to 6E Provided are fields of view of IHC brightfield images stained for CD20 in B-cell lymphoma.

[0035] Figure 6A The original image is shown in which the DAB stain signal is very strong.

[0036] Figure 6B Shown is a hematoxylin channel image generated by a color unmixing algorithm, where the morphology of individual nuclei is difficult to discern.

[0037] Figure 6C The unmixed DAB image is shown.

[0038] Figure 6D The output image after applying the Frangi filter with a scaling factor of 2 is shown.

[0039] Figure 6E Hematoxylin channel images are shown ( Figure 6B )and Figure 6C Inverse image combination of Frangi filter enhanced images.

[0040] Figure 7A An image of the unmixed hematoxylin channel is shown.

[0041] Figure 7B Shown is a combined image, ie, a combination of the hematoxylin image channel image and the inverse of the DAB image channel image enhanced using a Frangi filter.

[0042] Figure 7C Show the Figure 7A Results obtained by applying the automated nucleus detection algorithm to the hematoxylin channel image.

[0043] Figure 7D Show the Figure 7B The results of applying the automated nuclear detection algorithm to the image are obtained, Figure 7C and Figure 7D The comparison shows the advantage of using Frangi-enhanced combined images for nuclear detection.

[0044] Figure 8A The input image is shown.

[0045] Figure 8BShown by unmixing Figure 8A The hematoxylin image channel image is obtained by imaging the image.

[0046] Figure 8C Shows the Frangi filter applied to Figure 8B Image of the hematoxylin channel.

[0047] Figure 8D Shown by Figure 8C Apply thresholding to the Frangi filter enhanced image and obtain the nuclear mask image.

[0048] Figure 8E Shown in the Figure 8D The nuclear mask image shows the nuclear centers obtained after running the automated nuclear detection algorithm.

[0049] Figure 9A Multiplexed images are shown, containing signals corresponding to Ki67, KRT, PDCD1, DAPI, CD8A, CD3E.

[0050] Figure 9B The DAPI channel is shown.

[0051] Figure 9C Shows the Frangi filter applied (with a scaling factor of 5) to Figure 9B The enhanced image obtained after the image is .

[0052] Figure 9D Shows the Frangi filter applied (with a scaling factor of 2) to Figure 9B The enhanced image obtained after the image is .

[0053] Figure 9E Cytokeratin (KRT) image channel images are shown.

[0054] Figure 9F Shows the Frangi filter applied to Figure 9E The enhanced image obtained after the image is .

[0055] Figure 9G Combined images of all T cell stainings are shown.

[0056] Figure 9H Shows the Frangi filter applied to Figure 9G The enhanced image obtained after the image is .

[0057] Figure 9I Show Figure 9D and Figure 9H The weighted combination image.

[0058] Figure 9J Show Figure 9C and Figure 9H The weighted combination image.

[0059] Figure 9K shows that by applying a fixed threshold to Figure 9I The nuclear mask image generated by the image.

[0060] Figure 9L The seed detection results of the original DAPI image channel are shown.

[0061] Figure 10A The second derivative of the Gaussian kernel probe inside / outside contrast in the range (-s, s) is shown.

[0062] Figure 10B Summarizes how the eigenvectors of the two-dimensional Hessian indicate the presence of local structure, where H = high, L = low, N = noisy but usually small, and + / - indicates the sign of the eigenvalue. DETAILED DESCRIPTION

[0063] It should also be understood that in any method claimed herein that includes multiple steps or actions, the order of the steps or actions is not necessarily limited to the order of the steps or actions recited in the method unless explicitly indicated to the contrary.

[0064] As used herein, the singular terms "a," "an," and "the" include plural referents unless otherwise indicated. Similarly, the word "or" is intended to include "and" unless the context clearly indicates otherwise. The term "comprising" is defined as inclusive, such that "comprising A or B" means comprising A, B, or A and B.

[0065] As used herein, the term "or" should be understood to have the same meaning as "and / or" defined above. For example, when separating the items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one of a plurality of elements or a list of elements and optional other unlisted items, but also including more than one. Only explicit indication of the opposite term, such as "only one" or "exactly one", or when used in the claims, "consisting of..." will refer to only one element of one or more elements. Generally, when preceded by an exclusive term such as "either", "one", "only one", "exactly one", the term "or" used herein should only be interpreted to indicate an exclusive choice (e.g., "one or the other, but not both"). When used in the claims, "consisting essentially of..." should have the ordinary meaning used in the field of patent law.

[0066] As used herein, the terms "including," "comprising," "having," and the like are used interchangeably and have the same meaning. Similarly, "comprising," "including," "having," and the like are used interchangeably and have the same meaning. Specifically, the definition of each term is consistent with the definition of "comprising" under ordinary U.S. patent law, and therefore each term is understood to be an open term meaning "at least the following" and is also interpreted to not exclude additional features, limitations, aspects, etc. Thus, for example, "a device having components a, b, and c" means that the device includes at least components a, b, and c. Similarly, the phrase: "a method involving steps a, b, and c" means that the method includes at least steps a, b, and c. Furthermore, although steps and processes may be outlined herein in a particular order, those skilled in the art will recognize that the order steps and processes may vary.

[0067] As used herein, with respect to a list of one or more elements, the phrase "at least one" should be understood to mean at least one element selected from any one or more elements in the list of elements, but does not necessarily include at least one of each element specifically listed in the list of elements, nor does it exclude any combination of elements in the list of elements. This definition also allows that in addition to the elements specifically identified in the list of elements to which the phrase "at least one" refers, other elements may optionally be present, whether related or unrelated to those specifically identified elements. Thus, as a non-limiting example, in one embodiment, "at least one of A and B" (or equivalently "at least one of A or B," or equivalently "at least one of A and / or B") may mean at least one (optionally including more than one) A, with no B (and optionally including elements other than B); in another embodiment, it may mean at least one (optionally including more than one) B, with no A (and optionally including elements other than A); in yet another embodiment, it may mean at least one (optionally including more than one) A and at least one (optionally including more than one) B (and optionally including other elements), and so on.

[0068] As used herein, the terms "biological sample," "biological sample," "specimen," or similar terms refer to any sample comprising biomolecules (e.g., proteins, peptides, nucleic acids, lipids, carbohydrates, or combinations thereof) obtained from any organism, including viruses. Examples of other organisms include mammals (e.g., humans; veterinary animals, such as cats, dogs, horses, cows, and pigs; and laboratory animals, such as mice, rats, and primates), insects, annelids, arachnids, marsupials, reptiles, amphibians, bacteria, and fungi. Biological samples include biological samples (e.g., tissue sections and biopsies of tissues), cell samples (e.g., cytological smears, such as cervical smears or blood smears, or obtained by microdissection), or cell fractions, fragments, or organelles (e.g., obtained by lysing cells and separating their components by centrifugation or other means). Other examples of biological samples include blood, serum, urine, semen, feces, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus, biopsy tissue (e.g., obtained by surgical biopsy or needle biopsy), nipple aspirate, cerumen, breast milk, vaginal secretions, saliva, swabs (e.g., buccal swabs), or any material containing biomolecules and derived from a first biological sample. In certain embodiments, the term "biological sample" as used herein refers to a sample prepared from a tumor or a portion thereof obtained from a subject (e.g., a homogenized or liquefied sample).

[0069] As used herein, a "spot" refers to an area in a digital image where certain properties are constant or approximately constant. In some embodiments, all pixels in a spot may be considered similar to each other in a certain sense.

[0070] As used herein, the term "image data" encompasses raw image data acquired from a biological sample, for example by an optical sensor or sensor array, or pre-processed image data. In particular, the image data may comprise a pixel matrix.

[0071] As used herein, the terms "image," "image scan," or "scanned image" encompass raw image data acquired from a biological sample, for example by an optical sensor or sensor array, or pre-processed image data. In particular, the image data may comprise a pixel matrix.

[0072] As used herein, the term "multi-channel image" or "multiplexed image" encompasses a digital image obtained from a biological sample in which different biological structures, such as nuclei and tissue structures, are simultaneously stained using specific fluorescent dyes, quantum dots, chromosomes, etc., each of which fluoresces or can be otherwise detected in a different spectral band, thereby constituting one of the channels in the multi-channel image.

[0073] As used herein, the term "slide" refers to any substrate of any suitable size on which a biological specimen can be placed for analysis (e.g., a substrate made entirely or partially of glass, quartz, plastic, silicon, etc.), and more particularly refers to a "microscope slide" such as a standard 3 x 1 inch microscope slide or a standard 75 mm x 25 mm microscope slide. Examples of biological specimens that can be placed on a slide include, but are not limited to, cytological smears, thin tissue sections (e.g., from a biopsy), and biological specimen arrays, such as tissue arrays, cell arrays, DNA arrays, RNA arrays, protein arrays, or any combination thereof. Thus, in one embodiment, a tissue section, DNA sample, RNA sample, and / or protein is placed on a specific location on a slide. In some embodiments, the term "slide" can refer to SELDI and MALDI chips, as well as silicon wafers.

[0074] As used herein, the term "specific binding entity" refers to a member of a specific binding pair. A specific binding pair is a pair of molecules that are characterized by binding to each other to substantially exclude binding to other molecules (e.g., the binding constant of a specific binding pair can be at least 10^3M-1, 10^4M-1, or 10^5M-1 greater than the binding constant of either of the two members of the binding pair of other molecules in a biological sample). Specific examples of specific binding moieties include specific binding proteins (e.g., antibodies, lectins, streptavidin, and avidin such as protein A). Specific binding moieties can also include molecules (or portions thereof) that are specifically bound by such specific binding proteins.

[0075] As used herein, the terms "stain," "staining," or similar terms generally refer to any treatment of a biological specimen that detects and / or distinguishes the presence, location, and / or amount (e.g., concentration) of a specific molecule (e.g., lipid, protein, or nucleic acid) or a specific structure (e.g., normal or malignant cells, cytoplasm, nucleus, Golgi apparatus, or cytoskeleton) in the biological specimen. For example, staining can be used to compare a specific molecule or a specific cellular structure in a biological specimen with the surrounding area, and the intensity of the staining can be used to determine the amount of the specific molecule in the specimen. Staining can be used not only with brightfield microscopy, but also with other observation tools such as phase contrast microscopy, electron microscopy, and fluorescence microscopy to assist in the observation of molecules, cellular structures, and organisms. Some staining performed by the system allows for clear visualization of cell outlines. Other staining performed by the system may rely on staining specific cellular components (e.g., molecules or structures) without staining other cellular components or with relatively little staining of other cellular components. Examples of various types of staining methods performed by the system include, but are not limited to, histochemical methods, immunohistochemical methods, and other methods based on intermolecular reactions (including non-covalent binding interactions) such as hybridization reactions between nucleic acid molecules. Specific staining methods include, but are not limited to, primary staining methods (such as H&E staining, cervical staining, etc.), enzyme-linked immunohistochemistry methods, and in situ RNA and DNA hybridization methods, such as fluorescence in situ hybridization (FISH).

[0076] Disclosed herein are systems and methods designed to enhance brightfield and / or darkfield images to better enable nuclear detection within the images. In some embodiments, a filter is applied that is suitable for enhancing boundary structures or continuous edges around cell nuclei. In some embodiments, the filter applied is a Frangi filter. After generating the enhanced image, a refined image can be calculated, wherein the refined image is based at least in part on the enhanced image. In some embodiments, the refined image can then be used as an input image for an automated nuclear detection algorithm. Applicants believe that the systems and methods disclosed herein can enable improved nuclear detection, particularly in situations where membrane or nuclear stains are weak or undetectable. In some embodiments, the systems and methods are relatively more efficient and require fewer computing resources to achieve detection of cell nuclei. In some embodiments, the systems can detect nuclei more quickly than prior art systems.

[0077] At least some embodiments of the present disclosure relate to digital pathology systems and methods for analyzing image data captured from biological samples, including biological samples stained with one or more primary stains, such as hematoxylin and eosin (H&E), and one or more detection probes, such as probes containing specific binding entities that facilitate labeling of a target within the sample. In some embodiments, the biological sample is stained to detect cell nuclei and / or cell membranes (e.g., the biological sample is stained to detect the presence of a membrane biomarker or a nuclear biomarker).

[0078] Figure 1 A digital pathology system 200 for imaging and analyzing specimens according to some embodiments is shown. In some embodiments, the digital pathology system includes, for example, a digital data processing device (such as a computer, including an interface for receiving image data from a slide scanner, a camera, a network and / or a storage medium). In other embodiments, the digital pathology system 200 may include an imaging device 12 (such as a device having a microscope slide device for scanning a specimen) and a computer 14, whereby the imaging device 12 and the computer may be coupled together in a communicative manner (such as directly, or indirectly via a network 20). The computer system 14 may include a desktop computer, a laptop computer, a tablet computer or the like, digital electronic circuitry, firmware, hardware, memory, computer storage media, a computer program or instruction set (such as the program is stored in the memory or storage medium), one or more processors (including a programmed processor), and any other hardware, software or firmware modules or combinations thereof. For example, the Figure 1 The computing system 14 shown in FIG may include a computer having a display device 16 and a housing 18. The computer may store a digital image in binary form (locally, such as in a memory, on a server, or on another network-connected device). The digital image may also be divided into a matrix of pixels. The pixels may include a digital value of one or more bits defined by a bit depth. Those skilled in the art will recognize that other computer devices or systems may be utilized, and the computer system described herein may be coupled to additional components (such as a specimen analyzer, a microscope, other imaging systems, automated slide preparation equipment, etc.) in a communicative manner. Some of these additional components and various available computers, networks, etc. will be further described herein.

[0079] Generally speaking, the imaging device 12 (or other image source including pre-scanned images stored in a memory or one or more memories) may include, but is not limited to, one or more image capture devices. Image capture devices may include, but are not limited to, cameras (e.g., analog cameras, digital cameras, etc.), optical devices (e.g., one or more lenses, sensor focusing lenses, microscope objectives, etc.), imaging sensors (e.g., charge-coupled devices (CCDs), complementary metal oxide semiconductor (CMOS) image sensors, etc.), photographic film, etc. In digital embodiments, the image capture device may include multiple lenses that can work together to provide instant focus. An image sensor, such as a CCD sensor, can capture a digital image of the specimen. In some embodiments, the imaging device 12 is a brightfield imaging system, a multispectral imaging (MSI) system, or a fluorescence microscopy system. The digitized tissue data can be generated, for example, by an image scanning system, such as the VENTANA DP200 scanner from VENTANA MEDICAL SYSTEMS, Inc. (Tucson, Arizona), or other suitable imaging devices. Other imaging devices and systems are further described herein. Those skilled in the art will recognize that the digital color image captured by the imaging device 12 can generally be composed of primary color pixels. Each color pixel can be encoded on three digital components, each containing the same number of bits, and each component corresponding to a primary color, typically red, green, or blue, also referred to by the term "RGB" components.

[0080] Figure 2 The various modules utilized within the digital pathology system 200 of the present disclosure are summarized. In some embodiments, the digital pathology system 200 employs a computer device or computer-implemented method having one or more processors 220 and at least one memory 201 storing non-transitory computer-readable instructions for execution by the one or more processors to cause the one or more processors (220) to execute instructions (or store data) in one or more modules (e.g., modules 202 to 210).

[0081] Further references Figure 2In some embodiments, the system includes: (a) an imaging module 202 adapted to generate image data of a stained biological sample, such as a first image stained for the presence of one or more protein biomarkers and a second image stained for the presence of one or more nucleic acid biomarkers; (b) an unmixing module 203 for unmixing an acquired image having more than one stain (e.g., a brightfield image) into a single channel image; (c) a Frangi filter module 204 adapted to apply a Frangi filter to enhance certain boundary structures within the acquired input image; (d) an image refinement module 205 for further processing the image, such as Frangi filter enhancement of the image; (e) a nucleus detection module 206 for detecting nucleus seed centers; and (f) a visualization module 207 for generating certain image overlays. Those skilled in the art will recognize that Figure 2 Additional modules or databases not described herein may be incorporated into the workflow. For example, an image preprocessing module may be run to apply certain filters to the acquired image or to identify certain histological and / or morphological structures within the tissue sample. Furthermore, a region of interest selection module may be utilized to select specific portions of an image for analysis.

[0082] Steering Figure 3A The present disclosure provides a computer-implemented system and method for enhancing boundary structures within a first input image (step 300). The boundary structures can be enhanced by applying a Frangi filter, such as by using the Frangi filter module 204. In some embodiments, the boundary structures enhanced by applying the Frangi filter include membrane structures, structures in the cytoplasm, and nuclear structures. After generating an image enhanced by applying the Frangi filter, a nuclear seed center is detected at least within the generated image (step 310). Of course, the boundary structure enhanced image generated in step 300 can be combined (e.g., additively or weightedly) with other images (e.g., other input images or other enhanced images obtained), and the combined image or a variation thereof can be used to detect the nuclear seed center.

[0083] In some embodiments, and with reference to Figure 3B , a computer-implemented method is provided, the method comprising: (a) obtaining one or more input images, such as unmixed image channel images, dark field images, etc. (step 310); (b) applying a Frangi filter to at least a first one of the obtained input images to provide at least a first enhanced image (step 311); (c) generating a refined image based on at least the first enhanced image (step 312); and (d) detecting nuclei in the refined image (step 313). Each of these steps is further described herein.

[0084] Image acquisition module

[0085] In some embodiments, as an initial step and with reference to Figure 2 , the digital pathology system 200 operates the imaging module 202 to capture an image or image data of a biological sample with one or more stains (e.g., from the scanning device 12) (step 310). In some embodiments, the received or acquired image is an RGB image or a multispectral image (e.g., a multi-channel bright field and / or dark field image). In some embodiments, the captured image is stored in the memory 201. In some embodiments, the acquired image is used as the one or more acquired input images (see Figure 3B Step 310).

[0086] Images or image data (used interchangeably herein) can be acquired using the scanning device 12, for example, in real time. In some embodiments, as described herein, the images are acquired from a microscope or other instrument capable of capturing image data of a microscope slide bearing a specimen. In some embodiments, the images are acquired using a two-dimensional scanner, such as a scanner capable of scanning image patches, or a line scanner, such as a VENTANA DP 200 scanner, capable of scanning an image line by line. Alternatively, the images can be images that have been previously acquired (e.g., scanned) and stored in one or more memories 201 (or retrieved from a server via the network 20).

[0087] In some embodiments, the image received as input is the entire slice image. In other embodiments, the image received as input is a portion of the entire slice image. In some embodiments, the entire slice image is decomposed into several parts, such as tiles, and each part or tile can be analyzed separately (e.g., using Figure 2 The modules listed in and at least Figure 3B After analyzing the sections or patches individually, the data for each section or patch can be stored individually and / or reported at the whole slide level.

[0088] The biological sample can be stained by applying one or more stains, and the resulting image or image data include signals corresponding to each of the one or more stains. Chromogenic stains include hematoxylin, eosin, fast red or 3,3'-diaminobenzidine (DAB). In certain embodiments, the biological sample is stained with a primary stain (e.g., hematoxylin). In certain embodiments, the biological sample is stained with a secondary stain (e.g., eosin). In certain embodiments, the biological sample is stained for a specific biomarker in IHC assays. Of course, those skilled in the art will recognize that any biological sample can also be stained with one or more fluorophores.

[0089] In some embodiments, the input image is a simplex image having only a single stain (e.g., stained with 3,3'-diaminobenzidine (DAB)). In some embodiments, the input channel image obtained is a series of simplex images. In some embodiments, the biological sample can be stained in a multiplex analysis of two or more stains (thereby providing a multiplex image). In some embodiments, the biological sample is stained for at least two biomarkers. In other embodiments, the biological sample is stained for the presence of at least two biomarkers and is also stained with a primary stain (e.g., hematoxylin).

[0090] In some embodiments, the sample is stained for the presence of at least one lymphocyte marker. Lymphocyte markers include CD3, CD4, and CD8. In general, CD3 is a "universal marker" for T cells. In some embodiments, further analysis (staining) is performed to identify the specific types of T cells, such as regulatory, helper, or cytotoxic T cells. For example, CD3+ T cells can be further distinguished as cytotoxic T lymphocytes that are positive for the CD8 biomarker (CD8 is a specific marker for cytotoxic T lymphocytes). CD3+ T cells can also be distinguished as cytotoxic T lymphocytes that are positive for Perforin (Perforin is a membrane decomposition protein expressed in the cytoplasmic granules of cytotoxic T cells and natural killer cells). Cytotoxic T cells are effector cells that actually "kill" tumor cells. It is believed that the mechanism of action of these cells is to introduce the digestive enzyme granzyme B into the tumor cell cytoplasm by direct contact, thereby killing tumor cells. Similarly, CD3+ T cells can be further distinguished as regulatory T cells that are positive for the FOXP3 biomarker. FOXP3 is a nuclear transcription factor and the most specific marker for regulatory T cells. Similarly, CD3+ T cells can be further differentiated into helper T cells that are positive for the CD4 biomarker.

[0091] In some embodiments, the sample is stained for one or more immune cell markers, including at least CD3 or total lymphocytes detected by hematoxylin and eosin staining. In some embodiments, at least one additional T cell-specific marker may also be included, such as CD8 (a marker for cytotoxic T lymphocytes), CD4 (a marker for helper T lymphocytes), FOXP3 (a marker for regulatory T lymphocytes), CD45RA (a marker for natural T lymphocytes), and CD45RO (a marker for memory T lymphocytes). In a specific embodiment, at least two markers are used, including human CD3 (or total lymphocytes detected by H&E staining) and human CD8, wherein a single section of tumor tissue can be marked simultaneously with both markers, or serial sections can be used. In other cases, at least one of the immune cell biomarkers is a lymphocyte identified in hematoxylin and eosin stained sections.

[0092] In some embodiments, samples are stained for the presence of lymphocyte biomarkers and tumor biomarkers. For example, in epithelial tumors (cancer), cytokeratin staining is used to distinguish tumor cells from normal epithelium. This information, along with the fact that tumor cells abnormally overexpress cytokeratin compared to normal epithelial cells, enables one to distinguish tumors from normal tissue. For melanoma tissue derived from the neuroectoderm, the S100 biomarker has a similar purpose.

[0093] T cells (e.g., CD8 positive cytotoxic T cells) can be further distinguished by various biomarkers (including PD-1, TIM-3, LAG-3, CD28, and CD57). Therefore, in some embodiments, T cells are stained with at least one of various lymphocyte biomarkers (e.g., CD3, CD4, CD8, FOXP3) to achieve their identification, and are stained with additional biomarkers (LAG-3, TIM-3, PD-L1, etc.) to achieve further differentiation. In some embodiments, biological samples are stained for lymphocyte biomarkers and PD-L1. For example, tumor cells can be distinguished as cells that are positive for the biomarker PD-L1, which is believed to affect the interaction between tumor cells and immune cells.

[0094] A typical biological sample is processed in an automated staining / assay platform that stains the sample. There are a variety of commercially available products suitable for use as staining / assay platforms, including the Discovery TMThe product is one example. The camera platform can also include a brightfield microscope, such as the VENTANA iScan HT or VENTANA DP 200 scanners from Ventana Medical Systems, Inc., or any microscope with one or more objectives and a digital imager. Other techniques can be used to capture images at different wavelengths. Further, camera platforms suitable for imaging stained biological specimens are known in the art and are commercially available from companies such as Zeiss, Canon, and Applied Spectral Imaging, and such platforms can be readily adapted for use in the systems, methods, and devices disclosed herein.

[0095] In some embodiments, the input image is masked so that only tissue regions are present in the image. In some embodiments, a tissue region mask is generated to mask non-tissue regions from the tissue region. In some embodiments, a tissue region mask can be created by identifying the tissue region and automatically or semi-automatically (i.e., with minimal user input) excluding background regions (such as the entire slice image region corresponding to no sample glass, such as regions where only white light from an imaging source is present). Those skilled in the art will recognize that in addition to masking non-tissue regions from tissue regions, the tissue masking module can also mask other target regions as needed, such as a portion of tissue identified as belonging to a certain tissue type or to a suspected tumor region. In some embodiments, a tissue region mask image is generated by masking tissue regions from non-tissue regions in the input image using segmentation techniques. Likewise, suitable segmentation techniques are known in the art (see Digital Image Processing, 3rd edition, Rafael C. Gonzalez, Richard E. Woods, Chapter 10, page 689 and Handbook of Medical Imaging, Processing and Analysis, Isaac N. Bankman Academic Press, 2000, Chapter 2).

[0096] In some embodiments, image segmentation techniques are utilized to distinguish between digitized tissue data and slides in an image, the tissue data corresponding to the foreground and the slide corresponding to the background. In some embodiments, an area of ​​interest (AOI) in the entire slice image is calculated to detect all tissue regions in the AOI while limiting the amount of background non-tissue regions analyzed. A variety of different image segmentation techniques (such as image segmentation based on HSV colors, laboratory image segmentation, mean shift color image segmentation, region growing, level set methods, fast marching methods, etc.) can be used to determine, for example, the boundary between tissue data and non-tissue or background data. Based on at least some of the segmentation techniques, the method can also generate a tissue foreground mask that can be used to identify those portions of the digitized slide data that correspond to the tissue data. Alternatively, the method can generate a background mask for identifying portions of the digitized slide data that do not correspond to the tissue data.

[0097] This recognition can be achieved through image analysis operations (such as edge detection, etc.). Tissue region masks can be used to remove non-tissue background noise in images (such as non-tissue regions). In some embodiments, the generation of the tissue region mask includes one or more of the following operations (but not limited to the following operations): calculating the brightness of a low-resolution analysis input image, generating a brightness image, applying a standard deviation filter to the brightness image, generating a filtered brightness image, and applying a threshold to the filtered brightness image, thereby setting pixels with brightness higher than a given threshold to 1, and setting pixels below the threshold to 0, and generating the tissue region mask. Additional information and examples related to the generation of tissue region masks are disclosed in U.S. Publication No. 2017 / 0154420, entitled “An Image Processing Method and System for Analyzing a Multi-Channel Image Obtained from a Biological sample Being Stained by Multiple Stains,” the disclosure of which is incorporated herein by reference in its entirety.

[0098] Demixing module

[0099] In some embodiments, the received input image may be a multiplexed image, i.e., the received image is an image of a biological sample stained with more than one stain. In these embodiments, the multiplexed image is first unmixed into its constituent channels, such as by an unmixing module 203, before further processing, wherein each unmixed channel corresponds to a specific stain or signal. In some embodiments, the multiplexed brightfield image is first unmixed to obtain at least two image channel images, such as a DAB image channel image or a hematoxylin image channel image. In some embodiments, the unmixed image channel images may be used as the one or more input images obtained, and a Frangi filter may be applied to these images (see Figure 3B Step 310).

[0100] In some embodiments, in a sample comprising one or more stains, a separate image can be generated for each channel comprising the one or more stains. One skilled in the art will recognize that features extracted from these channels can be used to describe different biological structures (e.g., nuclei, membranes, cytoplasm, nucleic acids, etc.) present in any image of the tissue.

[0101] In some embodiments, the multispectral image provided by the imaging module 202 is a weighted mixture of the underlying spectral signals associated with individual biomarkers and a noise component. At any particular pixel, the blending weight is proportional to the marker expression of the underlying co-localized biomarker at a specific location in the tissue and the background noise at that location. Therefore, the blending weight varies between different pixels. The spectral unmixing method disclosed herein decomposes a multichannel pixel value vector into a set of constituent biomarker members or components at each pixel and estimates the proportion of the individual constituent stains for each biomarker.

[0102] Unmixing refers to the process of decomposing the measured spectrum of a mixed pixel into a set of component spectra or end members representing the proportion of each end member in the pixel, and a set of corresponding fractions or abundances. Specifically, the unmixing process can extract stain-specific channels, so that the local concentration of a single stain can be determined using reference spectra that are known for standard types of tissue and stain combinations. The unmixing can use reference spectra retrieved from control images or estimated from images under observation. Unmixing the component signals of each input pixel can retrieve and analyze stain-specific channels, such as the hematoxylin channel and eosin channel in H&E images, or the diaminobenzidine (DAB) channel and counterstain (such as hematoxylin) channel in IHC images. The terms "unmixing" and "color deconvolution" (or "deconvolution") or similar terms (such as "deconvolution", "unmixing") are used interchangeably in the prior art.

[0103] In some embodiments, the multiple images are unmixed in a linear unmixing manner with the unmixing module 205. For example, linear unmixing is described in "Zimmermann 'Spectral Imaging and Linear Unmixing in Light Microscopy' Adv Biochem Engin / Biotechnology (2005) 95:245-265' and in C.L. Lawson and R.J. Hanson, 'Solving least squares Problems', Prentice Hall, 1974, Chapter 23, page 161", the disclosures of both of which are incorporated herein by reference in their entirety. In linear stain unmixing, the measured spectrum (S(λ)) at any pixel is considered to be a linear mixture of the stain spectral components and is equal to the sum of the proportions or weights (A) of the color reference (R(λ)) of each individual stain represented at the pixel.

[0104] S(λ)=A1·R1(λ)+A2·R2(λ)+A3·R3(λ).......A i ry(λ)

[0105] More generally, it can be expressed in matrix form as

[0106] S(λ)=ΣA i ry(λ) or S=R·A

[0107] If there are M acquired channel images and N individual stains, then the columns of the M x N matrix R are the optimal colorimetric systems derived herein, the N x 1 vector A is the unknown number of individual stain ratios, and the M x 1 vector S is the multi-channel spectrum vector measured at the pixel. In these equations, the signal in each pixel (S) is measured during the acquisition of the multi-channel image, and the reference spectrum described herein, i.e., the optimal colorimetric system, is derived. By calculating the values ​​of the various stains (A i ) to the contribution of each point in the measured spectrum to determine their contribution. In some embodiments, an inverse least squares fitting method is used to solve the following system of equations to minimize the squared difference between the measured and calculated spectra.

[0108]

[0109] In this equation, j represents the number of detection channels and i is equal to the number of stains. The solution of the linear equation generally allows for constrained unmixing, forcing the weights (A) to add together.

[0110] In other embodiments, unmixing is accomplished using the methods described in WO 2014 / 195193, filed May 28, 2014, entitled "Image Adaptive Physiologically Plausible Color Separation," the disclosure of which is incorporated herein by reference in its entirety. Generally, WO 2014 / 195193 describes an unmixing method for separating component signals of an input image using an iteratively optimized reference vector. In some embodiments, image data from an assay is correlated with expected or ideal results specific to a characteristic of the assay to determine a quality metric. In cases where image quality is low or correlation is poor compared to the ideal result, one or more reference column vectors in matrix R are adjusted, and unmixing is iteratively repeated using the adjusted reference vectors until the correlation indicates a high-quality image that meets physiological and anatomical requirements. The anatomical, physiological, and assay information can be used to define rules that are applied to the measured image data to determine the quality metric. This information includes how the tissue was stained, which structures within the tissue are intended or not intended to be stained, and the relationship between structures, stains, and markers specific to the assay being processed. The iterative process generates stain-specific vectors that can generate images that accurately identify target structures and biologically relevant information without any noise or unwanted spectra, making the process suitable for analysis. The reference vectors are adjusted to a search space. The search space defines the range of values ​​that the reference vectors can represent for the stain. The search space can be determined by scanning a variety of representative training measurements, including known or commonly occurring problems, and identifying a set of high-quality reference vectors for the training measurements.

[0111] FRANGI filter module

[0112] In some embodiments, the system 200 may utilize a Frangi filter module 204 to apply a Frangi filter to one or more of the obtained input images (individually or in combination), including the obtained unmixed image channel images, to provide one or more enhanced images (see Figure 3BIn some embodiments, a Frangi filter may be applied to enhance membrane-stained cells and / or enhance blurred nuclei in one or more input images. In some embodiments, the Frangi filter module 204 may be used to generate at least two enhanced images, such as a first enhanced image with enhanced membrane boundary structure and a second enhanced image with enhanced nuclear boundary structure (or, for that matter, two images with enhanced nuclear boundary structure or alternatively, two images with enhanced membrane boundary structure). In other embodiments, the Frangi filter module 204 may be used to generate multiple enhanced images, such as three or more enhanced images (including any combination of membrane-enhanced images and / or nuclear-enhanced images). In some embodiments, the enhanced images generated by the module 204 may then be used as input to the image refinement module 205.

[0113] The Frangi filter is described by Alejandro F. Frangi, Wiro J. Niessen, Koen L. Vincken, and Max A. Viergever in "Multiscale Vessel Enhancement Filtering" (MICCAI 1998), the disclosure of which is incorporated herein by reference in its entirety. The Frangi filter enhances line structures using the eigenvector directions and eigenvalues ​​of the Hessian matrix of the line structures. In some embodiments, the Frangi filter is used to enhance boundary structures within an input image, such as membrane boundary structures or nuclear boundary structures. In some embodiments, the Frangi filter is applied to generate an image in which elongated structures (such as membranes) and spot-like structures (such as cell nuclei) are enhanced. In some embodiments, the Frangi filter can be configured to detect structures with varying levels of local linearity. In some embodiments, membrane and nuclear edges exhibit a certain degree of linearity at each local point. To address this issue, in some embodiments, the variable β of the Frangi filter (as shown below) can be modified (e.g., set to β = 0.5), as further described herein.

[0114] The "vascularity" index is obtained from all eigenvalues ​​of the Hessian matrix. Analyzing the second-order information (Hessian) in the context of vessel detection has an intuitive basis. The second-order derivative of the Gaussian kernel on scale s generates a probe kernel that measures the contrast between regions inside and outside the range (-s, s) in the direction of the derivative (see the formula shown here and Figure 4A ).

[0115] The Hessian matrix of an “n” dimensional continuous function “f” includes second-order derivatives. The Hessian matrix of a two-dimensional image is

[0116]

[0117] Compute the Hessian matrix H at each pixel location x0 and scale s o,s The Frangi filter uses s as the standard deviation (σ) of the second-order derivative of the Gaussian approximation. According to the Hessian matrix H 0,s The eigenvalue λ1<λ2, using the "dissimilarity index R B The "vascularity" feature V0(s) at pixel position x0 is calculated using the formula of "second-order structurality" S (see below). For 2D images, let λ k is the Hessian eigenvalue with the kth smallest magnitude (i.e., |λ1| ≤ |λ2|). In particular, pixels belonging to vascular regions will be indicated by a signal with a small magnitude (ideally zero) of λ1 and a large magnitude of λ2 (the symbols represent brightness / darkness). The Frangi filter for detecting bright linear structures of scale s is defined as follows:

[0118]

[0119] in

[0120]

[0121] Where β and c are constants that control the sensitivity of the filter. D is the size of the image. R B is a measure of spottyness in 2D and describes the eccentricity of the second-order ellipse. B This filter accounts for deviations from speckled structures but cannot distinguish background noise from actual blood vessels. Because the derivative magnitude of background pixels is small, a small eigenvalue, S, helps distinguish noise from background. The parameters β and c are user-defined and can be adjusted to control the sensitivity of the filter. In some embodiments, a smaller β parameter makes the Frangi filter more sensitive to thinner structures. In some embodiments, a smaller c parameter makes the filter more sensitive to weak signals. The parameter s represents the spatial scale of the filter, corresponding to the width of the Gaussian kernel probe used to calculate the derivative. Using a larger s parameter makes the Frangi filter more sensitive to wider structures.

[0122] Figure 10A The second derivative of the inside / outside contrast of the Gaussian kernel probe in the range (-s, s) is shown. Figure 10A In the specific example described in , s=1. Figure 10BFigure 2 shows how the 2D Hessian eigenvectors indicate the presence of local structure. For example, H = high, L = low, N = noisy but usually small, and + / - indicates the sign of the eigenvalue. In some embodiments, if the absolute values ​​of both eigenvalues ​​are small (N, N), there is no obvious linear structure. In other embodiments, if one eigenvalue is low and the other is high, it indicates the presence of linear structure, which can be a bright structure (L, H-) (like a "ridge") or a dark structure (L, H+) (like a "valley"). In other embodiments, if both eigenvalues ​​are high, it indicates the presence of a spotty structure, which can be a bright structure (H-, H-) or a dark structure (H+, H+).

[0123] The versatility of the Frangi filter in detecting bright and dark linear structures of different scales can be used to detect various boundary-defined structures of cells (e.g., membrane structures, nuclear structures, cytoplasmic structures). For this specific task, since the morphological linearity of cell boundary-defined structures (e.g., the edge of the cell nucleus or the cell membrane) is intuitively between a perfect straight line and a spot, the parameter β is first set to 0.5 to make the filter more sensitive to any structures that are close to ellipses in local morphology.

[0124] Depending on the original staining pattern, i.e., whether the biomarker to be detected is a nuclear marker or a membrane / cytoplasmic marker, a Frangi+ or Frangi- filter can be used. In some embodiments, the cell "edge" appearing in nuclear or cytoplasmic staining images should be detected as a dark structure (i.e., using a Frangi- filter) because the dark side of the cell "edge" typically has better signal uniformity in IHC images. On the other hand, in membrane staining images, the membrane appears as a "ridge" in the staining intensity map and should be detected as a bright structure (i.e., using a Frangi+ filter).

[0125] Regarding the scale of the Frangi filter, in cell detection, it is believed that there is no significant variation in the width of the cell edge or membrane. Experiments have shown that a fixed value generally works well for various cell types in different tissue markers. In some embodiments, the value s is set to approximately 1 micron, which roughly corresponds to approximately 2 pixels in a microscopic image scanned at approximately 20x magnification.

[0126] Finally, parameter "c" is the only parameter that needs to be set empirically based on the signal level of the image. In some embodiments, because the filter response is tightly controlled by the speckle measurement, this parameter does not make the filter strongly signal-level dependent. By setting parameter "c," we actually get closer to setting the minimum signal level requirement, which is desirable in real applications.

[0127] In some embodiments, the scaling factor used in applying the Frangi filter to the obtained input image is about 2 pixels to 5 pixels, which corresponds to about 1 micron to 3 microns in an image scanned at 20 times magnification, or about 0.5 microns per pixel. In other embodiments, the scaling factor is selected to promote enhancement of cell boundaries, for example, a scaling factor of about 2 (see Figure 9C In other embodiments, the scaling factor is selected to promote enhancement of the cell body, for example, the scaling factor is selected to be about 5 (see Figure 9D In some embodiments, when the nucleus of a cell is present in the image, it is desirable to enhance the cell body, which is beneficial for completing cell segmentation (ie, dividing the area covered by each individual cell).

[0128] In some embodiments, the Frangi filter may be applied multiple times to the same input image obtained, but wherein the Frangi filter is applied independently to each image using a different scaling factor. For example, the Frangi filter may be first applied to an input image obtained by DAPI staining at a first scaling factor (e.g., 2), and then applied to the same original image at a second scaling factor (e.g., 5), respectively, to provide a first enhanced image and a second enhanced image, each having a different enhancement pattern (compare Figure 9C and Figure 9D ). In some embodiments, by using a relatively large scaling factor (e.g., a scaling factor of 5), the entire cell width within a range of scales will be enhanced. In some embodiments, for wider (larger) cells, the outlines will be enhanced so that all cells (regardless of the original signal level) are enhanced to have a similar contrast to the background, making it easier to perform cell detection / segmentation on the image enhanced by the Frangi filter than on the original image (i.e., the image to which the Frangi filter enhancement was not applied). In other embodiments, a smaller scaling factor (e.g., a scaling factor of 2) is used to enhance the edges of cells (the dark side is required due to the uniformity of the signal in the background), which may be optional in addition to enhancing other cell boundary-defining structures (e.g., membranes).

[0129] Image refinement module

[0130] After generating one or more enhanced images by applying the Frangi filter to the acquired image channel images, an image refinement module 205 is used to generate a refined image ( Figure 3B Step 312).

[0131] In some embodiments, the refined image generated by the image refinement module is generated by combining at least two images, where the combination can be, for example, an additive combination or a weighted combination. In some embodiments, input images (e.g., acquired input channel images, enhanced images, or any combination thereof) are combined, such as by averaging or summing (on a pixel-by-pixel basis), to produce an output combined image. In other embodiments, the combined image is a weighted combination of two or more images. In some embodiments, a weighted combined image is used when the dynamic range of one input image differs from the dynamic range of the Frangi filter output image. In some embodiments, image weighting is used to prevent or mitigate undesirable inverse image enhancement (e.g., formation of valleys in the membrane region of a nuclear stain image) when membrane signals are present at the top of the nucleus. This is believed to occur in elongated cells or when membrane fragments from other cells are present at the top of the nucleus. For example, when the DAPI signal is low, the weight of the enhanced membrane image may be higher; alternatively, when the DAPI signal is high, the weight of the enhanced membrane image may be relatively lower.

[0132] Steering Figure 4A In some embodiments, one or more of the obtained input images may be enhanced by applying a Frangi filter (step 411) (step 410). Those enhanced images may then be combined with the image refinement module 205 to generate a combined image derived from at least one of the enhanced images (step 412). In some embodiments, the combined image is derived from the enhanced image and a second image. In some embodiments, the second image is the obtained input channel image. In other embodiments, the second image is another enhanced image. In other embodiments, the combined image is derived from at least three images, the at least three images being derived from any combination of the enhanced images or the obtained input images. By way of example, assume that the four input channel images obtained are A, B, C, and D. Additionally, assume that two enhanced images are generated by applying a Frangi filter to provide B' and C' enhanced images. According to this example, the combined image may include A+B'. Alternatively, the combined image may include A+A'. Another possible combined image includes A+C'+D.

[0133] In some embodiments, the input image or enhanced image may be further processed using the image refinement module 205 and then the processed image may be combined with another image. In some embodiments, the refinement module 205 generates an inverse image of the image received as input. For example, the refinement module may include instructions for generating an inverse image (or complementary image) of the enhanced image (compare, for example, Figure 6D and Figure 6E). To further illustrate the above example, an inverse image of the enhanced image B' may be generated to provide B". The resulting combined image may comprise an additive combination of, for example, B"+A; or may comprise an additive or weighted combination of, for example, B"+A+D.

[0134] In some embodiments, the refinement module 205 generates a segmentation mask image from the enhanced image or the combined image. A "segmentation mask" is an image mask created, for example, by a segmentation algorithm that allows one or more pixel blobs (serving as "foreground pixels") to be separated from other pixels (constituting "background"). For example, a segmentation mask can be generated by a nuclear segmentation algorithm, and applying the segmentation mask to an image showing a tissue section can identify nuclear blobs in the image.

[0135] Steering Figure 4B In some embodiments, the obtained one or more input images may be enhanced (step 420) by applying a Frangi filter (step 421). The resulting enhanced image may then be provided to the image refinement module 205 so that a segmentation mask image may be generated (step 422), which in turn may be used to detect kernels (step 423).

[0136] In some embodiments, the segmentation mask image is generated from a single enhanced image. For example, a single enhanced image can be generated and thresholded to provide the segmentation mask image. In other embodiments, the segmentation mask image is generated from a combined image derived from at least the first enhanced image and another image (e.g., the second enhanced image, another input channel image, or any combination thereof). The combined image can be generated using any of the procedures described above and shown in the examples above.

[0137] Any filter known to those of ordinary skill in the art can be applied to provide a segmentation mask that meets the standards of the present disclosure. In some embodiments, foreground segmentation is achieved by applying a series of filters, including global thresholding, local adaptive thresholding, morphological operations, and watershed transformation. The filters can be run sequentially, or in any order deemed necessary by those of ordinary skill in the art. Of course, any filter can be applied iteratively until the desired result is obtained. In some embodiments, a segmentation mask image is generated by thresholding at least the first enhanced image or combined image derived from the calculated enhanced image. Generally speaking, thresholding is a method for converting an intensity image (I) into a binary image (I'), which assigns a value of 1 or 0 to all pixels whose intensity is greater than or less than a certain threshold (here, a global threshold). In other words, thresholding is applied to partitioned pixels according to the intensity value. Other thresholding methods are described in U.S. Patent Publication Nos. 2017 / 0337695 and 2018 / 0240239, the disclosures of which are incorporated herein by reference as a whole. In some embodiments, the generated segmentation mask image is provided to the nuclear detection module 206.

[0138] In some embodiments, morphological operators are applied to remove artifacts and / or fill holes.In fact, any morphological operators known in the art can be applied, provided that the application of these morphological operators results in the removal of artifacts and / or the filling of holes as desired.

[0139] Morphology is a set-theoretic method that treats an image as a set of elements and manipulates them into geometric shapes. The basic idea is to probe an image with a simple, predefined shape, where the algorithm draws conclusions about how that shape fits or mismatches shapes within the image. This simple probe is called a structuring element. In some embodiments, morphological operations are performed using a disk-shaped structuring element. In some embodiments, the radius of the disk-shaped element ranges from about 2 to about 3. In other embodiments, the radius of the disk-shaped element is 2.

[0140] In some embodiments, a "closing" morphological operation is performed so that the circular nature of foreground objects can be preserved. It is believed that "closing" merges narrow gaps and fills holes and gaps in the image. In some embodiments, when a "fill holes" morphological operation is performed, any "holes" with a size less than or equal to about 150 pixels are identified using connected components. It is believed that after filling any internal holes, the "fill holes" operation helps return meaningful blobs and ultimately a more accurate segmentation mask.

[0141] Alternatively, holes can be filled using morphological operations (i.e., conditional dilation operations). For example, let A be a set containing one or more connected components. Form an array X0 (of the same size as the array containing A) whose elements are zero (background values) except for each position in a known connected component in A (set to 1, foreground value). The goal is to start from X0 and find all connected components by the following iterative process:

[0142] Xk=(Xk-1⊕B)∩A k=1,2,3,...

[0143] Where B is a suitable structural element. The procedure terminates when Xk=Xk-1 and Xk contains all connected components of A. Then, the result of the point detection module is provided to the classification module.

[0144] In some embodiments, connected component labeling is then applied to a foreground segmentation mask (i.e., a binary image generated by applying the segmentation filters described above, where the connected component labeling provides access to individual kernels in the segmentation mask). In some embodiments, the connected component labeling process is used to return contiguous regions in the binary image using a standard algorithm as described in Hanan Samet and Markku Tamminen, "An improved approach to connected component labeling of images," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, Florida, 1986, pp. 312-318, the disclosure of which is incorporated herein by reference in its entirety. As used herein, connected component labeling scans an image and groups its pixels based on their connectivity, i.e., all pixels in a connected component have non-zero values ​​at corresponding locations in the binary foreground segmentation mask and are connected to each other in some manner.

[0145] Automated nuclear detection module

[0146] After the refined image is generated, the refined image is provided to the nuclear detection module 206 to automatically detect the nuclei within the refined image. In some embodiments, the refined image received as input is processed to detect the nucleus center (seed) and / or segment the nucleus. In some embodiments, automated candidate nucleus detection can be performed by applying a radial symmetry-based method (see: Parvin and Bahram et al., "Iterative voting for inference of structural saliency and characterization of subcellular events"). Image Processing, IEEE Transactions on 16.3 (2007): 615-623, the disclosure of which is incorporated herein by reference in its entirety). In some embodiments, the nucleus is detected using radial symmetry to detect the center of the nucleus and then the nucleus is classified based on the staining intensity around the cell center. In some embodiments, a nuclear detection operation based on radial symmetry is performed as described in commonly assigned and co-pending patent application WO / 2014 / 140085A1, which is incorporated herein by reference in its entirety. For example, the image size can be calculated within the image and one or more votes at each pixel can be accumulated by adding the sum of the sizes within the selected area. Mean-shift clustering can be used to find the local center of the region, which represents the actual nucleus location.

[0147] In some embodiments, cell nucleus detection based on radial symmetry voting can be performed on color image intensity data, explicitly utilizing the prior knowledge that nuclei are elliptical blobs of varying sizes and eccentricities. To accomplish this, in addition to the color intensity in the input image, image gradient information is used for radial symmetry voting and combined with an adaptive segmentation process to accurately detect and locate cell nuclei. For example, as used herein, "gradient" refers to the intensity gradient of a particular pixel calculated by considering the gradient of the intensity values ​​of a group of pixels surrounding the particular pixel. Each gradient can have a specific "direction" relative to a coordinate system whose x- and y-axes are defined by two orthogonal edges of the digital image. For example, nucleus seed detection involves defining a seed as a point that is assumed to be located within the cell nucleus and serves as a starting point for locating the nucleus. The first step is to detect a seed point associated with each cell nucleus using a highly stable method based on radial symmetry, thereby detecting elliptical blobs resembling nuclei. In the radial symmetry method, the gradient image can be processed using a kernel-based voting procedure. Each pixel whose votes are accumulated by the voting kernel is processed, thereby creating a voting response matrix. The kernel is based on the gradient direction calculated at that particular pixel, the expected range of minimum and maximum kernel sizes, and the voting kernel angle (typically in the range [π / 4, π / 8]). In the resulting voting space, local maximum locations with voting values ​​above a predetermined threshold are saved as seed points. Irrelevant seeds are discarded during subsequent segmentation or classification. Other methods are discussed in U.S. Patent Publication No. 2017 / 0140246, the disclosure of which is incorporated herein by reference in its entirety.

[0148] In some embodiments, the nucleus can be identified using other techniques known to those of ordinary skill in the art. For example, the image size can be calculated from the thinned image channel, and a number of votes can be assigned to each pixel around a specified size, which is the sum of the sizes in the area around the pixel. Alternatively, a mean shift clustering operation can be performed to locate the local center within the voting image that represents the actual location of the nucleus. In other embodiments, nucleus segmentation can be used to segment the entire nucleus based on the currently known nucleus center through morphological operations and local thresholding. In other embodiments, model-based segmentation can be used to detect nuclei (i.e., a shape model of the nucleus is learned from a training data set and used as prior knowledge to segment the nucleus in the test image).

[0149] In some embodiments, the kernels are then segmented using thresholds calculated separately for each kernel. For example, since it is believed that the pixel intensities in the kernel region may vary, Otsu's method can be used to perform segmentation operations in the region surrounding the identified kernel. As will be appreciated by those of ordinary skill in the art, Otsu's method is used to determine the optimal threshold by minimizing the intra-class variance, and the method is known to those skilled in the art. More specifically, Otsu's method is used to automatically perform clustering-based image thresholding, or to reduce a grayscale image to a binary image. The algorithm assumes that the image contains two classes of pixels (foreground pixels and background pixels) that follow a bimodal histogram. The optimal threshold separating the two classes of pixels is then calculated so that a minimum or equal combined diffusion (intra-class variance) is achieved (because the sum of the pairwise squared distances is a constant), thereby maximizing their inter-class variance.

[0150] In some embodiments, the systems and methods further include automatically analyzing the spectral and / or shape characteristics of nuclei identified in the image to identify non-tumor cell nuclei. For example, spots can be identified in the first digital image in the first step. As used herein, a "spot" can be, for example, a region of a digital image where some attribute (such as intensity or grayscale value) remains constant or varies within a specified range of values. In some sense, all pixels within a spot can be considered similar to one another. For example, differential methods based on the derivative of a position function on the digital image and methods based on local extrema can be used to identify spots. A nuclear spot is a spot whose pixel and / or contour shape indicates that it is likely generated by a nucleus stained with a first stain. For example, the radial symmetry of a spot can be evaluated to determine whether the spot should be identified as a nuclear spot or any other structure, such as a staining artifact. For example, if the spot is elongated and lacks radial symmetry, the spot may not be identified as a nuclear spot, but rather as a staining artifact. Depending on the embodiment, a spot identified as a "nuclear spot" can represent a group of pixels identified as candidate nuclei that can be further analyzed to determine whether the nuclear spot represents a nucleus. In some embodiments, any type of nuclear spot can be directly used as an "identified nucleus." In some embodiments, the identified nuclei or nuclear spots are filtered to identify nuclei that are not biomarker-positive tumor cells and the identified non-tumor cell nuclei are removed from the list of identified nuclei or are not added to the list of identified nuclei in the first place.

[0151] For example, the additional spectral and / or shape features of the identified nuclear spots can be analyzed to determine whether the nucleus or nuclear spot is the nucleus of a tumor cell. For example, the nucleus of a lymphocyte is larger than the nucleus of other tissue cells (such as lung cells). In the case where the tumor cell is derived from lung tissue, the nucleus of the lymphocyte is identified by identifying all nuclear spots whose minimum size or diameter is significantly larger than the average size or diameter of the nucleus of a normal lung cell. The identified nuclear spots associated with the lymphocyte nucleus can be removed from the set of identified nuclei (i.e., "filtered"). By filtering the nuclei of non-tumor cells, the accuracy of the method can be improved. Since non-tumor cells can also express the biomarker to a certain extent according to the biomarker, an intensity signal that is not derived from a tumor cell can be generated in the first digital image. By identifying and filtering nuclei that do not belong to tumor cells from the total number of identified nuclei, the accuracy of identifying biomarker-positive tumor cells can be improved. U.S. Patent Publication 2017 / 0103521 describes these and other methods, the disclosure of which is incorporated herein by reference in its entirety. In some embodiments, once a seed is detected, a local adaptive thresholding method can be used to create spots around the center of the detection. In some embodiments, other methods can also be introduced, for example, a marker-based watershed algorithm can also be used to identify cell nuclear spots around the detected nuclear center. PCT Publication No. WO2016 / 120442 describes these and other methods, the disclosures of which are incorporated herein by reference in their entirety.

[0152] Visualization Module

[0153] A visualization module 207 can be utilized to allow for the visualization of detected nuclei or seed points to facilitate rapid and robust analysis. In some embodiments, the generated overlay image can be superimposed on the entire slice image or any portion thereof (e.g., to facilitate communication of the results to a reviewer). In some embodiments, the visualization module 207 is used to calculate an overlay image that can then be superimposed on the input image. Figure 7D and Figure 8E An example of a suitable overlay is provided in which the individual detected spots (red dots) of each cell are superimposed on a portion of the entire slice image. In some embodiments, the detected nuclei and / or seed points may be shown as shaped objects (e.g., dots, hollow circles, squares, etc.) of a specific color.

[0154] In some embodiments, each detected nucleus is visualized by a seed point (such as a seed point centered within each cell). The seed point is derived by calculating the centroid or center of mass of each identified lymphocyte (such as based on the derived region of the lymphocyte). Methods for determining the center of mass of irregular objects are known to those of ordinary skill in the art. After the center of mass of the nucleus is calculated, it is marked. In some embodiments, the location of the centroid or center of mass can be superimposed on the input image, which can also constitute the entire slice image or any portion of the slice image.

[0155] Example—Detection of membrane-stained cells with poorly morphological nuclei

[0156] In some embodiments, the systems and methods disclosed herein can be used to detect nuclei with poor morphology in an image by enhancing cells stained by membranes. In some embodiments, for brightfield images, color unmixing (such as implemented by unmixing module 203) is first applied to separate the nuclear stain and the target biomarker stain. In some embodiments and with reference to Figure 5A In a first step 510, membrane-stained cells are enhanced by applying a Frangi filter to provide a first enhanced image. In some embodiments, a scaling factor of 2 is applied. Subsequently, a refined image can be generated (step 511). In some embodiments, the refined image is a combined image. In some embodiments, the combined image is a combination of at least the first enhanced image and the second image. In some embodiments, the second image is another input image, i.e., an image to which the Frangi filter has not yet been applied. In some embodiments, the second image is an image having a signal corresponding to a nuclear stain (e.g., hematoxylin or DAPI). After the refined image is generated, nuclei are detected in the refined image (step 512). In some embodiments, the detected nuclei are superimposed on the entire slice image or a portion thereof.

[0157] Figure 6A The image showing DAB-stained CD20 lymphoma is an example (where DAB and hematoxylin are used to detect the CD20 biomarker in a B-cell lymphoma tissue sample). Since CD20 is a membrane marker, Figure 6A As shown, CD20+ cells are characterized by "brown rings." Meanwhile, nuclei that typically appear as "blue spots" in CD20- cells may or may not be present in CD20+ cells. Therefore, cell detection methods designed to detect "rings" or "spots" may not detect cells that do not conform to the typical staining pattern. Figure 6B Provides unmixed hematoxylin image channel images (I HTX ). Such as I HTX As shown in the images, the morphology of individual nuclei is difficult to discern (again demonstrating that the nuclei of CD20+ cells generally do not have a well-defined "speckled" appearance).

[0158] like Figure 5A As shown, the Frangi filter is applied to the unmixed DAB image I DAB ( Figure 6C ), and the output is as follows Figure 6D The unmixed hematoxylin image ( Figure 6B ) and Frangi filter output image ( Figure 6D ) combination. In this particular embodiment, the combined image is a combination of the nuclear stain image and the inverse image of the enhanced membrane stain image. In other words, an inverse image of the Frangi_DAB enhanced image is generated and combined with the unmixed hematoxylin image channel image. Figure 6E The resulting images shown were subjected to nuclear detection.

[0159] I 组合1. =I HTX +Max(I Frangi+_DAB )-I Frangi+_DAB

[0160] In some embodiments, the unmixed image is derived from an 8-bit RGB input image. Although it is believed that there is no theoretical upper limit to the intensity value (representing optical density), the optical density value is generally in the range of 0 to about 5, which is considered to make the addition operation numerically reasonable. In addition, compared with the original membrane stained image (i.e., 1 DAB ) compared to the enhanced membrane image (I Frangi+_DAB ) is much less spatially noisy and has less intensity variation (due to the output dynamic range being limited to 0 to 1). In the resulting combined image ( Figure 6E ), both CD20- cells and CD20+ cells appear as clear "spots" on a black or gray background. Therefore, conventional "spot" detection methods can be applied to the combined image for cell detection, for example, Figure 6E The output image shown in the figure is from running the automated image analysis algorithm.

[0161] Figures 7A-7D Comparative illustrations of certain embodiments of the present disclosure (e.g. Figure 7B ) Applying an automated nucleus detection algorithm to the thinned image to reduce the likelihood of failing to detect nuclei (compare Figure 7D and Figure 7C ). Figure 7C An example of cell detection results obtained by applying a radial symmetry based cell detection method (as described herein) on an unmixed HTX image (in which CD20+ cells are considered undetectable) is shown. In contrast, by using a combined image (such as Figure 5A ), “blue spots” and “brown rings” were detected ( Figure 7D). This example demonstrates that after the proposed image pre-processing steps, a single conventional cell detection method can be used to detect highly aggregated cells with different staining patterns.

[0162] Example - Detection of Blur Kernels

[0163] In some embodiments, the systems and methods disclosed herein can be used to detect blurred nuclei in acquired images. Figure 5B In a first step 5230, the nuclear-stained cells are enhanced by applying a Frangi filter to provide a first enhanced image. In some embodiments, the scaling factor applied is 5. Subsequently, a refined image can be generated (step 521). In some embodiments, the refined image is a segmentation mask image. After the refined image is generated, nuclei are detected in the refined image (step 522). In some embodiments, the detected nuclei are superimposed on the entire slice image or a portion thereof.

[0164] Figure 8A HER2 staining of breast cancer scanned with a Ventana DP200 scanner is shown. As shown, the hematoxylin stain appears very blurry (see Figure 8B ). The kernels in the hematoxylin unmixed image channel image are enhanced using a Frangi filter with a large scale parameter (e.g., a scale factor of 5) to provide Figure 8C The enhanced image shown. A fixed threshold is then applied to the Frangi filter output to generate the kernel mask image ( Figure 8D ). This is believed to be an effective method for enhancing weakly stained nuclei, and since the Frangi filter output is in the value range (0 to 1), it can be relatively simple to derive a threshold to generate a segmentation mask.

[0165] Image_Combination2=Frangi_HTX_Scale5

[0166] Figure 8E Provides a kernel-checked output where the kernel is overlaid on Figure 8A on the entire slice image.

[0167] Example—Detection of Multiplexed Membrane-Stained Cells

[0168] In some embodiments, as with multiplexed darkfield images, visualization of a biomarker is achieved using individual channel images. In some embodiments, nuclear and membrane stains can be enhanced simultaneously by applying a Frangi filter, and automated nuclear detection / segmentation methods can be applied to the resulting refined images. Thus, the systems and methods disclosed herein can be used to detect blurred nuclei in acquired images and also to enhance membrane boundaries in acquired images, such as Figure 5C and Figure 5D As shown. In some embodiments, nuclear-stained cells are enhanced by applying a Frangi filter to provide a first enhanced image (step 530 or 540). In some embodiments, the Frangi filter is run a second time on the same input image, but using a different scaling factor, to provide a second enhanced image (step 541). Subsequently or simultaneously, membrane-stained cells are enhanced by applying a Frangi filter to provide a third enhanced image (step 531). In some embodiments, the Frangi filter is applied to different input images, each membrane-stained, such as to provide a third enhanced image and a fourth enhanced image (step 542 or 543). After generating the enhanced image, a refined image is generated (step 532 or 544). In some embodiments, the refined image is a combined image, such as an image composed of at least two of the first enhanced image, the second enhanced image, the third enhanced image, or the fourth enhanced image, where the combination can be an additive combination or a weighted combination. In some embodiments, any of the first enhanced image, the second enhanced image, the third enhanced image, or the fourth enhanced image can be further processed before being combined into a combined image (e.g., an inverse image of one of the enhanced images can be generated and used to derive the combined image). In some embodiments, a threshold is then applied to the combined image to generate a segmentation mask image. In some embodiments, nuclei are then detected in the generated segmentation mask image (step 533 or 545). In some embodiments, the nucleus detection results are visualized, for example, superimposed on the entire slice image or any portion thereof.

[0169] For example, Figure 9A 5Plex images are shown, in which Ki67, KRT, PDCD1, DAPI, CD8A, CD3E are represented by pseudo colors. In particular, 6-channel immunofluorescence (IF) plates are used to detect 5 different biomarkers, including CD3, CD8 and PD1 membrane markers, to differentiate different types of T cells (where cytokeratin is a cytoplasmic marker for differentiating tumor cells; Ki67 is a nuclear marker for differentiating proliferating cells; and finally, DAPI is used to stain all cell nuclei). This example can differentiate cells with three different staining patterns by the following strategy: Frangi filters are applied to enhance membrane, cytoplasm and nuclear stains respectively; The structure defined by the enhanced boundaries of all generated cells is then combined with the DAPI channel image to form a refined image, which can be provided to the nuclear detection module 206.

[0170] In some embodiments, a Frangi filter is applied to cytokeratin (KRT) stained images ( Figure 9E )(I Frangi-_Cyto ) to provide enhanced Figure 9FWe also applied the Frangi filter to the combined images of PD1, CD8, and CD3 staining. Tcell ( Figure 9G ), where the scale used = 1 micron (see Figure 9H ). The final combination of images is provided as follows:

[0171] I T细胞 = Maximum (I CD8 ,I CD3 ,I PD1 ),

[0172] I Frangi = Maximum (I Frangi+_T细胞 ,I Frangi-_Cyto ,I Frangi-_Ki67 )

[0173] I 组合2 =I DAPI (1.0-I Frangi ),

[0174] Among them I T细胞 is the maximum value of all three T cell marker images; I Frangi+_T细胞 , I Frangi-_Cyto and I Frangi-_Ki67 I T细胞 , Cytokeratin Image I Cyto and Ki67 image I Ki67 The Frangi filter output.

[0175] In another embodiment, the DAPI channel (nuclei) is enhanced by a membrane stain and the DAPI line structure signal (acquired using a Frangi filter at a 1 micron scale, with 1 Frangi+_DAPI(s＝1) to avoid suppressing the nuclear DAPI signal when the effect of T cells on the nuclear signal is stronger.

[0176] I 组合3 =I DAPI ·(1-I Frangi+_DAPI(s＝1) )·(1.0-I Frangi )

[0177] As an alternative, DAPI images are DAPI The image of the nuclei can be enhanced by applying a larger scale Frangi filter (Frangi+) (e.g., a scale of 3 microns). Applying a larger scale Frangi filter (Frangi+) results in improved enhancement of the nuclei bulk, at least in this particular embodiment, compared to applying a smaller scale Frangi filter (Frangi+). The enhanced nuclei image can then be combined with the enhanced cell boundary demarcation structure image, as shown in the following formula:

[0178] I 组合4 =I Frangi+_DAPI(s＝3) (1.0-I Frangi )or

[0179] I 组合5 =I Frangi+_DAPI(s＝3) ·(1-I Frangi+_DAPI(s＝1) )·(1.0-I Frangi )

[0180] In this particular example, all cells appear as well-defined "spots" with similar contrast to the background. This process is believed to simplify cell detection / segmentation. Figure 9K It is shown that simple thresholding and connected component analysis can produce good segmentation results.

[0181] Other components for implementing the disclosed embodiments

[0182] The system 200 of the present disclosure can be coupled to a specimen processing device capable of performing one or more preparation processes on the tissue specimen. The preparation processes can include, but are not limited to, deparaffinizing the specimen, conditioning the specimen (e.g., cell conditioning), staining the specimen, performing antigen retrieval, performing immunohistochemical staining (including labeling) or other reactions, and / or performing in situ hybridization (e.g., SISH, FISH, etc.) staining (including labeling) or other reactions, as well as other processes for preparing the specimen for microscopic examination, microscopic analysis, mass spectrometry, or other analytical methods.

[0183] The processing device can apply a fixative to the specimen. The fixative can include crosslinking agents (e.g., aldehydes such as formaldehyde, paraformaldehyde, and glutaraldehyde, as well as non-aldehyde crosslinking agents), oxidizing agents (e.g., metal ions and complexes such as osmium tetroxide and chromic acid), protein denaturants (e.g., acetic acid, methanol, and ethanol), fixatives with unknown mechanisms (e.g., mercuric chloride, acetone, and picric acid), combination reagents (e.g., Carnoy's fixative, Methacarn, Bouin's solution, B5 fixative, Rossman's solution, and Gendre's solution), microwave and other fixatives (e.g., excluded volume fixation and vapor fixation).

[0184] If the specimen is a paraffin-embedded sample, the sample can be deparaffinized using a corresponding deparaffinization solution. After the paraffin is removed, any number of chemicals can be applied to the specimen in succession. These chemicals can be used for pretreatment (e.g., reversing protein crosslinks, exposing nucleic acids, etc.), denaturation, hybridization, washing (e.g., stringent washing), detection (e.g., attaching a display or marker molecule to a probe), amplification (e.g., amplifying proteins, genes, etc.), counterstaining, coverslipping, etc.

[0185] The specimen processing equipment can apply a variety of different chemical substances to the specimen. These chemical substances include but are not limited to stains, probes, reagents, rinses and / or conditioners. These chemical substances can be fluids (such as gases, liquids or gas / liquid mixtures) or similar substances. The fluid can be a solvent (such as a polar solvent, a non-polar solvent, etc.), a solution (such as an aqueous solution or other type of solution) or a similar substance. Reagents can include but are not limited to stains, wetting agents, antibodies (such as monoclonal antibodies, polyclonal antibodies, etc.), antigen retrieval solutions (such as water-based or non-aqueous antigen repair solutions, antigen retrieval buffers, etc.) or similar substances. The probe can be an isolated nucleic acid or an isolated synthetic oligonucleotide attached to a detectable label or reporter molecule. Labels can include radioactive isotopes, enzyme substrates, cofactors, ligands, chemiluminescent or fluorescent agents, haptens and enzymes.

[0186] The specimen processing device can be an automated device, such as the BENCHMARK XT instrument and the SYMPHONY instrument sold by Ventana Medical Systems, Inc. Ventana Medical Systems, Inc. is the assignee of multiple U.S. patents that disclose systems and methods for performing automated analysis, including U.S. Patent Nos. 5,650,327, 5,654,200, 6,296,809, 6,352,861, 6,827,901, and 6,943,029, and U.S. Published Patent Application Nos. 20030211630 and 20040052685, the disclosures of each of which are incorporated herein by reference in their entirety. Alternatively, specimens can also be processed manually.

[0187] After specimen processing is complete, the user can transport the specimen slide to an imaging device. In some embodiments, the imaging device is a brightfield imager slide scanner. One brightfield imager is the iScan Coreo brightfield scanner sold by Ventana Medical Systems, Inc. In automated embodiments, the imaging device is a digital pathology device disclosed in International Patent Application No. PCT / US2010 / 002772 (Patent Publication No. WO / 2011 / 049608), entitled "IMAGING SYSTEM AND TECHNIQUES," or U.S. Patent Publication No. 61 / 533,114, filed on September 9, 2011, entitled "IMAGING SYSTEMS, CASSETTES, AND METHODS OF USING THESAME." The disclosures of International Patent Application No. PCT / US2010 / 002772 and U.S. Patent Application No. 61 / 533,114 are incorporated herein by reference in their entirety.

[0188] The imaging system or device can be a multispectral imaging (MSI) system or a fluorescence microscopy system. The imaging system used in this article is MSI. Generally speaking, MSI equips a computerized microscope-based imaging system for the analysis of pathological specimens by accessing the spectral distribution of images on a pixel layer. Although there are various multispectral imaging systems, these systems have one thing in common in their operation, namely, the ability to form multispectral images. Multispectral images refer to images that capture image data at specific wavelengths or specific spectral bandwidths of the electromagnetic spectrum. These wavelengths can be selected by optical filters or using other instruments that can select predetermined spectral components, including electromagnetic radiation with wavelengths outside the visible light range, for example, infrared (IR).

[0189] The MSI system may include an optical imaging system, a portion of which includes a spectral selection system adjustable to define a predetermined number N of discrete optical wavelength bands. The optical system may be suitable for imaging a biological sample that is illuminated by a broadband light source transmitted to an optical detector. In one embodiment, the optical imaging system may include a magnification system, such as a microscope, having a single optical axis that is generally spatially aligned with the single light output of the optical system. When the spectral selection system is adjusted or tuned (e.g., using a computer processor), the system forms a sequence of images of the tissue, such as to ensure that images are captured in different discrete spectral bands. The device may additionally include a display that can display at least one visually perceptible image of the tissue from the captured image sequence. The spectral selection system may include an optical dispersive element, such as a diffraction grating, a set of optical filters, such as a thin film interference filter, or any other device adapted to select a specific passband from the spectrum of light transmitted from the light source through the sample to the detector in response to user input or preprogrammed processor commands.

[0190] In an alternative embodiment, the spectrally selective system defines a plurality of light outputs corresponding to N discrete spectral bands. This type of system takes the transmitted light output from the optical system and spatially redirects at least a portion of the light output along N spatially distinct optical paths so as to image the sample in an identified spectral band onto a detector system along the optical path corresponding to the identified spectral band.

[0191] The subject matter described in this specification and the embodiment of operation can be implemented in digital electronic circuit or in computer software, firmware or hardware (including the structure disclosed in this specification and its equivalent structure), or be implemented with their one or more combinations.The embodiment of the subject matter described in this specification can be implemented as one or more computer programs, i.e. one or more modules of computer program instructions, which are encoded on a computer storage medium to be executed by a data processing device or to control the operation of a data processing device. Any module described herein may include the logic executed by a processor. As used herein, "logic" refers to any information in the form of instruction signals and / or data, which can be applied to affect the operation of a processor. Software is an example of logic.

[0192] A computer storage medium can be, or be contained in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be, or be contained in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.

[0193] The term "programmable processor" encompasses all kinds of devices, equipment, and machines for processing data, including, by way of example, a programmable microprocessor, a computer, a system on a chip, or multiple or combinations of the foregoing. The device may include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more thereof. The device and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0194] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages) and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0195] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0196] By way of example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The essential elements of a computer are a processor for performing actions according to instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include or be operably connected to receive data from or transmit data to, or receive data from and transmit data to, one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks). However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0197] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as an LCD (liquid crystal display), an LED (light emitting diode) display, or an OLED (organic light emitting diode) display, and a keyboard and pointing device (e.g., a mouse or trackball) for displaying information to the user, and the user may provide input to the computer via the keyboard and pointing device. In some embodiments, a touch screen may be used to display information and receive input from the user. Other types of devices may also be used to provide interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback (such as visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including sound, voice, or tactile input). In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from a web browser.

[0198] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks). For example, Figure 1 The network 20 may include one or more local area networks.

[0199] The computing system may include any number of clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. The relationship between the client and server is generated by computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input therefrom). Data generated at the client device (e.g., the result of a user interaction) can be received from the client device at the server.

[0200] Additional embodiments

[0201] In another aspect of the present disclosure, a method for enhancing the detection of nuclei in a multi-channel image (e.g., a multi-channel brightfield image) of a biological sample stained with hematoxylin and eosin and / or stained for the presence of one or more biomarkers is provided, the method comprising: unmixing the multi-channel image into a plurality of unmixed image channel images, wherein each unmixed image channel image comprises a signal corresponding to one of hematoxylin, eosin, or a single biomarker; applying a Frangi filter to at least a first unmixed image channel image of the plurality of unmixed image channel images to provide at least a first Frangi-enhanced image channel image; generating a first combined image, wherein the first combined image comprises at least (i) the first Frangi-enhanced image channel image or a derivative of the first Frangi-enhanced image channel image; and (ii) at least one of (a) a second unmixed image channel image and / or (b) a second Frangi-enhanced image channel image; generating a refined image based on at least the first combined image; and detecting nuclei in the first refined image using an automated image analysis algorithm. In some embodiments, the refined image is an inverse image of the first combined image. In some embodiments, the refined image is a segmentation mask image. In some embodiments, at least the first Frangi-enhanced image channel image is a Frangi-enhanced membrane image channel image. In some embodiments, the Frangi-enhanced membrane image channel image is derived from one or more of the unmixed membrane image channel images. In some embodiments, at least the first Frangi-enhanced image channel image is a Frangi-enhanced nuclear image channel image.

[0202] In some embodiments, at least a first Frangi-enhanced image channel image is combined with a second Frangi-enhanced image channel image, wherein the at least first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image are derived from different unmixed channel images. In some embodiments, at least a first Frangi-enhanced image channel image is combined with a second Frangi-enhanced image channel image and a third Frangi-enhanced image channel image, wherein the at least first Frangi-enhanced image channel image and one of the second or third Frangi-enhanced image channel images are derived from different unmixed channel images. In some embodiments, at least the first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image are derived from the same unmixed image channel image, but wherein Frangi filters are applied to the same unmixed image channel image at different scaling factors. In some embodiments, a Frangi filter is applied to a first copy of the unmixed image channel image at a first scaling factor to generate an image channel image enhanced for elongated structures; and wherein a Frangi filter is applied to a second copy of the unmixed image channel image at a second scaling factor to generate an image channel image enhanced for speckled structures. In some embodiments, the Frangi enhanced membrane image channel image is combined with at least one unmixing kernel image channel image. In some embodiments, the unmixing kernel channel and enhanced membrane image channel image are further combined with at least a second enhanced image channel image.

[0203] In another aspect of the present disclosure, a system for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, wherein the system includes: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories being configured to store computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: obtaining one or more input image channel images, wherein each obtained input image channel image contains a signal corresponding to one of a membrane stain or a nuclear stain; enhancing the membrane stain or the nuclear stain in the first of the one or more obtained input channel images by applying a filter suitable for enhancing boundary structure to the first of the one or more obtained input channel images to provide a first enhanced image; generating a combined image based at least on the first enhanced image; and detecting nuclei within the generated combined image or a segmentation mask image derived from the generated combined image.

[0204] In some embodiments, the combined image is generated by computing the sum of the first enhanced image, or a further processed variant thereof, and at least a second image. In some embodiments, the second image is a second of the one or more acquired input channel images. In some embodiments, the first of the one or more acquired input channel images comprises a membrane stain, and wherein the second of the one or more acquired input channel images comprises a nuclear stain. In some embodiments, the combined image is derived from an inverse of the first enhanced image.

[0205] In another aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the method comprising: obtaining one or more input image channel images, wherein each obtained input image channel image contains a signal corresponding to one of a membrane stain or a nuclear stain; enhancing the membrane stain or the nuclear stain in the first of the one or more obtained input channel images by applying a filter suitable for enhancing boundary structure to the first of the one or more obtained input channel images to provide a first enhanced image; generating a combined image based at least on the first enhanced image; and detecting nuclei within the generated combined image or a segmentation mask image derived from the generated combined image.

[0206] In some embodiments, the combined image is generated by computing the sum of the first enhanced image, or a further processed variant thereof, and at least a second image. In some embodiments, the second image is a second of the one or more acquired input channel images. In some embodiments, the first of the one or more acquired input channel images comprises a membrane stain, and wherein the second of the one or more acquired input channel images comprises a nuclear stain. In some embodiments, the combined image is derived from an inverse of the first enhanced image.

[0207] In another aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, the method comprising: (a) obtaining one or more input image channel images, wherein each obtained input image channel image contains a signal corresponding to one of a membrane stain or a nuclear stain; (b) enhancing the membrane stain or the nuclear stain in the first of the one or more obtained input channel images by applying a Frangi filter to the first of the one or more obtained input channel images to provide a first enhanced image; (c) generating a refined image based at least on the first enhanced image; and (d) automatically detecting nuclei within the generated refined image, wherein the nuclei are detected by applying an automated nuclear detection algorithm. In some embodiments, the detected nuclei are then superimposed on the original entire slice image or any portion thereof (i.e., made visible). In some embodiments, the membrane stain is DAB and the nuclear stain is hematoxylin. In some embodiments, the nuclear stain is DAPI. In some embodiments, the obtained input channel images are unmixed brightfield images. In some embodiments, the obtained input channel images are darkfield images.

[0208] In some embodiments, the refined image is a segmentation mask image. In some embodiments, the segmentation mask image is derived from a combined image generated from at least the first enhanced image and the second image, thereby combining the at least the first enhanced image and the second image in an additive or weighted manner. In some embodiments, the combined image is derived from at least the first enhanced image and the second enhanced image, such as a combination of a membrane enhanced image and a nuclear stain enhanced image.

[0209] In some embodiments, the refined image is a combined image derived from at least a first enhanced image and a second image (e.g., the second enhanced image, another obtained input image, or a further processed variant thereof). In some embodiments, the first image and the second image are combined in an additive manner, i.e., they are summed. In some embodiments, the first enhanced image is further processed prior to generating the combined image. In some embodiments, the further processing of the first enhanced image includes generating an inverse image of the first enhanced image.

[0210] In another aspect of the present disclosure, a method for enhancing the detection of cell nuclei within an image of a stained biological sample is provided, wherein the system comprises: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories configured to store computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: obtaining one or more input image channel images, wherein each obtained input image channel image comprises a signal corresponding to one of a membrane stain or a nuclear stain; enhancing the membrane stain or the nuclear stain in a first of the one or more obtained input channel images by applying a Frangi filter to the first of the one or more obtained input channel images to provide a first enhanced image; generating a refined image based at least on the first enhanced image; and detecting nuclei within the refined image. In some embodiments, the refined image is generated by thresholding the first enhanced image. In some embodiments, the refined image is generated by combining the first enhanced image with at least a second image to provide a combined image. In some embodiments, the refined image is generated by (i) combining the first enhanced image with at least the second image to provide a combined image; and (ii) thresholding the combined image. In some embodiments, the second image is a second one of the one or more input channel images. In some embodiments, the second image is a second enhanced image. In some embodiments, any combined image may be derived from three or more images, including any combination of the original input image, channel images, or enhanced images.

[0211] In another aspect of the present disclosure, a system for enhancing the detection of cell nuclei in an image of a stained biological sample is provided, wherein the system comprises: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: (a) obtaining at least one input channel image comprising a signal corresponding to a nuclear stain; (b) enhancing the nuclear stain in the at least one acquired input channel image (comprising the signal corresponding to the nuclear stain) by applying a Frangi filter to the at least one acquired input channel image (comprising the signal corresponding to the nuclear stain) to provide a first enhanced image; (c) generating a refined image from the at least first enhanced image; and (d) automatically detecting nuclei in the generated refined image, wherein the nuclei are detected by applying an automated nuclear detection algorithm. In some embodiments, the method further comprises: obtaining at least one input channel image comprising a signal corresponding to a membrane stain; enhancing the membrane stain in the at least one acquired input channel image (comprising the signal corresponding to the membrane stain) by applying a Frangi filter to the at least one acquired input channel image (comprising the signal corresponding to the membrane stain) to provide a second enhanced image. In some embodiments, at least the first enhanced image and the second enhanced image are combined, such as in an additive or weighted manner, to provide a refined image. In some embodiments, the kernel can be detected using the combined image; or alternatively, a threshold can be applied to the combined image, and the resulting segmentation mask image can be used to detect the kernel.

[0212] In another aspect of the present disclosure, a system for enhancing the detection of cell nuclei in an image of a stained biological sample is provided, wherein the system comprises: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: (a) obtaining at least one input channel image comprising a signal corresponding to a membrane stain; (b) enhancing the membrane stain in the at least one acquired input channel image (comprising a signal corresponding to the membrane stain) by applying a Frangi filter to the at least one acquired input channel image (comprising a signal corresponding to the membrane stain) to provide a first enhanced image; (c) generating a refined image from at least the first enhanced image; and (d) automatically detecting nuclei in the generated refined image, wherein the nuclei are detected by applying an automated nuclear detection algorithm. In some embodiments, the refined image is a combined image comprising: (i) a further processed version of the first enhanced image, and (ii) a second acquired input channel image comprising a signal corresponding to the nuclear stain. In some embodiments, the further processed version of the first enhanced image is an inverse image of the first enhanced image. In some embodiments, other acquired input channel images contain signals corresponding to hematoxylin.

[0213] In another aspect of the present disclosure, a system for enhancing detection of cell nuclei within an image of a biological sample stained with hematoxylin and eosin and / or stained for the presence of multiple biomarkers is provided, wherein the system comprises: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising: obtaining one or more input image channel images, wherein each obtained input image channel image comprises a signal corresponding to one of hematoxylin, DAPI, or a stain indicative of the presence or absence of a biomarker; enhancing boundary structures within at least a first one of the one or more image channel images by applying a Frangi filter to at least a first one of the one or more image channel images to provide at least a first Frangi-enhanced image channel image; and detecting nuclei within a refined image derived from the at least one Frangi-enhanced image channel.

[0214] In some embodiments, the refined image is a segmentation mask image. In some embodiments, the segmentation mask image is generated by thresholding at least a first Frangi-enhanced image channel image. In some embodiments, the segmentation mask image is generated by thresholding a combined image derived from at least the first Frangi-enhanced image channel image. In some embodiments, the combined image is derived by combining: (i) at least the first Frangi-enhanced image channel image; and (ii) at least one of (a) a second of one or more input image channel images and / or (b) a second Frangi-enhanced image channel image. In some embodiments, the combined image comprises a combination of at least the first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image. In some embodiments, the first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image are derived from different input image channel images. In some embodiments, the first Frangi-enhanced image channel image and the second Frangi-enhanced image channel image are derived from the same input image channel image. In some embodiments, the first and second Frangi-enhanced image channel images are derived by applying a Frangi filter at different scaling factors. In some embodiments, one of the first Frangi-enhanced image channel image or the second Frangi-enhanced image channel image comprises enhanced membrane boundaries; and the other of the first Frangi-enhanced image channel image or the second Frangi-enhanced image channel image comprises enhanced nuclear boundaries. In some embodiments, the combined image comprises a combination of at least the first Frangi-enhanced image channel image, the second Frangi-enhanced image channel image, and the third Frangi-enhanced image channel image, wherein at least one of the first, second, or third Frangi-enhanced image channel images is derived from a different input channel image.

[0215] In some embodiments, the refined image is a combined image obtained by combining: (i) an inverse image of the first Frangi-enhanced image channel image; and (ii) a second one of the one or more input channel images. In some embodiments, the combined image is obtained by combining: (i) at least the first Frangi-enhanced image channel image; and (ii) at least one of (a) the second one of the one or more input image channel images and / or (b) the second Frangi-enhanced image channel image.

[0216] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications mentioned in this specification and / or listed in the Application Data Sheet are incorporated herein by reference in their entirety. Aspects of the embodiments may be modified, if necessary, to employ concepts from the various patents, applications, and publications to provide yet further embodiments.

[0217] Although the present disclosure has been described with reference to a number of illustrative embodiments, it should be understood that numerous other modifications and embodiments can be devised by those skilled in the art that will fall within the spirit and scope of the principles of the present disclosure. More particularly, within the scope of the foregoing disclosure, the drawings and the appended claims, reasonable variations and modifications in the component parts and / or arrangements of the subject combination arrangement are possible without departing from the spirit of the present disclosure. In addition to variations and modifications in the component parts and / or arrangements, alternative uses will also be apparent to those skilled in the art.

Claims

1. A method for enhancing detection of cell nuclei within an image of a stained biological sample, the method comprising: (a) obtaining one or more unmixed image channel images from an image of a whole slide stained with at least one nuclear stain and at least one membrane stain, wherein each obtained unmixed image channel image comprises a signal corresponding to the at least one membrane stain or the nuclear stain; (b) enhancing the signal corresponding to the at least one membrane stain or the at least one nuclear stain in the first of the one or more obtained unmixed image channel images by applying a Frangi filter at a first scaling factor to the first of the one or more obtained unmixed image channel images to provide a first enhanced image; (c) generating a refined image, wherein the refined image is generated by combining the first enhanced image with at least a second image to provide a combined image, wherein the second image is a second enhanced image, wherein the second enhanced image is derived from the first of the one or more unmixed image channel images, and wherein the second enhanced image is generated by applying the Frangi filter at a second scaling factor, wherein the second scaling factor is different from the first scaling factor; (d) Detecting nuclei within the resulting thinned image; as well as (e) Seed points representing detected nuclei are superimposed on the whole slide image. 2 . The method of claim 1 , wherein generating the refined image further comprises thresholding the combined image.

3. The method of claim 1, wherein the first of the one or more unmixed image channel images comprises a membrane stain.

4. The method of claim 1, wherein the combined image is derived from at least the first enhanced image, the second image, and a third image. The method of claim 4 , wherein the third image is enhanced by applying the Frangi filter. The method of claim 4 , wherein the third image comprises a nuclear stain. The method of claim 4 , wherein the combined image is further derived from a fourth image.

8. The method according to any one of claims 1 to 7, wherein the refined image is a combined image comprising an inverse image of the first enhanced image and at least a second image.

9. The method of any one of claims 1 to 7, wherein the membrane stain is DAB, and wherein the nuclear stain is hematoxylin.

10. The method according to any one of claims 1 to 7, wherein the obtained unmixed image channel image is an unmixed bright field image or a dark field image.

11. A system for enhancing detection of cell nuclei within an image of a stained biological sample, the system comprising: (i) one or more processors, and (ii) one or more memories coupled to the one or more processors, the one or more memories configured to store computer-executable instructions that, when executed by the one or more processors, cause the system to perform a method according to any one of claims 1 to 10.

12. A non-transitory computer-readable medium storing instructions for executing the method according to any one of claims 1-10.

Images